Extracting Hand Orientation and Wrist Angle of Individual with Spinal Cord Injury From Egocentric Video

Dousty, Mehdy (1,2), Visée, Ryan (1,2), Zariffa, Jose (1,2)

1: Institute of Biomaterial and Biomedical Engineering, University of Toronto, ON

2: Toronto Rehabilitation Institute – University Health Network, ON

Videos from wearable cameras (egocentric video) can be used to evaluate the hand function of individuals with spinal cord injury (SCI) at home. Wrist tenodesis is a grasping strategy that enables individuals with C6 or C7 SCI to passively close their fingers using wrist extension. Thus, accurate evaluation of wrist flexion/extension angle is crucial to understand grasping strategies after SCI. Here, we propose a new method based on computer-vision algorithms to extract wrist angle data from an egocentric video frame.
85 random egocentric images were selected for this study from a dataset of persons with cervical SCI performing activities of daily living in a home simulation laboratory. Images differ by hand posture, skin color, and length of shirt sleeve. The method consists of two major consecutive steps: machine learning to extract hand joint coordinates (including a wrist point), and image processing to identify a point on the forearm that is connected to the wrist point via a line parallel to the forearm. By combining the forearm orientation and the hand joint positions, the wrist angle can be estimated.
In the machine learning step, the You Only Look Once v2 algorithm was used to locate hand bounding boxes. Subsequently, we applied the OpenPose algorithm to identify joint coordinates.
In the image processing step, after rotating the image based on the angle between the wrist and middle finger metacarpophalangeal (MCP) joint, signal intensities in the Hue domain were extracted along several parallel lines intersecting the forearm at intervals below the wrist point. The offsets between these signals were used to identify the edge of the forearm, and therefore compute the forearm angle.
Finally, the angle between the forearm point, wrist point and index MCP was extracted as an estimate of wrist flexion/extension angle.
For performance evaluation, the dataset was manually annotated for forearm point and index finger MCP. We compared the resulting wrist flexion/extension angles and forearm orientations from annotated and estimated points.
The forearm orientation error was 15.1+/- 13.3 degrees and the wrist angle error was 14.2+/-15.3 degrees.
The method is the first attempt to extract wrist angles from egocentric images. The results show the efficacy of this approach. The algorithm can be used to incorporate wrist information into the analysis of hand function from egocentric video of individuals with SCI at home.