fimi - A Smart Mirror Application For Workouts With Form And Movement Correction Feedback

(Note: fimi is currently in development)

Motivation

Whenever you do regular workouts or physiotherapy exercises, you can quickly get injuries that last for a longer time, if not forever. The most important thing in workouts is your body form, but especially in Corona times or in times when no expert is available, you need to stay in training even without a trainer or a physiotherapist. It might help performing the workout in front of a mirror, but yet, you ideally need an expert to point you to incorrect body forms and to sharpen your awareness of your movements.

fimi is an application that aims to teach you to watch your body pose during workouts based on pose estimation and evaluation on a set of predefined rules per exercise. While doing workouts in front of the one-way mirror, the application can overlap your mirror image and notify you via on-screen and audio notifications. To lower the hardware requirements and therefore the costs, fimi needs to perform with a standard RGB webcam as the only sensory source.

The theory behind rule based evaluation

The rule based approach is based on the fact that there is a limited set of rules that exist for every workout exercise since the number of bones and their range of motion is as well limited. For each exercise there will be a different combination of rules relevant to be noticed. Before starting with this project, the experts of Velamed who have helped in the EXGAVINE project were consulted to check if it is feasible and reasonable to develop a rule based algorithm to evaluate the body poses. Currently, physiotherapist and trainers alike applying such rule sets to supervise trainees in their form and movement.

For evaluating such a system, a suitable movement needs to be selected that is complex enough to show the possibilities and limitations of it. The exercise selected for this task is a combination of a squat and a sideways arm abduction, shown in the GIF below:

Ten different rules were classified that need to be checked during the performance of the exercise:

Frontal aspects

• Head vertically in line
• Arm elevation symmetrical into horizontal line
• Elbow straight
• Wrist straight and in line with elbow
• Kneecap always in line with the feet, also during movement (preventing X/O legs)
• Feet slightly rotated outwards (depending on how wide the feet are, based on the hip bone)

Sagital aspects

• Neck slightly bend (natural 20 degree bending)
• Backbone straight
• Hip rotated up to max 100 degree
• Knee joint rotated to 90 degree

The technology

The application is based on the XNECT real time 3D pose estimation algorithm and is extended with a rule based bone evaluation algorithm.

The current 3D body pose of up to 10 people gets estimated by XNECT and transferred to the application via the WebSocket protocol. The application, written in C# and using the Unity backbone, is adjusting a 3D skeleton and evaluates each joint pose pair against the predefined rules. With that, it is possible to evaluate the poses of up to 10 people in real time in the same time. The application is coloring bones and displaying on-screen notifications in case of compromised body form. The flow diagram for the relationship between both algorithms is shown below:

The rules are defined in a YAML format and thus easy to extend and adjust. An example is visible here:

exercises:
  - type: squatArmRaise
    rules:
    - !rangeOfMotionRule
      notificationText: "Don't overstretch your knees"
      colorize: true
      priority: 10
      lowerThreshold: 0
      upperThreshold: 95
      bones:
        - LeftLowerLeg
        - LeftThigh
    - !verticallyRule
      notificationText: "Keep your head, neck and body vertical"
      colorize: true
      priority: 10
      tolerance: 10
      bones:
        - LowerBody
        - UpperBody
        - Neck
        - Head

Currently, seven different rule types are supported, such as angles, range of motion, bone symmetry or speed.

Open challenges

Using only a 2D image to estimate a 3D pose of a human body is for sure producing a lot of misplacements and noise. The aim is to use camera and skeleton calibration to make the estimations more stable and useful. This is currently in development.

Secondly, the user interface and user interaction need to be easy to use, practical and need to provide an effective way of notifying the user of compromised body form. In the same time, the application should be motivating. Currently, I experiment with multiple forms of user feedback, like coloring the affected bones, showing on-screen notifications and using gamification elements.

Third, the smart mirror hardware needs to be set up and built by selecting a one-way mirror panel, building the mirror frame and setting up the application so it overlaps the mirror image of the user.

Interested? Get in touch

If you have any questions, ideas or comments, just get in touch (in English or German, whatever you prefer). I’ll be happy to answer everything: 7reichel@inf…

• software
• smart mirror
• master's thesis