Auto-theme switch Figma Plugin
Let’s play a rhythm game with a DIY console for exercising!! Stay healthy but in a fun way!!!!!!!
Used tools: Tiny Motion Trainer, Machine Learning, P5.js, Solidworks, 3D Printer
01 inspiration
I randomly saw this another on youtube. It looks extremely fun to someone who is obsessed with music beat games. I thought it would be cool to have the microcontroller recognize your leg movements, then send OSC to Unreal to create some fun animation for satisfaction.
02 Brainstorm
03 Machine Learning
The Arduino Nano BLE 33 has a motion sensor inside for motion capture. I used Tiny Motion Trainer to be my machine learning helper. It cooperates with my Arduino and I can set the capturing threshold, numbers of samples, and delay between captures. The more dataset I can collect, the more accurate my motion capture will be.
But the first machine learning result was not ideal for detecting exercise motion. I think it is extremely important that the Arduino position is the same while training and playing, so I 3D printed a case for the console to stabilize the Arduino position to get the accurate XYZ. With the stable Arduino and more trained dataset, my motion capture model is good to go! It is also important to design the game motion according to the X-Y-Z axis of the microcontroller. Although I think increasing the number of samples can overcome the complexity of this issue. After having the model work, I start to see what would be a cool interface for this game.
04 Gaming interface
At first, I tried Unreal because I found a rhythm game plugin. However, the message from OSC is not able to send into the plugin as input, so I just switch to p5.js. Here is some image for the icon:
05 Interactive coding
I have created a class for motion in p5.js. The hardest part I encountered is how to assign the index to each motion. After learning a lot of js array, I was able to do it but I think there might be a cleaner way.
06 TO-do list
For the demo music, I just used my friend’s recommendation: WAP by Cardi B. Even speed, strong beats. In the future, I would like to use the p5 sound library to give it the ability to analyze whatever song so that people can exercise with the song they love. I might also import the entire project over unity for better UI & Interaction Design to make it more like a game. More motion need to be designed and added to include different muscle in the body. The hardware also needs to be redesigned to be more ergonomic.