top of page


MIT Hackathon March 2022

A new way to "coasting" at work, have fun! Built with Spectacles by Snap, Javascript, Lens studio, Python, PyTorch

Ft. Violet Han, Tim Wang, Yiming Zhang


The two of us (Jiyun and Violet) met while interning at OnePlus last summer, our internship experiences were unique and enjoyable because we were able to get to know each other and share enjoyable moments together at the office: Whether skateboarding in the office to get water, or bringing flowers and sending them to workers, or playing a few rounds of arcade basketball... We wanted to leverage the unique power of augmented reality technology to bring back our summer of fun and joy.

We can't help to think, what if everybody's workplace could be this fun, no matter what physical fun elements are already present at your workplace or not, or if you are someone who is a little shy to walk away for a little while... We want to bring simple fun relaxations to everyone everywhere.


What it does

The user starts with drawing a very simple doodle. Our robustly trained machine learning model will recognize what the user doodled, and bring the doodle to life! For example, if the user drew a basketball, a basketball and a frame will appear for you to shoot. The lens will hand-track your gesture for different operations to control the ball. In this special case, there are two modes of the ball where you can either challenge yourself to aim the basket or shoot whatever but always gain satisfaction.


How we built it

We first brainstormed for different kinds of activities you can do based on your doodles. After deciding to choose basketball as our example, we first try out with the Magic Leap. However, the setup and the size of Magic Leap are not convenient enough for our user case. We then switch to Spectacles. Violet and Yiming focused on training the ML model for recognizing the doodle and sending back labels with Python. Jiyun and Yufan focused on developing the interaction in the Lens Studio with Javascript.

Challenges we ran into

After training and validating a working image-recognition model specifically for doodles, it was hard to work with it as a custom MLComponent in Lens Studio: We used the camera captured “deviceCameraTexture” as input for the model at each frame. The input was a “BlackBox” for us, we knew very little about it. We weren’t able to obtain critical information of the input (e.g. its dimension, its raw pixel values, different distribution of image statistics due to the wide-angle FOV and color adjustments…), which led to non-ideal inputs for our model and non-ideal outputs. Perhaps with some more time and guidance, this issue could be resolved. We weren’t familiar with Lens Studio nor JavaScript, which led to longer development time and less-satisfactory results. We spend one-and-a-half day trying to set up and develop for MagicLeap with Unity and then switched to Snap.

Accomplishments that we're proud of

We are all new to both Magic Leap and Spectacles. But we are not afraid of learning them even they have a frightening learning curve at the beginning, we still are able to get our hands on them. We are able to quickly prototype the small games based on the template, we are also implementing ML in the Lens Studio without many documentation supports.

What we learned

We have learned what will be the general experience for using AR devices, what are the basics, and what we can implement in the future. With the device in our hands, it is easy to test the user flow and find the most reasonable user experience during the development stage.

What's next for Minions@Work

We want to add audio and visual effects for more user satisfaction. Implementing the ML model for detecting doodles. We only developed one game that matches our doodle in the 2 days after we switched devices. We would like to develop more game modes and use ML for them that eventually can cover most of the doodles in the dataset. The categories of interactions are listed below:

  • Fun interactions style:

    • Easy small games with Interact with the hand gesture

    • E.g doodles: Basketball/Skateboard

  • Creative display interactions

    • Help creators relax and open their imagination

    • E.g doodles: Castle/Fireworks

  • Multi-users game interaction

    • Get your colleagues involved too!!!

    • E.g doodles: Soccer court, then use pens with colleagues to play

bottom of page