After his win, I followed up with Henry to learn more about his project. To explain his idea, his background, and his thoughts about Goodwall, Henry kindly agreed to do an interview with me.<\/p>\n\n\n\n

Here\u2019s the transcript of our interview, with my questions in bold:<\/p>\n\n\n\n

Henry, congratulations again on your win! Before we talk about the award, could you tell me a bit more about yourself and the part of Switzerland you come from?<\/strong><\/p>\n\n\n\n

Of course. I am from both Switzerland and the United States and while I haven’t always lived in Geneva, I was born here in 2003. When I was seven I moved to Shanghai, China where I lived for seven more years. I think that I was incredibly lucky to have this opportunity to experience a different part of the world and for me, it is probably one of the things that will influence me the most for the rest of my life. <\/p>\n\n\n\n

The diversity of cultures in the expatriate community there definitely affected the way I approach and view the world. It is also there that I became interested in technology thanks to my math and computer science teacher Mr. Cressey. When I moved back to Switzerland a lot changed. It was nice to be able to see a different part of the world as most of my previous travels were in Asia and Australia but I also had to set out on my own with programming since I didn’t know anyone here who could help me – this has led me to be more active in communities such as GitHub and Stack Exchange as well as on my website. <\/p>\n\n\n\n

In Geneva, which is in the south-western, French-speaking part of Switzerland, I have met some really great people as well – I’ve got it all from an actor\/doctor to physicists – and it’s nice to live by the lake. Being the city with the second-highest population in the country, it is always quite busy around here so it is just as enjoyable to drive a few hours to the mountains to get a break from the urban life which wasn’t as easy to do in China where everything is much further apart.<\/p>\n\n\n\n

Okay, so let’s get to your pitch – quite ambitious and impressive, I might add! You call it the “Keyboard for the Disabled,” and you describe it as a solution using “a camera with simple #AI motion tracking to locate a residual limb in an image which can then allow a person to interact with a gesture keyboard (like on your phone). Pack this into a small USB device and anyone missing an arm could input keystrokes into their computer as easily as we do. This is much cheaper and therefore more accessible than a full prosthetic limb and could allow these people to obtain jobs that were previously unreachable due to their technical requirements.” <\/strong><\/p>\n\n\n\n

Forgive me for my ignorance, but could you perhaps go into more detail on how it works? Me personally, I don’t quite understand the “residual limb,” does that imply eye-tracking, for example?<\/strong><\/p>\n\n\n\n

Well, a “residual limb” is the term for the bit that is left past a healthy joint after an amputation. It is also referred to as a stump. My idea was inspired primarily by two things. A few years ago, I did some volunteer work with an organization called Handivoile that took people with various mental and physical disabilities sailing on lake Geneva. One of the other volunteers there was a retired Swiss Paralympic team member who had no legs or arms and I was thinking to myself that if this guy, who had plenty of character and was clearly very generous, could easily interact with technology, he could probably achieve a lot more than most people on the internet. <\/p>\n\n\n\n

The second thing was a project that I worked on. I don’t know if you have seen this video by SpaceX in which Elon Musk demonstrates their internal 3D modeling software which allows users to create parts using just their hands in VR. To me, this was awesome. I was attempting to recreate the hand tracking technology and add it to my 3D graphics library to produce something similar when I had the idea to combine these two experiences. <\/p>\n\n\n\n

The theory was that a residual limb – specifically an arm – could be used as a sort of pointer if we could locate the shoulder joint as well as the end of the limb in an image. Once this was done, the person could perform various gestures that the computer would track and then interpret. My mind then went to the keyboard on my phone, which allows you to move your finger in a continuous motion over the characters to spell out a word and so this is what I decided to use the limb tracking for. The software would allow people to draw patterns that correspond to words in the air with their arms in the same way I do with my finger on my phone screen.<\/p>\n\n\n\n

With your 2,000 CHF prize, do you think you’ll be able to turn this into a viable product?<\/strong><\/p>\n\n\n\n

The cash prize should definitely allow me to create a complete product by experimenting with better hardware parts, higher performance cloud computing to train the neural network, and better data acquisition. One of the things that I have to look into, however, is distribution, which I think would be best done by getting pre-orders from organizations such as support programs or hospitals as this might require more capital.<\/p>\n\n\n\n

You’re using the Raspberry Pi, one of the greatest gadgets of this past decade, especially for those without extensive financial resources. In case there are any tech geeks reading this, could you tell us your setup (e.g., Pi model, peripherals, programs)?<\/strong><\/p>\n\n\n\n

I hate to disappoint but my setup isn’t all that interesting. I have a Pi running Ubuntu server edition. This is one of my favorite operating systems because it makes Linux so incredibly easy to set up, use, and maintain. Then I don’t have any real peripherals, just a Logitech webcam as I am working with items I already had at hand. <\/p>\n\n\n\n

Of course, I plan to change this in the near future, I think that for a first version, the Pi is quite handy although a proper camera and 3D printed case to accommodate that are necessary improvements. I have also looked into Nvidia’s Jetson boards because of their hardware acceleration.<\/p>\n\n\n\n

On the software and AI side of your project, could you also give us a brief description of the program(s) and platforms you’re using?<\/strong><\/p>\n\n\n\n

So, I am currently working with Python3 as it is the language that I am most familiar with. To facilitate the development of the neural network, I use the frameworks Keras and Tensorflow. My model is based on ResNet, for which the original paper was published in 2015, and is inspired by the work done in a 2018 paper called “GANerated Hands” in which the researchers performed 3D hand tracking with a single camera.<\/p>\n\n\n\n