Machine Learning Piscine Leads to Development of the Neurosleeve
Daniel Cordova and Jose Cruz Y Celis started the 42 Silicon Valley program in September 2018. Only seven months later, they did a project in the machine learning piscine that changed their understanding of how signals from the brain can go directly to a computer via the decoding of neural muscle activity with the help of machine learning and signal processing. The machine learning piscine is overseen by the head of the Robotics Lab, Dan Goncharov. In three weeks, after learning the basics in the machine learning piscine, Daniel and Jose created a device called the Neurosleeve that further explores brain machine interfaces (BMI).
With the Neurosleeve, Daniel and Jose were able to measure the sEMG (Surface electromyography) by placing eight electrodes positioned on the forearm, giving them four signal channels. They created a data set and trained the neural network to recognize their hand gestures so they can use them as commands into a computer. Daniel and Jose trained their Neural Net and obtained a 92% accuracy on never-before-seen data. Jose documented their research in a Medium article where he shared, “After connecting our streaming signals from the forearm we were to accurately send commands to a computer, this means it can be any device that operates with controlled human input. This includes drones, autonomous cars, games, AR environments, robotic arms, etc.”
Brain Electrical Activity Research and the Exploration of Brain Machine Interfaces
This type of research project also means getting a few steps closer towards the goal of creating an entire brain machine interface. According to Brain Vision UK, “A brain computer interface, also known as mind-machine interface, is a direct communication interface between an external device and the brain, bypassing the need for an embodiment. The signal directly goes from the brain to the computer, rather than going from the brain through the neuromuscular system to the finger on a mouse.”
Jose shared that centuries-old research of the electrical activity of the human brain is related to the work that is being done on brain machine interfaces today. Brain Vision UK elaborates, “Hans Berger’s innovation in the field of human brain research and its electrical activity has a close connection with the discovery of brain computer interfaces. Berger is credited with the development of electroencephalography, which was a major breakthrough for humans and helped researchers record human brain activity – the electroencephalogram (EEG). This was certainly a major discovery in human brain mapping, which made it possible to detect brain diseases. Richard Canton’s 1875’s discovery of electrical signals in animal brains was an inspiration for Berger.”
Decoding Signals from the Forearm
Despite early research concentrating on the brain, Jose explains in his Medium article how there are promising applications in decoding signals from the forearm, “The forearms have the highest density of neurons outside of the brain or spinal cord, it’s the pipe which carries the signals going to the hand…Given the high neural bandwidth and the advantage of having no invasiveness into the human body, the forearm appears to be a promising option for the first stages of BMI development. Applications would include rehabilitation for amputees, VR/AR experiences and a more flowy/natural interaction with our increasingly connected and intelligent environment.”
We sat down with the Neurosleeve team to learn more about their project:
What was your Experience like in the Machine Learning Piscine?
Daniel: We started from scratch because we didn’t know what machine learning was. We got familiar with neural nets and got to learn how to apply them to real problems. There are different techniques for machine learning and we focused mostly on neural nets. Before I came to 42 I attended college for one year and studied industrial engineering. Despite that experience, I didn’t code before I came to 42. A friend visited 42 Silicon Valley and encouraged me to think about going. I think it is one of the best things that could have happened to me.
Jose: We really liked it. I believe we are at the golden age of machine learning applications. But we are still missing a few key ideas to really understand intelligence. Before 42 I completed a college degree in sustainable development engineering. This was a combination of electrical, chemical and mechanical engineering. I didn’t know how to code before I came to 42 either. There was a project I was working on where I wanted to contribute to software development. I wasn’t able to, so I looked into different programs.
What is the Purpose of Your Project?
Daniel: The purpose is to give better input from the human to the computer.
Jose: Basically it is trying to get closer to a more natural way of interacting with computers and harnessing our neural potential. So we have a very asymmetrical input and output of information and we want to be able to tap into that neural output. We can become better adapted to the future we are creating when we can keep ourselves relevant and be able to manage it better.
How Does the Neurosleeve Work?
Daniel: We measured the sEMG in four spots of the forearm. Essentially the micro voltage of four of the main muscles that move our hand and wrist. Then using the neural network, we classified that into different positions that we pre-selected.
Jose: We created a data set and from that data set we trained the neural net. After that, the Neurosleeve is able to recognize our hand gestures and we can use that as commands into the computer.
What Part of the Project Did You Work On?
Jose: It was very symbiotic. We were working on it sequentially until the last part when we divided some of the work. We were feeding on each other’s research in the beginning, and we were both implementing different models so we could do different experiments.
Daniel: We took different approaches to pre-process the signal. We built different models based on different ways of pre-processing the signals.
What was the most Difficult Aspect?
Daniel: Getting our signals into the computer. We know what to do when the signals come into the computer, but it was harder to get signals from the device into the computer.
Jose: The buildup and the skill acquisition was the hardest part. Having the amount of knowledge and skills necessary to implement the project was the hardest. A key aspect was really having a good foundation and good research skills on which to implement on.
What did you Enjoy Most About Your Project?
Daniel: Seeing the whole field that machine learning can be applied to such as images, natural language processing, simple classification, and regression. It is something that is very applicable to today’s market. It was really satisfying for me, making my first true ML project, which was the Neurosleeve, to work. We had some “vanilla” projects that are essential to building your knowledge on how to do machine learning. But I didn’t enjoy that as much as I enjoyed working on the Neurosleeve.
Jose: I think learning about the whole field is fascinating because it also shines a light and gives you an introduction into how we learn. It is semi-inspired by how the brain works. We are teaching computers how to see, how to hear, and I think that is pretty cool. Of course, implementing an end-to-end project such as the Neurosleeve in real life and getting good results was pretty cool. We were able to work closely with the state-of-the-art tools of the field and really understand the project.
What did you Learn from this Project?
Daniel: We mastered Python and we improved our teamwork. I didn’t have a lot of experience working on projects as a team. It was also about learning how to learn. We needed to acquire the ability to go through recent research and define what was useful or not and get the best of it.
Jose: A cool thing I learned is, a topic can seem super intimidating at first and so out there. But if you just start and break it down and go little by little putting in the effort every day, you will reach it. Everything is doable. At first, all of this seemed like science fiction, or super out there only done by PhDs. But then you start getting into it and understanding it and breaking it into smaller parts. And just having a good environment where you can learn, share, and test yourself. And you end up learning the cutting-edge stuff. A key aspect is really being curious and certain that machine learning is something that you are interested and passionate about and want to do no matter what.
What did you Learn from the Machine Learning Piscine?
Daniel: We learned a lot of theory. When you do it on your own you don’t grasp it. In the Robo Lab, we took the time to understand some of the theory so you can take full advantage of this concept. You know what machine learning can do and what it can’t. We got stuck many times during the piscine. But when you get stuck you basically need to take a step back and break down the problem that is preventing you from moving forward. After that, you see everything is doable.
Jose: Learning how to learn from cutting-edge scientific research. This is different from your typical learning because the information is pretty recent. So that was really cool. I guess really getting a better sense of the math in machine learning. Learning all of the different libraries and all the things it can be applied to. Python, and all of the basic libraries, and then tensorflow, training models on the cloud, cloud computing. Learning about all the resources out there for keeping up to date in this field. We also learned how valuable, you keep on experiencing this at 42, but especially in this piscine, how valuable it is to learn with someone. To have a tight exchange of learning and testing between your peers really makes a difference.
What Future do you see for the Neurosleeve?
Daniel: We need to improve the hardware to improve the results. If we can improve the hardware and apply different techniques that we learned, it can extend into VR or prosthetics. Those are the main two fields in which deep coding neural muscle activity is being used.
Jose: I think it can definitely be built upon later. We talked about the different possibilities it can go towards. We are using 4 channels for the resolution we are getting for model neurons. There is still a lot of potentials that we can tap into. We have about 15,000 motor neurons in the forearm to control 14 muscles, that translates to 14 outputs. So instead of 14 outputs, in theory, we could have 15,000 outputs, that would be the dream. Tapping into all of this potential, so instead of moving 14 muscle,s we can utilize the 15,000 motor neurons in VR. In the immediate term instead of discrete commands, we are aiming to have a continuous estimation of the hand position.
Meet the Team
Name: Daniel Cordova
Hometown: Toluca, Mexico
Interests: I am passionate about machine learning and football (soccer).
Dream Job/Career Interests: I am interested in building my own enterprise.
Name: Jose Cruz Y Celis
Hometown: Mexico City, Mexico
Interests: Mostly interested in neurotechnology at the moment. I like sports such as surfing and climbing.
Dream Job/Career Interests: It is not completely clear but contributing to the field of brain computer interfaces.
Watch the Neurosleeve in Action!
published by Stacey Faucett – April 26, 2019