“My thinking was, what I do well is take on large, squirly projects that have complexity to them and — make things happen.”
It was just days after the neurodiversity hackathon at Microsoft Vancouver and Gregor Noriskin, Principal Software Engineer Lead for Microsoft News, was trying to decide which project he wanted to dive deeper into. The decision was hard. They all had merits, they all could make a difference, they were all going to be personal.
“The Neurodiversity Hackathon was a motivational turning point for me. My son got his official Autism Spectrum Disorder (ASD) diagnosis just days before it, and after working with a few of the teams there, I left thinking that I would throw myself into one project,” he recalls. “But, I wanted to do them all and explore the best possible outcomes for my son.”
Gregor reached out to a fellow hacker from the event, and together, they mapped out potential projects that could drive awareness of ASD and create safer spaces for those on the spectrum. Within weeks they’d chosen a direction and started the groundwork for their device, called NeuroPi.
“The goal of the project is to build a wearable device that can detect another person’s emotional state – where they might be angry, frustrated or otherwise signaling you in a way that is non-verbal,” explains Gregor. “This is something that an individual on the autism spectrum may miss.”
The prototype clips to a backpack or can be worn like jewelry. It has a camera, an off-the-shelf Raspberry Pi and a rumble motor cannibalized from an old Xbox controller for haptic feedback. Using Azure Cognitive Services, it’s their hope that the device will be able to detect emotion through micro-expressions and then deliver a rumble that informs the wearer.
A micro-expression is a brief, involuntary facial expression tied to the emotion being experienced. They happen fast, in as little as 1/25 of a second. As a non-verbal clue to reading a person or situation, they are an essential part of human interaction.
“We register micro-expressions subconsciously. It’s something we sense more than see and then we adjust our interaction accordingly,” explains Gregor. “But an individual on the spectrum may not notice or respond to them.”
The hope is that the device would capture them, identify them and then be able to increase the intensity of the rumble to match the intensity of the detected emotion.
“If you go broad, the things that can come out of it, will,” he notes. “For me, success would be the ability to measure anger, contempt, and disgust, because that’s when an individual with ASD may be most vulnerable to misreading a situation and experiencing a consequence.
While the global hackathon provides much needed time for experimentation, regular visits to The Garage have been a critical piece in Gregor’s early work on NeuroPi. With Stacey’s help and access to hardware, he learned how to design and build the parts, including a 3D-printed enclosure and electronic circuits to control the rumble motors. He’s also had the opportunity to expand his software engineering skills.
“I’ve been writing code for many years, and used many languages and APIs, but when it comes to data science and machine learning, my experience was limited. We went with the Raspberry Pi running Raspbian, so I’m learning that, and we’re using Python, OpenCV and Cognitive Services,” Gregor explains. “This project has also provided me with a platform to engage a number of people in the company and in the community.”
This project is the beginning of a lot of different things, including the creation of a brain trust of people who are passionate, interested and motivated. As Gregor dug into it, he realized how many people care about advancing awareness for neurodiversity and finding solutions for those with ASD. It’s something Gregor sees as a defining element in Microsoft’s DNA.
“Microsoft is a good place to do this because we live our values. Despite its size, the company has an ethical and moral backbone that is maintained,” he notes. “I’ve seen people come here and it changes them, we have a corporate culture that makes us better human beings.”
While the hackathon wraps up this week, NeuroPi will continue. There’s more research to be done and so much more is possible.
“As I’ve gotten into this neurodiversity space, I’ve become more aware that a lot of kids today who are in every other way ‘typical’ do not necessarily understand emotionality. EQ appears to be diminishing, and there are a lot of questions around why that is,” notes Gregor. “Is it an environmental thing? Is it because they are spending more time interacting through devices instead of directly with each other? We don’t know.”
At the end of the day, his goal is awareness — across industry, academics, service providers and for parents like him who’s children receive a complicated diagnosis and who worry about the future.
“The hackathon is sort of a coming out for my project, where I can stand next to this little device and say, this is my story. Frankly, it’s what I’d like to do with the rest of my career.”