Humans have forty-three muscles in their face that can be combined to create 10,000 unique expressions. Imagine programming a computer to interpret all of them. And, when you’re done with that, add in the other factors that effect social interaction – body language, culture, gender, and more.

Impossible?

That’s a word that doesn’t compute for Dr. Ehsan Hoque ’04, a Penn State Behrend Computer Engineering graduate who was recently featured in MIT Technology Review as one of the top 35 innovators under the age of 35. He was recognized for developing two computer technologies— MACH (My Automated Conversation coach) and ROCspeak—that train people to excel in social settings.

Dr. Ehsan Hoque ’04, a Penn State Behrend Computer Engineering graduate, innovator and assistant professor of computer science at the University of Rochester. Photo credit: Adam Fenster

What doesn’t kill you makes you stronger

Hoque has been turning “impossible” situations into opportunities since his undergraduate days at Behrend.

“A scheduling mix-up required me to take some courses out of order and without having had the recommended prerequisites,” he said. “I really struggled, but it was a good thing because it made me resilient.”

His resiliency was put to the test in his final year at Behrend when his team took on a senior design project that they weren’t sure could be done. Not only did the team complete the project—a robot that could see, hear, and recognize people—but they also won the “Best Design” award.

So when Hoque was a doctoral student at MIT presenting his work on a computer program capable of gauging a user’s mood, and was challenged to expand it, he didn’t balk.

Challenge accepted

“After demonstrating my research at a workshop, a gentleman approached me and said he was very awkward in social situations and wondered if my technology could be used to create some sort of automated program that he could use at home to practice making eye contact and improve his social skills,” Hoque recalled.

As the primary caretaker of his brother who has Down Syndrome, Hoque had witnessed first-hand how difficult social interactions can be for some people. He also suspected that social stigma and shame kept many from seeking help.

“A computer, however, is completely objective, standardized, and non-judgmental,” he said. “It’s a safe place for people with social anxiety or awkwardness to practice and learn.”

Hoque, now an assistant professor of computer science at the University of Rochester, started working on MACH, a system for people to practice social interactions in face-to-face scenarios with a 3D character that can see, hear, and make its own decisions in real time.

Conversations with computers

In MACH, a virtual businesswoman has been programed to recognize the user’s expressions and statements. She can also nod, smile, and even ask questions. At the end of the conversation, the businesswoman gives users feedback about their interpersonal performance, including body language, intonation, and eye contact.

The program went live in 2015 and, to date, more than 20,000 people have used it to improve their social skills. Users can choose to share their results for research purposes and more than half have done so. Hoque uses the data to further improve the program.

“Building the original platform was easy, but then you have to add in all the insights,” he said. “That never really ends. It is an ongoing, evolving process.”

Hoque also designed a pared-down mobile version, Rocspeak, free for anyone with Internet access to use. There’s no animated character; instead, it records video and sends you a report on various aspects of your performance, such as speed of speech, pitch, intensity of smiles, and how often you repeat wrds.

Applied science

Applications for MACH and Rocspeak have gone far beyond the original intended audience. Hoque said his tool has been used by all kinds of people, including customer service representatives, nervous students with looming classroom presentations, and individuals simply preparing for a big date or a job interview. It can also help those who have social anxiety or challenges, such as those with Asperger’s.

Hoque, who earned a master’s degree from the University of Memphis and a Ph.D. from MIT, said he enjoys being able to use his skills to make a real-world impact.

“I have these advanced degrees and technical skills and, if I can use those to solve problems for people, then it is knowledge and time well spent,” he said.

Blast from the past

Check out Hoque’s Penn State Behrend senior project presentation: VisionPSU, a human interactive robot (below).

Also, peruse these vintage photos, where you may recognize some School of Engineering faculty members, including Dr. Ralph Ford, now Penn State Behrend’s chancellor.