“That little synthesizer had a function that allowed you to make your own sounds, and I became completely captivated by the possibilities,” he says.
Synthesizers were taking a more prominent place in pop music during the 1980s, adding to his fascination with the technology.
As he advanced through school, Smaragdis wanted to study music made with synthesizers. But this was not an educational path offered by the more traditional European conservatories at the time.
Instead, he enrolled at Berklee College of Music in Boston, into a program where he could explore the use of computers and synthesizers to shape and manipulate sound.
“With computers and synthesizers, I could make new sounds that have never been heard before,” he says.
At the same time, Smaragdis became immersed in jazz fusion. He declared bass as his major at Berklee since it would allow him more opportunities to play with jazz ensembles.
“I found the bass to be sonically more interesting than the guitar. It was a more unexplored domain, and I’ve always gravitated toward unexplored domains,” he says.
That interest in the unexplored carried over into his coursework. He and mentor Richard Boulanger, a professor of electronic production and design, created new tools to help people use computers to write, produce, and enhance music.
Through his studies at Berklee, Smaragdis embarked on a collaborative project with Professor Barry Vercoe (who passed away in 2025) at the MIT Media Lab to create software synthesizers for music creation. Building on that project, Smaragdis pursued graduate studies in Vercoe’s group, where he completed his master’s degree and PhD, followed by a postdoc.
“In the Media Lab, I got exposed more to the computer part of computer music. That’s where my attention shifted a bit. Now I really wanted to make computers that could understand sound,” he says.
Tuning in to unexplored domains
At that time, many students in the Media Lab were using signal processing techniques to help machines handle sound. Smaragdis chose a less explored approach — machine learning — to separate himself from the pack.
He found that training machine-learning models to understand sound offered advantages that couldn’t be attained with traditional signal processing techniques.
“Over time, it became really clear to me that we need to teach computers how to listen. We can’t just tell them how to listen. If anything, we don’t even know how we humans listen,” he says. “We don’t have a vocabulary to explain what it means to listen, or how to tell the difference between a cat’s meow and the sound of a violin.”
After finishing his postdoc, Smaragdis began a career at Mitsubishi Electric Research Labs, where he worked on artificial intelligence systems that could scan video soundtracks and understand what is happening on screen.
“What if you record an entire soccer game, but you only want to watch the highlights? In computer vision, it is very difficult to identify what a highlight is. But on the other hand, if you only listen to the sounds, then when the crowd goes crazy you would know that something interesting is happening,” he says.
That system was eventually incorporated into electronic devices, including a personal video recording system Mitsubishi was developing at the time, and into traffic monitoring systems and cell phones.
Smaragdis later joined Adobe, where he created the audio research team that built user-friendly tools to help artists, musicians, and producers manipulate sound without needing a lot of expertise.
“As we did that work, I kept coming up with deeper technical problems that needed to be resolved. I jumped into academia to address that itch,” he says.
Smaragdis joined the faculty at the University of Illinois at Urbana-Champaign, where he and his group incorporated the latest machine-learning techniques into audio-processing systems.
One of the technologies they developed utilized neural networks for separating individual sounds from audio mixtures. That project caught the attention of the team working on the 2021 documentary mini-series “The Beatles: Get Back.”
The production team enlisted Smaragdis to help clean up some archival Beatles recordings.
“That was a very different way of audio processing than what I was doing in academia. Now, I had to do something that sounded good but didn’t necessarily have to look good on paper. We didn’t have a formula we could optimize,” he says.
Every recording involved different challenges. For instance, in one clip John Lennon was playing music while Paul McCartney was horsing around, reading headlines from newspapers and making fun of George Harrison.
“We were trying to come up with a technological solution that could help us do everything, as opposed to addressing each problem on its own,” he says.
His contributions helped the team win an Emmy Award for sound editing for a nonfiction or reality program, in 2022.
A concerted effort
After 15 years at the University of Illinois, Smaragdis decided to join the faculty at MIT to tackle the thorny, interdisciplinary problems that were emerging in the field.
His group now focuses on two main areas. For one, they develop systems that make computers more useful tools for musicians.
One project in this area is enabling machines to understand how to listen to music, so they can provide constructive feedback to musicians.
“We want to enhance the toolbox a musician has. We don’t want to use computers to replace musicians,” he says.
On the other side, his group focuses on the fundamental tools scientists and artists alike can use to process, analyze, and manipulate audio.
