UW computer scientists are working on a way for you to talk to the dead

Five years ago I sat down with my Grandma Sue and a tape recorder, and interviewed her for two hours. I asked her to tell me stories of her childhood in New York City, her marriage, anything about her life. I learned that she got married at 16 years old, in a jail, along with several other young women and their soldier fiancés. They later divorced and, on tape, she advised me not to marry a bum.

Unfortunately, a few weeks later I accidentally deleted the recording. And before I could schedule another visit to re-record, she died. Now, the only recording I have of Grandma Sue’s thick, New York accent is a five second video on an old, out-of-service cell phone.

What if you could have one more conversation with someone who passed away? Or many conversations. Would you do it? Eventually, this may be possible. Computer scientists at the University of Washington are working on bringing photos and video to life.

“In the end we hope to get something like a 3-D model on a screen that you can talk to, that will try and respond to your questions the way they would do,” says UW graduate student Supasorn Sawajanakorn. “That’s the end goal of this. It’s going to take more work, a lot more work, to get to that point.”

Sawajanakorn helped develop an algorithm that can reconstruct 3-D models of a person from photos and video and bring them to life using a puppeteering application. Right now they’re experimenting with celebrities, because there is so much video available. Sawajanakorn says President Obama’s footage was especially ideal because all of the shots of him speaking are head-on.

“I can replace his voice with my voice and then have his lips move the way he should speak.”

So it might not be possible to do something like this with my grandma, because we don’t have enough photos and videos to make the 3-D model.

“But imagine if you have a [recorded] Skype conversation. You basically look into the camera and you do a lot of talking and that could be the valuable data for reconstructing normal people too.”

Right now, Sawajanakorn can make President Obama and Tom Hanks move their lips in synch with a recorded voice. But in order to have a back and forth conversation with one of these models, and ask questions, there has to be an artificial intelligence component.

“I think it has to combine with the AI part. You might want them to say something they’ve never said before. Maybe you want to ask them personal questions and you have to maybe learn what they know in their past life and then transfer that over. I think you have to also solve that AI problem.”

The scientists have already been working on this for the past five years, and Sawajanakorn thinks the project needs at least another five to work properly. The idea is that maybe one day you could put on a pair of augmented reality glasses and see the 3-D model of your dead grandma, or a celebrity you’d never have the opportunity to meet, and have a conversation as if they’re right there.

But the real question is, would you even want to do this? Is speaking to an artificially intelligent version of someone who passed away healing and cathartic or just plain creepy?