Building a Genome from a Picture aka The Scanning Room

Time to explain my piece that is at ZKM

So you want to know what is in your genome, maybe like me, you spent 20 minutes spitting into a tube and sent it off to a company like 23andMe and have your genome arrayed for Single Nucleotide Polymorphisms (SNP but never snip. I tend to not pronounce acronyms unless they are phonetic otherwise it prevents people from looking them up on the internet). What 23andMe does is match these SNPs to studies that have been performed that associate the DNA sequence to a specific phenotype in a human being. They tell you that you are caucasian and have a probability of having wet earwax and blue hair.

I had an idea, what if we could create a noninvasive way to composite and predict a genome, like predicting a protein sequence by its structure. What if we could take phenotypic features from a picture of someone and composite a genome? What if just from a picture of someone you could tell if they had a gene associated with a specific disease or disease risk? Did you know that some genes(alleles) occur in greater than 90% of people with certain phenotypic traits?


This project is speculative. Our current understanding of our genomes only allows major traits to be distinguished. No one has studied things such as the association of chin to lip length on genes that are associated with bad teeth. This will happen though. Eventually our understanding of our genomes will allow one to learn alot about a person from only a picture. Data Science in this area is very primitive at the moment. With most genomes studied there are not pictures associated with them or much else.

From what I can tell no one has publicly(looking at you governments) tried to determine genetics from a photo. To me this project was an interesting idea to see how invasive one could be with just a picture. To maybe look a little bit at what the future holds. What if instead of giving a company your love interests DNA to sequence, like in GATTICA, you just upload a picture to a webserver?



This was the idea and this is the piece.

How it works:

It starts with a picture.

I wrote all the code in C++, which I consider an awful programming language. The program is multithreaded and uses the OpenCV library. It uses a webcam to try and find a human face in the streaming video it is constantly taking. Then comes the machine learning, I used a number of different machine learning algorithms that are built into the OpenCV API and ran them on a database of somewhere between 500-1000 faces that were sorted by sex and ancestry(race). Combining the different algorithms I performed "boosting" to create a meta-algorithm of sorts, which really helped (I have not statistically quantified by how much though). Because I wanted my training set to be reliable I needed a way to build this database of training images as I could not find a dataset like this that existed and the NSA wouldn't answer my emails. I ended up scraping OkCupid and using people's self-identified race and sex to build my training dataset. I also hand-curated these images.

Side Note:
If you ever post a picture on a dating website please look forward and directly at the camera with no sunglasses or hat on because that might make the day of a data scientist. Seriously, I had a program that did face and eye detection and cropped the photos and only about 20%-30% were usable!

The interesting thing is that machines see more than you or I. We as humans are limited usually, to what we have been trained to look for. Machine learning allows computers to explore associations that humans cannot perform. With decent lighting the sex and race detection are actually pretty good!

Sadly, since my datasets are pretty big and using different algorithms for comparison on a semi-old laptop it takes about 20 seconds for the detection to run. I needed a faster method otherwise people who interacted with it would become bored. I found servers online that provide APIs to do the same thing but I needed to sacrifice the freedom to have any of the races I wanted and to create an expanding dataset. Sucks, but it runs in about 2 seconds instead of 20. And it is also good that I have both a version that can run without the internet(original version) and the one with the online API.

Next, the program starts breaking down different features of the face, eye color, hair color, skin color. Color is an interesting thing. On computers the way we define them most frequently is through Red Green Blue(RGB) values. But really what is Red? RGB: 255 0 0? Of course! RGB: 255 100 100? Maybe??? RGB: 255 255 255? Definitely not. even genetic studies reference "blue" eyes, "brown" hair when their blue could include blue green and their brown could also be blackish. So when I was going to associate an SNP with blue eyes, I first needed to figure out what RGB values I thought blue was. That was interesting. I kind of just typed in values and created cut-offs based on what I saw, so pretty arbitrary...

Then I built my database of alleles, that were associated with traits such as race, sex, skin color, &c.

The program also talks to you and shows you videos and your genes.

So from the start:

You walk in and see a large display on the wall with a live feed. It detects your face, it uses machine learning to identify traits about you and tells you about your genome, showing you DNA sequences of the alleles you are predicted to have based on these traits. While it is doing this the program is talking to you generatively by using Google Translate English to English as a text-to-speech engine(little trick I picked up that works GREAT!).

Code for the version using the Online API
https://drive.google.com/folderview?id=0B_R75gIJvkFUczBxNXk3ZGtWRDg&usp=sharing


A bunch of face images cropped, aligned and sorted by self-identified sex (male. female) and self-identified race (caucasian, black, asian)
https://drive.google.com/folderview?id=0B_R75gIJvkFUdVAxNE45NXhSXzA&usp=sharing

 

This video plays when your presence is 

detected in the Scanning Room by the software

What is Art and Science

Lately I have been making the claim that technology revolutions cause changes in culture and those changes are usually integrated into our Art. This is evidenced by Digital Art, Robotic Art and others. Science and Biotechnology are becoming more integrated with our culture and with it brings Science.Art. Sadly, many scientist.artists create pieces that are neither Science or Art. Maybe it is a bit egotistical but I feel a bit shamed to be placed in the same category as these people. The thing is that I never wanted to be placed in the category of Artist. It is not till now when people want to curate my works in museums that I really need to ask myself "Am I an Artist?"

As Kanye said "I ask cause I'm not sure, does anyone make real shit anymore?". Many Science.Art pieces lack either a Scientific part or an Artistic part and to me end up being something that is difficult to appreciate. I often see artists claiming Science but their works are very missleading and are not actually what they say they are or believe them to be. And Scienctists claiming Art when their works are neither. I will give a few as reference.

Christina Agapakis puts bacteria on plates mostly (http://agapakis.com/art.html). This is really not Science. As for it's Artistic merit some could argue for it but this kind of work has been done many many times before in slightly different contexts ala Steve Kurtz and many others. The depth and skill seems lacking and way behind on the times. Maybe it's interesting and speculative nature would have been there 10 or 20 years ago but not now, not like this.

Jalila Essaidi told the world she made "bullet-proof skin". What I thought and imagined was that she trangenically expressed proteins in epidermal cells that allowed them to be bullet-proof. I was interested and I emailed her. What she actually did was take a bullet-proof material and then put human(mammalian?) cells on it and call it skin. (http://jalilaessaidi.com/). She didn't respond to my second email when I asked her the specifics of how the project was done and why she thought it was skin.

Ginger Dosier claims she is an Architect and Scientist. From what it appears, she uses Scientific work others have done and claims it as her own (http://vergelabs.com/). I tried to contact her about her work but she did not respond. It seems the people she impresses don't know enough about biocementation and bioclogging to know better. That is a field I have been starting to work in at NASA. Using engineered bacteria and proteins to harden soils and regoliths. (If you read this Ginger and have performed some new work in biocementation please contact me and let me know what it is. I am sure Scientists would appreciate knowing your advances) 

There are many many more but you get the picture. Maybe people see this as hating, I see it as having a high standard. Having respect for the quality of the work one puts out.


What is Art? People always say that this is a difficult question to answer but I have never thought so. I think that people can argue Artistic Merit of certain works but I think what makes something Art is that it is done by an Artist. Just as Science is done by a Scientist. So who is an Artist? If anything can be Art then what is the need for Artists? Why can't we all just stick petri plates on a wall and be done with it? An Artist to me is someone who is trained and.or dedicated to their craft. Someone who can create works that require "skill" and I think that is the word that differentiates Artists from non-Artists, skill. This is what differentiates a Scientist, a Footballer, a Medical Doctor from everyone else, skill. Skill can be somewhat quantified. If someone claiming to be a Scientist doesn't know Science then they are not. If someone claiming to be an Artist has never been dedicated to the craft then they are not.

I often refrain from calling myself an Artist directly. My work is mostly Science and Engineering with about 10% Art. Lately it has started to travel into the realm of speculative Science and Engineering and that is where the 10% Art comes in.  I think this speculative  but functional nature of some of my works, such as The Chromochord, (What would a future be like when organics are components in electrically engineered projects?) can perhaps be viewed as Artistic. Both the Science and the Art are original.

Look at works of Artists I know, Micah Zayner (https://www.facebook.com/micahjzayner) and Zachary Williams (https://www.facebook.com/pages/Zachary-J-Williams/284392201665097). Their craft is skilled, well thought out and has depth. Just because you go out with your friends and pay $30 to drink wine and paint pictures doesn't make you an Artist or your work a piece of Art. It is some oil or acrylic based colors dried on some material. Art takes dedication and skill.

Scientists can tell the difference between Science and science. Science takes skill, is well thought out and has depth, science does not.


For me Science.Art and Technology.Art require that the Artist add something original to both Science and Art. Maybe these are very stringent requirements and it takes dedication and skill to create a piece of that nature. To me

THAT IS THE POINT OF ART!

Stochastic Labs and The Infinity Engine

I haven't talked much about The Infinity Engine or my residency and grant from Stochastic Labs. In July of 2014 I heard about this mysterious organization known as "Stochastic Labs" that had no website or information about what it was there was just a form to fill out for a grant for ~$20,000. I posted my application on my blog (http://doitourselfscience.blogspot.com/2014/07/stochastic-labs-application.html). They contacted and said that they wanted to fund me, but not me alone, they wanted me to work with someone else and they had this person in mind and her name was Lynn Hershman the director of Strange Culture. Lynn was interested in doing some works around genetic engineering and so was I. I was pretty skeptical at first because I didn't think someone who works in media would have a good idea of how to work on Science projects but I agreed anyway, I mean they were giving me $10k.

I found out Stochastic Labs was in a mansion in Berkeley and it is a nice mansion. Stochastic Labs is funded by The Minerva Foundation. The residency started on August 15th and ended on October 15th. It was short but it was pretty intense. We had dinners there every Monday and gatherings on random days. Alot of cool and interesting people were invited to the Monday dinners. The funny part was I never knew who they were. I would be talking to someone and find out they were an Actor or a Nobel Prize winner or director of a movie I saw.

I would head up to Berkeley after work at NASA and would stay till traffic abated or I ran out of Red Bull around 9 or 10PM and give fellow grantee Greg Leppert a ride home.

Anyways, I had lots of fun up there and worked hard to create a program or "device" I guess which takes a picture of someone and attempts to probabilistically reverse engineer their genome from their physical traits. It also talks to you and tells you what it is doing and what it found. The current iteration is pretty basic it uses facial features to identify ancestry, sex, skin color, eye color, hair color, &c. and maps those to the genes that have been identified to be associated with those traits. Eventually I want it to construct pseudo genomes using an ancestry base genome and editing in traits it identifies from each person it captures.

This will be part of an installation that will be on display starting at ZKM in Karlsruhe Germany on Dec. 12th to the end of March and then will visit other places in the world.