When design and neural networks mix the results are both practical and innovative.

Over the Spring 2019 semester, Brown University professor Thomas Serre and Clement Valla, an associate professor at Rhode Island School of Design (RISD), collaborated on a series of workshops that introduced Valla’s students to artificial intelligence.

The workshops culminated in several projects that used neural networks as design tools, and students said they gained an understanding of how to integrate artificial intelligence with art and design.

“There are so many resources to develop and train machines – something I did not know about prior to these workshops,” said Fabiola Millan, a student in Valla’s class. “I can imagine myself creating a project that would require this sort of collaboration between Brown and RISD students.”

According to Valla, the interdisciplinary, cross-college collaboration began when Serre reached out to RISD faculty members to spread word of the innovation zone, a new collaborative space in the Robert J. and Nancy D. Carney Institute for Brain Science where faculty and students from various disciplines can gather to brainstorm ideas and hold workshops, hackathons and data science challenges.

Valla said his course, Seeing Machines, was particularly relevant to Serre’s expertise because it discusses the intersection between computation, technology and culture. Serre’s research focuses on visual processing from both a biological and computer vision perspective. Students in the class read texts about computer vision and its historical precedents, Valla said.

The two professors initially discussed the possibility of Serre speaking to the class about his work in computer vision, as Valla was interested in exposing his students to how artificial intelligence can be incorporated in the design process. But the idea eventually evolved into several meetings that combined lectures, project discussions and presentations, Valla said.

In the first meeting, Serre delivered a “crash course” on neural networks and their applications, including facial recognition and the tracking of pose and motion, he said.

“I wanted the students to understand the kinds of problems that computer vision can solve today,” said Serre, an associate professor of cognitive, linguistic and psychological sciences who is affiliated with the Carney Institute. “I feel like there is a big disconnect between where computer vision is at and what people know is possible.”  

Serre, his lab members and the students then engaged in a dialogue about the types of projects they wanted to pursue. One such project revolved around typography and the generation of new fonts. A few students were interested in creating fonts that were influenced by art styles and artists, Valla said.

The students thought of grabbing samples of text in art and interpolating, or mixing, them with a font. For example, if the students wanted to create a Picasso-style font, they would find text from Picasso’s works and develop a font based on these samples by using a neural network, Valla said.

Though this idea did not come into fruition due to the time limitation, Serre’s lab members trained a neural network to create new typefaces by blending preset fonts, Serre said. They were able to train the network using a data set of more than 600 fonts that the students had provided, he added.

According to Serre, students were able to combine two fonts and choose the degree of blending; a scale set at 50 percent generated a new typeface that had equal influence from the two original fonts.

For another project, the students used models of Ikea furniture to train the neural network, which was then able to generate novel designs of tables and chairs.

During the workshops, students also discussed using image recognition to explore visual culture and automate music selection, Valla said. One student was interested in how Google Image results differed when entering a search term in various languages, while another student wanted their computer to play music based on their facial expressions. These projects, however, would require additional time and data, Valla added.

“Overall, the students were energized by the experience,” Valla said, adding that the projects Serre’s team presented were “visually intriguing.”

The researchers said they hope to continue their collaboration. Serre and Valla are considering co-teaching a class that would explore the topics of the workshop in more detail after they both return from sabbatical, Serre said.

Valla added that it would also be interesting to introduce some of Serre’s students to art history and to incorporate design in their work.

During the semester, Serre also delivered a lecture on machine learning to a class taught by Paolo Cardini, associate professor at RISD. The course, Cardini said, covers “the study of artificial intelligence applications within the health care context.” The two professors are currently discussing additional projects for future collaborations, he added.