Machine Learning and the Arts

by John Favaro

02 June 2018

In my previous blog post here for the GRASPnetwork, while talking about the problem of the invisibility of software, I briefly mentioned artificial intelligence. I’d like to come back to that now and spend a little more time on it.

Ten days ago I was in Madrid to give a keynote presentation at the Seventeenth International Conference on Software Reuse. Now, most of us have heard by now of the new hot topic of “machine learning” and “deep neural networks”. This technology is the basis for advances we’re seeing like Siri for Apple smartphones and Google Translate. Unfortunately, it’s not only invisible like all software, it’s downright impenetrable. The reason is that this kind of software learns. It actually changes itself as it goes along, so that what you have after a while is no longer what you had in the beginning.

That may be fine for a program giving you advice on the nearest restaurant to visit, but this software is also being used in mission-critical systems like self-driving cars and medical diagnostics. In these contexts, it’s simply unacceptable not to have any idea what the software is doing. With a nod to our GRASPnetwork, I spent a few moments at the conference talking about the problem of “making the invisible visible” in these neural networks: how we can try to understand what the software has learned from its experience. Will it recognize that pedestrian? That bicycle? That pedestrian walking alongside her bicycle? In my talk at the conference, I pointed out a few attempts to make progress in this area (this one isn’t for the faint of heart), but we’re pretty much at the beginning, with not much research reported so far.

Can our GRASP concept of artful thinking help out here? I decided to look around and see what was already happening today. There is good news and bad news to report: the bad news is that I haven’t yet seen activity in the arts specifically oriented toward helping machine learning specialists understand their software. The good news, however, is that I am seeing a very high level of interest of artists in applying machine learning within their own profession. Since the first step in any potential collaboration is awareness, this is good news indeed. So I thought I would contribute now to this awareness-raising with an overview of some of the things I found.

Not surprisingly, Google itself has been active, starting the Artists and Machine Intelligence initiative: “By supporting this emerging form of artistic collaboration we open our research to new ways of thinking about and working with intelligent systems.” In his essay Art in the Age of Machine Intelligence, Blaise Aguera y Arca, who is the head of the Google AI group in Seattle, has a number of interesting observations that are relevant to our ideas. First, he notes that “Art has always existed in a complex, symbiotic and continually evolving relationship with the technological capabilities of a culture.” He then goes on to a discussion of the possibilities of neural-like systems to investigate topics as varied as culture, ideas, and “the working of our own minds”. Fortunately for our purposes in GRASP, he notes that this “… requires that we apply ourselves rigorously and imaginatively across disciplines,” and that there is “… no shortage of engineers and scientists who are thoughtful and eager to engage with artists and other humanists.”

It’s also not surprising to find that machine learning is being used to create a number of different types of art. The visual arts are well represented so far. Tom White is examining “the ability of neural networks to create abstract representations from collections of real world objects.” You can see some of the results of his “perceptron engine” implementations here. The photo associated with this blog entry is actually a photo of a statue in my home that has been elaborated by the Prisma app, which uses machine learning to transfer artistic styles to photos.

There is plenty going on in the relationship between machine learning and music, ranging from composition to performance. Want to compose piano music in the style of Chopin? No problem – “there’s an app for it” – based on Recurrent Neural Networks and Deep Learning.

And it’s also good news that a lot is being done to provide education in machine learning to artists. Machine learning software is notoriously difficult to create, of course – but it’s also notoriously difficult to use for non-experts. Fortunately, there have been some efforts to package the intricate algorithms and data structures into something that is more approachable for artists and musicians, such as Wekinator. You can even sign up for a free online class in Machine Learning for Musicians and Artists, if you like. This is especially important for our goals in GRASP – in order for artists to be able to help us to visualize and externalize the intricacies of machine learning modules, they need to have a basic … well, grasp … of what they are all about. For now, the artistic community is more interested in applying machine learning to their own creations, but once they become fluent in the technology, they will be able to start pondering ways to help us to understand our creations.

Page updated

Report abuse