How AI Data Sets Work – And How Artists Can Collaborate With Them

Some of the most iconic cultural moments happened as a result of collaborations, from Jean-Michel Basquiat and Andy Warhol’s famous 1983 union to today’s limited-edition drops by contemporary artists and fashion houses.

Collaborations bring together different perspectives, skills and influences, resulting in unique and often groundbreaking creations. Today, artists are expanding their collaborative horizons through partnerships with a rather unconventional creative force: artificial intelligence (AI).

“On days when you're dealing with writer's block, or the synapses aren't firing in a certain way, it's been really cool to see [AI] technology spark that creative flow and expand the realm of ideation and possibility for artists,” said Chris McGarry, founder and CEO of AI music platform WarpSound.

But if you’ve ever tried to use the AI language model ChatGPT, entered an experimental prompt into the text-to-image model Stablediffusion or played around with Google’s text-to-music generator MusicLM, you may have noticed that AI can’t read minds – at least not yet. Your first attempts at working with AI tools may result in generic outputs, or simply ones that don’t quite capture what you had in mind.

“They are bad,” one musician joked of his experience trying out AI music generators. The comment was made in May during a workshop on AI music production at the inaugural Wavelengths Summit in Brooklyn, New York, hosted by the music research network Water & Music. Audience participants agreed that AI tools, while sophisticated, still come with a noticeable learning curve.

For creatives, tackling machine learning requires an understanding of how to best feed it data and refine its algorithm to complement one’s artistic endeavors.

How datasets work

One of the primary ways artists can harness the power of AI is by creating their own datasets, typically comprised of text, images, sounds, melodies and video. Each machine-learning project may require different-sized training data sets to be successful. There are plenty of helpful courses and videos available online that best explain how to optimize data sets.

To create a powerful tool that still feels “authentic,” artists can train machines based on their previous work. This can help them distinguish their AI collaborations from those generated by free and open-source models, which tend to use publicly available data and can sometimes come across as dry and generic.

Generally speaking, a free AI model’s data comes from publicly-published sources. The data is often procured by internet-crawling bots that scoop up widely available raw material and catalog the information. Meanwhile, private companies and research firms are more likely to use proprietary datasets, particularly when copyright laws are a concern.

“We have to find ways of using data that are legal,” said Michael Turbot, technology promotion manager at Sony Computer Science Lab (CSL) Paris, an independent research laboratory, during the Wavelengths Summit. The lab is working on AI-assisted music production tools for artists.

According to Turbot, Sony CSL is subject to an auditing process from Sony Music Publishing in order to work with any artists currently under contract with the publishing house. Sony CSL has therefore been careful to include only legally air-tight sounds and songs in its datasets, opting for public domain songs only, or songs belonging to artists working in partnership with Sony CSL who have given explicit permission.

“We have to be extremely careful with our use of data,” said Turbot. “Otherwise we know that the doors would be instantly closed by people, either in their industry or with artists.”

Creating your own dataset

Like private companies and labs, artists can curate their own datasets using their own material. Compiling the right data set is crucial for ensuring that AI accurately captures an artist's unique style and voice. The data set acts as the foundation for the AI algorithms, guiding its decision-making process.

Refining an AI data set involves a multi-step process. It starts with collecting high-quality source material, either from the artist's music or works that reflect their desired style. Some artists may merge different data sets to represent eras in their creative evolution or opt to create unique datasets for every new project.

“Lately, I've been taking old drawings from when I was really young,” said photorealistic AI artist Claire Silver in a February Twitter Space. “I've been putting them in AI with prompts to try and recreate what I was seeing back then and trying to draw, but didn't have the skill to share. And that's been really cool.”

Silver said she uses a combination of AI models, including DreamStudio and NMKD.

Literary artist Sasha Stiles, who has been collaborating with GPT language models since 2018, says she customizes her datasets with a “compendium of training materials” that started with drafts of her manuscript and all of her “human poetry.” She then added research notes, reference material and inspiring literature.

“Tucked in there [were] poems and essays that were germane to the things that I was writing about,” Stiles said. “There are excerpts from T.S. Eliot, there are passages from Whitman and Dickinson and there's actually little bits and pieces from various news articles as well.”

Stiles added that working with an AI language model was like “taking all the things that were swirling in my head when I was working on my manuscript, putting them into a system and using algorithms to synthesize them.”

With every medium, the quality of an artist’s dataset matters in order to produce the best outcome, but for musicians in particular, audio quality is of high importance. “When a producer or artist uses a model, it's essential to understand that the input voice quality is vital,” said Geraldo Ramos, CEO of Moises, an app that lets musicians build a customized library of songs. “If the singer performs poorly, the output will be as bad as the input.”

Letting AI stretch you artistically

Despite the expected $42.6 billion being poured into generative AI investments today, artistic collaborations with AI are still experimental. Curating and refining an AI data set requires an understanding of machine learning, attention to detail, lots of patience and ample curiosity.

That said, using AI in creative industries doesn’t appear to be a passing trend. The idea itself goes back to 1959 when the first book on computer-aided music composition was published, and it only seems to be accelerating with the rise of metaverse gaming and AI-powered non-fungible tokens (NFTs).

While AI offers incredible opportunities for creatives, some continue to voice concerns about its use cases, arguing that AI cannot replace human emotion, nor the connection formed between artists and fans. But if you do want to collaborate with AI in ways that respect your craft while evolving your work, crafting your own data set is a great way to start.