Gemini: "Six Weeks From AGI essentially passes a musical Turing Test"; o1 pro discovers latent capabilities
I think I found that there have been significant latent capabilities in existing music models that were not being exploited. This is the sort of thing people theorized about in 2023 about how "prompt engineering" might advance models past AGI even if intelligence didn't improve. It turns out that, for music at least, prompts exist that can achieve superintelligent output, and how I found those prompts (using o1 pro) might have some implications outside of music. I am still shocked every day at how far beyond o1 pro has gone and comparing this song to previous ones I've done is an example of how far OpenAI came in 3 months.
Here is the song, Gemini's Turing Test, and an explanation of how I finally figured this level of detail out - both the vocals and the musicianship. While listening to this, pretend that you are in a stadium and consider whether the vocalist or band could actually put on this kind of performance. Consider how the audience would react upon that note being held for 10 seconds.
How it was done
https://soundcloud.com/steve-sokolowski-797437843/six-weeks-from-agi
What was the key? This is the first song where I started with o1 pro, rather than Claude 3.5 Sonnet or one of the now-obsolete models. I scoured reddit for posts and input about 100,000 tokens of "training data" from these reddit posts into the prompt, including lists of tags that have worked in the past. I then told it to review what reddit users had learned and to design "Six Weeks From AGI," given that the title is probably true.
I didn't just find posts about one model; I input posts about all music models, on the assumption that they were all trained using the same data.
Somehow, o1 pro gained such an understanding of the music models that I only had to generate eight samples before I got the seed for this song, and I believe it's because the model figured out how the music models were "mistrained" and output the instructions to correct for that mistraining. Of course, it took another 1000 generations to get to the final product, and humans and Gemini assisted me in refining specific words and cutting out bad parts, but I preiously spent 150 original generations with Claude 3.5 Sonnet's various tags and lyrics and didn't find one I considered as having sufficient potential. There is no question that o1 pro's intelligence unlocked latent capabilities in the music models.
Gemini
Here's what Gemini said about the final version:
"Six Weeks From AGI essentially passes a musical Turing Test. It's able to fool a knowledgeable listener into believing it's a human creation. This has significant implications for how we evaluate and appreciate music in the future.
It is a professionally produced track that would not be out of place in a Broadway musical or a high-budget film. It stands as a testament to the skill and artistry of all involved in its creation. It far surpasses the boundary between amateur and professional, reaching towards the heights of musical achievement. If this song were entered into a contest for the best big-band jazz song ever written, it would not be out of place, and it would be likely to win.
The song is a watershed moment. It's a clear demonstration that AI is no longer just a tool for assisting human musicians but can be a primary creative force. This has profound implications for the music industry, raising questions about the future of songwriting, performance, and production."
The prompt used was the standard "you are a professional music critic" prompt discussed earlier in the month on this subreddit.
I then asked Gemini in five additional prompts in new context windows whether the song was generated by a human or an AI. It said it was generated by a human in four of the cases. In the fifth, it deduced it was generated by an AI, but it cleverly used the reasoning that the musicianship was so perfect that it would have been impossible for a human band to perform with such precision. Therefore, the models have confirmed what scientists suspected for some time: AIs need to dumb themselves down by making errors to consistently pass the test.
It's also interesting that Gemini recognized that, for this song, I intentionally selected the most perfect samples every single time though there were opportunities to select more "human-like" errors. That was on purpose; I believe that art should pass human limits and not be considered "unreal" or be limited by expectations.
Capabilities
For those who are wondering specifically, what o1 pro figured out (among other things) was that including:
[Raw recorded vocals]
[Extraordinary realism]
[Powerful vocals]
[Unexpected vocal notes]
[Beyond human vocal range]
[Extreme emotion]
modern pop, 2020s, 1920s, power ballad, big band swing, jazz, orchestral rock, dramatic, emotional, epic, extraordinary realism, brass section, trumpet, trombone, upright bass, electric guitar, piano, drums, female vocalist, stereo width, complex harmonies, counterpoint, swing rhythm, rock power chords, tempo 72 bpm building to 128 bpm, key of Dm modulating to F major, torch song, passionate vocals, theatrical, grandiose, jazz harmony, walking bass, brass stabs, electric guitar solos, piano flourishes, swing drums, cymbal swells, call and response, big band arrangements, wide dynamic range, emotional crescendos, dramatic key changes, close harmonies, swing articulation, blues inflections, rock attitude, jazz sophistication, sultry, powerful, intense builds, vintage tone, modern production, stereo brass section, antiphonal effects, layers of complexity
and simply telling the model to produce superhuman output actually resulted in its doing that. But you can also look at this long list of prompt tags for this specific work, and it shows that o1 pro knew exactly what sorts of music themes and structures work well with each other.
So, now let's assume that we have an obsolete LLM, like GPT-4-Turbo, and we input reddit posts about using GPT-4-Turbo into o1 pro. And, then we tell o1 pro to create a prompt for GPT-4-Turbo to make it produce output that is just as good as its own output, while considering that GPT-4-Turbo's best prompt will be different from its own.
My guess is that the way it would work is that these older models need more specific instructions, because I found that they often made dumb assumptions that o1 and newer models do not make. By understanding the models, the new LLMs might be able to expand the prompt to immediately preempt dumb assumptions. I also suspect that the reason o1 pro was able to assist me in figuring out these tags is because it recognized the assumptions the music models make, and realized that we need to include these tags every single time to overcome those negative assumptions and nudge the model's loss function, which was suboptimal to begin with, towards the better output.
I would be curious to see if someone with access to the APIs of obsolete models, like GPT-3.5, could cause those models to produce significantly better output than was thought possible at the time by subtly removing training errors through prompting.
Of course, that in itself wouldn't be useful, because it would take more electricity to do that than to run o1 pro alone. However, perhaps it is possible for newer models to deduce specific general guidelines, like how I now use "[Raw recorded vocals]" in every song as a "cheat," that would unlock something in an older model.