Fear and Loathing in Vancouver - NeurIPS in a Nutshell
The first thing to note about #NeurIPS2024 is that there is so much of it. After a dizzying few days (and as I write this from the airport, conference workshops are still going on), nearly twenty-thousand total attendees, parallel sessions, sprawling halls with rotating posters creating feelings of anti-déjà-vu, it is too much for anyone to take in, let alone try to distill and summarize.
What else would you expect from the world's largest AI conference, especially nowadays?
To cut straight to the punchline, while it's impossible to encapsulate the conference's takeaways into a single statement, my feeling ultimately is that there isn't one; this despite the fact that many of us ask the question "Where is the world heading post-Generative-AI?"
You may or may not be reassured to learn that the world's foremost minds on AI are asking the same question.
Deepfakes & voice authentication
Much of the conference was aligned, unsurprisingly, to the most popular themes du jour, AI-wise. Large language models, image/video and text generation featured most prominently.
Out of ~4,500 papers featured at #NeurIPS2024:
Disappointingly, the authors of most of the deepfake detection papers did not make it to the conference to present their work; likely through no fault of their own, as visa issues are a prevalent challenge of international conferences. So these will be relegated to offline review, and may be referenced in future articles of the 10 Ideas series.
I will also touch upon speech synthesis and watermarks in one or more upcoming articles. In fact, the overarching theme of the remaining articles will be on AI safety and the responsibility of AI researchers and technologists. This theme also cropped up in a couple of the keynotes at NeurIPS.
For AI Leaders, By AI Leaders
With several keynotes by (or referring to) giants of AI research, I'd be remiss not to try and capture some ideas around leading AI research and development teams to success.
Professor Fei-Fei Li 's keynote highlighted how the AI revolution as we know it today was realized due to the confluence of three critical ingredients -
Recommended by LinkedIn
Recognize current limitations (e.g. lack of data) and address them. Keep abreast of latest innovations (e.g. GPU computing) and what they allow us to do, that was impossible in the past.
Dr. Lidong Zhou's keynote focused on the theme of taking a holistic approach to AI development. In his words, it was time for AI and systems engineering "to get married." The people building and optimizing systems to deliver AI at scale to the entire world would become more involved in AI development itself, and vice versa.
Tighter coupling between disciplines will open up possibilities neither would have (easily) thought up on their own
The special session, Standing the Test of Time, recognizing an important AI paper published at NeurIPS ten years ago recognized two landmark papers this year. The first was Ian Goodfellow et al's seminal work Generative Adversarial Networks, presented by coauthor David Warde-Farley.
As Dr. Warde-Farley recounted the now-classic story of a group of grad students brainstorming over drinks at a local Montréal pub, he touched upon one important aspect of how it was even possible for their team to realize such work (and mind-bogglingly turned around their paper in 11 days from inception to submission).
Warde-Farley et al. were all part of Prof. Yoshua Bengio's lab.
Prof. Bengio had wisely created an environment that encouraged wild ideas and allowed for their realization (concretely, one critical component was assembling the team that built the theano software library).
Where is AI Going Next?
What about the other paper that was recognized as a landmark work, originally featured at NeurIPS in 2014? It turns out that 2014 also featured the work entitled Sequence to sequence learning with neural networks, which introduces the neural network design for next-token prediction, and is nothing short of a core component of what would become large language models (LLMs), most famously featured as OpenAI's ChatGPT.
The first author of that paper? None other than Ilya Sutskever , co-founder and former CTO of OpenAI, who appeared in person (along with his coauthors) to ruminate on the legacy of this work and share his reflections on where things might be going next.
Besides the appearance of Sutskever in person, there are no other twists here. No surprise answers to our biggest question(s) on the future (with AI).
Sutskever's reflections were nonetheless intriguing enigmas, punctuated by his stoic, intentionally expressionless demeanor (though I'd say he was in a rather good mood).
Per Sutskever, we've come to the end of an era. The massive LLMs have fully swelled to bursting with the world's data, there's nothing left to consume internet-wise.
In the near term there will still be localized optimizations to try and improve things: improve inference compute, make AI agentic, and leverage synthetic data. But no further leaps with the current paradigm.
As Sutskever peered towards the back of the hall, and slightly above everyone's heads, into the future, he proclaimed that superintelligence was coming next. How? When? Unknown, unknown.
But this superintelligence would be self-aware, have the ability to reason, be truly agentic (in the sense of having and acting upon its individual wishes), will form understanding in a manner resembling our own, and will probably be deserving of rights as a sentient being. And the final stroke, to make this superintelligence complete: with all these characteristics it will subsequently become unpredictable.