How to speed up AI progress?

Like many others, I have been predicting that AI scaling won't lead to human-level general intelligence (HLGI). Finally, some excellent media commentary, including some surprising confessions from BigTech executives and VC heads that the current approach of scaling AI to gain inference improvements is hitting a wall.

This video I watched today got me thinking about what we can do to mitigate the emerging issues with scaling and data inadequacies and what measures we can take to keep the AI momentum progressing.

Due to apparent business interests, AI chip companies and cloud providers may want scaling. Still, no empirical or biological basis exists on which scaled-up homogenous mono-dimensional connectionism networks (on which approach current advanced AI models are based) will achieve HLGI. No, there are to be no whispers of AGI emerging from this approach ;-)

There is also the issue of these big models (by size of parameters) running out of training data, even running out of quality synthetic data, not to mention the enormous energy and water consumption.

Here are my recommendations to overcome the hurdles imposed by this irrational scaling-up approach:

1. The brain does not function as a homogenous entity, and even its basic unit (the biological neuron) is not just heterogeneous in its structure but has more capability than a hundred-parameter AI neural network model. The brain's intelligence functionality is distributed across specialized parts dedicated to cognitive reasoning, memory, visual inference, motor control, etc., which work together seamlessly to help humans navigate and excel in this world. An AI model that can attain HLGI or exceed this has to be a multi-modular (not just multi-modal) distributed intelligence network with the sum being more significant than the parts.
2. Further, the brain is highly energy efficient due to millions of years of evolution. It just about consumes 260 calories per day. In contrast, your average LLM consumes about 400kw per hour when being trained and around 3.6-36kJ per query when deployed. We already know that training, hosting, and running these big-scale AI models has a significant environmental impact due to data centers' water and energy consumption and greenhouse gas emissions. For sure, algorithmic optimization and model pruning and distillation can help but not make current large models as energy efficient as the brain. So how does the hundred trillion synaptic connection brain achieve this energy efficiency? Through sparse signaling, predictive coding, and selective neural resource use. These aspects must be explored to ensure our drive to achieve AI progress does not inadvertently speed up climate change.
3. The previously dominant symbolic approach with AI ran into its wall due to the sheer impracticality of representing every scenario in this complex world. Neural networks, with their pattern-based learning, were more practical and performed better than the symbolic models. Yet, these neural network models do not think or reason but are statistical predictive machinery no matter how AI companies market LLM inference as simulating human ‘thinking’! Yet there is hope for both approaches (neural nets and symbolic). AlphaFold, which made a buzz last year for its uncanny ability to predict protein structures and gained its creators this year's Chemistry Nobel, is based on a neuro-symbolic model (a hybrid approach combining neural networks and symbolic reasoning). I firmly believe there has been very little attention paid to Neurosymbolic AI, which has a much more problem-solving approach than pure connectivism models. We must focus on these approaches alongside energy-efficient causal reasoning models like hyper-dimensional computing.

The path forward for AI development requires a fundamental shift in our approach, moving beyond the current paradigm of simply scaling up neural networks. By drawing inspiration from the brain's modular architecture energy efficiency and combining the strengths of both neural and symbolic approaches, we can work toward AI systems that are more capable and sustainable. The challenges we face – data limitations to environmental concerns – should serve as catalysts for innovation rather than roadblocks. As we stand at this critical juncture in AI development, it's essential to recognize that actual progress will come not from brute-force scaling but from more innovative, nuanced approaches that better reflect the complexity and efficiency of biological intelligence. The future of AI lies not in bigger models but in better ones.