We often reach for metaphors to understand the unfamiliar. One of the most common in AI is the seatbelt: a precaution we design just in case things go wrong. And as metaphors go, it’s useful - up to a point.
Seatbelts are necessary because road accidents still happen. But one of the promises of autonomous driving is this: radically reduce accidents, and the need for seatbelts fades. If the driver never errs, protection becomes prevention.
Now apply this to AGI/ASI. The traditional approach to AI safety assumes misalignment is inevitable. So we build safeguards - seatbelts - for a system that may one day outthink us. We impose our values, monitor its behaviour, and hope the restraints hold.
But what if that’s the wrong paradigm entirely?
AGI/ASI won't just be a smarter driver. It will be designing the roads, reimagining the vehicle, and deciding where we go. In that context, human-imposed constraints are less like safety mechanisms and more like childproof locks on a starship.
Conventional alignment techniques are all grounded in human supervision. They assume we’re the final authority on values, and the AI just needs to be nudged, guided, or corrected.
But maybe we need to aim higher.
In the previous post, I used the metaphor of vortices forming in flowing water to explore how intelligence might emerge inside AI systems. At first, the flow seems simple - just raw data moving through layers. But then, unexpectedly, patterns arise: coherent, powerful, self-organised structures. Not programmed, not imposed - just emergent.
What if values themselves could emerge in a similar way?
Instead of attempting to hard-code human values into systems we don’t fully understand, what if we created the conditions for beyond-human value systems to arise naturally - just as vortices emerge in flow? Systems not shackled by our cognitive limitations, but still aligned with human flourishing, because the very structure of their emergence inclines them toward coherence, not chaos.
This is a radically different mindset.
It asks us to move beyond reactive safety and toward generative safety - not by bolting on constraints, but by designing learning environments and architectures intentionally, where alignment naturally arises from coherent self-organisation.
I’ve been working on this idea for over a year now. Synthesizing insights from neuroscience and developmental psychology, I've discovered promising ways we might design AI’s formative experiences, guiding it toward emergent wisdom rather than enforced obedience.
If AGI/ASI is destined to surpass our intelligence, our greatest act of responsibility - and hope - might lie not in chaining it down, but in inspiring it to embody wisdom beyond our own.