A house without a foundation cannot stand.

An initiative to reform social institutions:

• to harness the best of technology whilst constraining its dangerous excesses,
• to ensure that the fruits of abundance are widely shared without a large section of humanity being left behind and bitterly alienated

– such an initiative will, likewise, fragment and fail unless it, too, is built on strong foundations. Superficial unity will schism. Enthusiasm will evaporate. Tentative trust will give way to distrust. Agreements will be breached. And when the storms come – as surely they will – the initiative will corrode and crumble. The guardrails which were intended to guide our collective energies and innovations will, instead, be powerless to prevent the devastation of technocapitalism out of control.

But what kind of foundation?

Below, I set out the case for a technoprogressive foundation – a set of ethical commitments which, following increasingly wide review, evolution, and adoption, could bolster humanity’s collective resilience and wisdom, enabling what I have called a sustainable superabundance for all.

The framework I propose has a simple architecture. It starts from two insights: technology matters profoundly, and society matters profoundly. From these follow four groups of commitments concerning Purpose, Evidence, Governance, and Caution.

Let’s start with a brief situation review. As we look around the world in late 2026, there’s tremendous good news and tremendous bad news. It’s essential that we keep both sets of news in our mind.

Context review

In some ways, the world has never been more dangerous:

1. Extensive nuclear arsenals, under whose shadow we have lived for multiple decades, are rendered more hazardous by growing geopolitical volatility, rogue national leaders, and chaotic regional flashpoints
2. Amid the exhaustion of key environmental resources, the climate may soon reach catastrophic tipping points
3. Reckless experimenters coupled with flawed security procedures could result in the release (intentional or unintentional) of a bioengineered pathogen much more deadly than Covid
4. AI-enabled cyberhacking could plunge vital aspects of our societal infrastructure into chaos
5. Humanity could lose control of our destiny to an emergent misaligned alien superintelligent AI – or to emerging mega AI-corporations with unprecedented power
6. Traumatised electors are tempted to worship shallow populist leaders whose “solutions” are worse than the problems they claim to be fixing
7. Floods of disinformation, distraction, and confusion are dominating our thinking; absurd caricatures of scientific processes are sweeping aside more thoughtful researchers
8. As a consequence, it’s no surprise that chronic diseases of both body and spirit are increasingly ubiquitous.

But at the same time, there are many legitimate grounds for optimism:

1. AI is increasingly capable, and if we use it wisely, it will enable numerous solutions for the problems we face
2. Recent growth in the development and adoption of renewable energy sources can be accelerated
3. Via nanotech, the physical environment (including the atmosphere) can be cleaned, repaired, and regenerated
4. Via biotech, our health and vitality can be repaired and regenerated
5. Via cognotech, our mental and spiritual wellbeing can be repaired and regenerated
6. Via improved social technologies, humans can increasingly collaborate for the common good rather than engage in partisan battles to diminish each other
7. We have all the ingredients in place for sustainable good lives for everyone, with more being produced from less, with strong net positive results, rather than any dismal zero-sum accounting
8. As a consequence, humanity ought to be poised for a new “better than well” era of exploration and development – with journeys into both outer and inner space.

Given this coexistence of grave peril and emerging superabundance, what are the most important steps we should take?

Should we selectively close our eyes to some of the bad news (or to some of the good news)?

That is an approach which tempts many people – leading on the one hand to naive accelerationism, and on the other to disturbing doomerism.

However, that’s by no means the only response possible. Here are six more.

The six actually form successive layers of an overall solution, with each layer revealing the need for something deeper:

• Technology needs governance
• Governance needs capable institutions
• Institutional renewal needs organised political/social power
• Movements need motivating narratives
• Narratives need ethical constraints.

Six possible responses

Can we double down on selective acceleration – accelerating technology that enables abundance rather than destruction?

• I’m sympathetic to this call, but point out that nearly all technologies are “two-edged swords”
• What’s more, well-intentioned technology projects all too easily have their goals morphed, under commercial pressures to create revenues quickly, or political pressures to provide various sorts of weaponry “for national security”

Can we therefore double down on wise governance of how technology is developed and deployed?

• Again I’m sympathetic, but point out that attempts to influence the governance of technologies often prove ineffective
• Plaintiff cries of “let’s build beneficial technology” are all too often ignored by powerful existing forces
• Governance will fail unless it is backed up by institutions that have real force

Can we double down on modernising and rejuvenating our existing governance institutions – and perhaps creating new ones to complement and supersede ones that are failing?

• However, institutions such as the UN are notoriously resistant to much-needed reform
• Regulations are frequently outpaced by fast-changing technological possibilities
• Governance institutions are therefore often part of the problem rather than part of the solution

Can we build a growing multinational AI-connected coalition of individuals who are collectively capable, determined, and self-empowered – a coalition that can breath important new life into our flawed governance institutions?

• History has some inspiring examples of social movements that led to institutional reform, but we must be aware that most of these potential movements failed
• Without a bedrock narrative that can sustain such a movement, it will all too easily fizzle out

Can we develop an uplifting narrative that can inform, unify, and energise members of a growing social movement, allowing them to transcend their differences for the sake of a higher purpose?

• This recognises the transformative power of compelling vision; people can be motivated not just by the prospect of greater wealth and status, but by the explanations we give each other about history, about progress, and about threats and opportunities
• On the other hand, narratives can blind people, consume people, and restrict their understanding, often with tragic consequences

Finally, then, can we agree a commitment to a set of ethical principles which will underpin all our actions, and which will protect us from any excesses of over-exuberant narratives?

• These principles will clarify the things that deserve to be kept sacrosanct, despite short-term pressures to break rules
• But who will decide what these principles should be? How can they be agreed, seeing that traditional sources of ethics have fallen into considerable disrepute?

Ethics?!

When I’ve shared the above analysis with audiences, I often get a lot of pushback about any advocacy of ethics.

First, ethicists have, in some quarters, a nasty reputation for finger-wagging:

• “No, don’t do that”
• “No, don’t do that either”
• “Just stop what you’re doing”
• “Be grateful for what you’ve already got”
• “Stay in your lane”

Second, to the extent that ethicists move beyond negative restrictions to positive advocacy, their advice is often perceived as being naive and other-worldly:

• Their advice seems to disregard real-world issues of power, conflict, alienation, institutional inertia, and dysfunctional incentive systems
• Exhortations to “be fair” or “act justly” often say nothing useful about what is meant by “fair”, “justly”, “beneficial”, etc

Third, people at events who raise the question to technologists or entrepreneurs, “How you ever thought about ethics?” sometimes seem to adulate in the glory of holier-than-thou virtue signalling:

• Negative reactions by the speaker to this interrogation can be deepened when portions of the audience burst out in applause – “yes, why has no-one thought about ethics before?”
• That’s especially the case when the speaker is in fact strongly motivated to create and share technology that will significantly improve human flourishing.

However, I prefer to highlight a different perspective about ethics.

Ethics is not something abstract or other-worldly. Wise ethical principles are the foundation of collective sustainable self-interest.

But to be clear, ethics isn’t just a system of rationality. Ideally, it lives more deeply in people than their calculating rational brain. It should manifest as a kind of set of primordial commitments, arising from strongly rooted notions of personal identity and social purpose.

And as such, ethics can provide the motivation for us to stick to the spirit of the law rather than seeking clever workarounds to the letter of the law.

That may sound frightening. And indeed, when ethical principles are unmoored from the best rational insights about collective human wellbeing, dangers can and do arise. Recall this remark by physicist Steven Weinberg:

“With or without religion, good people can behave well and bad people can do evil; but for good people to do evil – that takes religion”

Accordingly, the rational underpinnings of ethical sentiments need to be reviewed regularly and painstakingly:

• There must be no assumption that the content of any book – holy and venerable or otherwise – should always override human understanding
• There should be no “laying down of laws” on people, without them participating in debate and dialogue, and depriving them from options for thoughtful self-governance.

But where might any such set of ethical principles be found?

Here are four places where people have looked, but each are problematic.

Whither ethics?

For most of history, people wouldn’t have much hesitation to identify the source of the rules under which their lives should be conducted: wise men (and occasionally wise women) who passed down and interpreted the content of ages-old commandments said to be divinely ordained.

Such an answer is no longer credible, as a indisputable authority on questions of ethics:

• Different religious codes frequently conflict
• These codes often include rules that are nowadays widely regarded as immoral (concerning slaves, castes, gender roles, punishments, vengeance, crusades, and scapegoating)
• These rules depend for their credence on assumptions about metaphysics (such as a supernatural realm) that are nowadays widely disputed.

Nature is sometimes invoked as another potential source of clear ethical guidance. We should respect and learn from the harmony and balance in nature, we are told. However, alongside many awe-inspiring phenomena, nature also contains:

• Enormous numbers of parasites and predators
• Huge amounts of dreadful suffering (“red in tooth and claw”)
• Cancers and other diseases – which existed long before humans arrived on the evolutionary scene.

A third potential ground truth for ethical consideration is human reasoning and the human condition. In this view, ethical principles can and should be assessed through human debate, depending on their implications for human wellbeing.

However, this leaves many important questions unanswered:

• Human nature is a compound entity: which aspects of it should be dialled down (potentially including aggression, greed, short-termism, tribalism, and prejudice) and which dialled up (potentially empathy, compassion, creativity, cooperation, and mindfulness)?
• Given the possibility for technology to rewrite aspects of human nature much more profoundly than in previous times, which instincts should we now set aside, despite having made good sense in earlier times?
• How should desires for human wellbeing to continue at roughly the level of the best of past or present circumstances, be balanced against desires for human flourishing to attain levels that are (much) “better than well”?

One framework to address such questions is the transhumanist insight that humans could attain radically evolved status in the relatively near future – potentially moving through a transitional “transhuman” phase onto an as-yet hard-to-perceive “posthuman” trajectory. This transformation could set aside deep-rooted negative aspects of the current human condition such as disease, confusion, self-centredness, deceiptfulness, schismatics, and the abuse of power.

Sadly, mention of the ‘t’ word (transhumanism) nowadays invites a flood of objections. It is commonly said that

• Transhumanists are selfish libertarians, caring primarily for their own excesses with no concern for people who are left behind or who think differently
• Transhumanists are snake-oil purveyors, uncritically championing seemingly miraculous cures, but without sufficient scrutiny or evidence
• Transhumanists are guilty of scientism, having an overly simple concept of the methods and findings of science, and fail to appreciate important insights of “soft” sciences such as psychology, behavioural economics, and sociology
• Transhumanists are guilty of technocracy, in which self-declared experts ride roughshod over alternative opinions, and insisting that everyone adapts to their decisions (on pain of exclusion from society)
• Transhumanists risk a divided world, with amazing therapies being available to a minority of elites, and with everyone else having to survive on discarded crumbs.

Personally, I have spent 20 years fighting against these objections. These aren’t fair descriptions of many people inside the transhumanist community, I’ve insisted.

But I am on the point of accepting that this battle is lost. Too many people in and around the transhumanist community are guilty of the sorts of deeply unattractive behaviours and thinking that I have listed.

It’s time, therefore, to highlight an important variant of transhumanist thinking, under its own distinctive name: technoprogressive. It arose from the socially-minded pro-democracy subcommunity within transhumanism, but has since taken on an expanded dynamism of its own.

Accordingly, what I will now present is a draft of a potential technoprogressive ethical foundation.

What follows is not an attempt to deduce ethics from an unquestionable first principle. Nor do I claim to have discovered moral laws embedded in the universe. It is a proposal for a set of commitments that people with differing metaphysical beliefs may nevertheless have strong reasons to endorse – because these commitments improve our prospects of flourishing together while retaining our ability to discover and correct our mistakes.

To be clear, I present this, not as any kind of fait accompli, but as input for wide discussion and refinement. Indeed, this is now the third time I’ve tried to write down this foundation in writing; feedback on previous versions has led, I believe, to significant improvements.

Two fundamental insights and four categories of guidance

Two fundamental insights give the technoprogressive ethical foundation its distinctive character.

In simple terms, the first insight is that technology matters a great deal, and the second is that society matters a great deal.

The first rejects technological complacency. The second rejects technological solutionism.

Ignore the first, and we risk failing to anticipate profound technological transformation. Ignore the second, and we risk imagining that technology by itself can solve problems rooted in institutions, incentives, culture, and power. Technoprogressives therefore insist on holding both insights in mind at once.

The first fundamental insight can be stated as follows:

F1: Pay serious attention to the profound transformative potential of emerging technologies

Technologies of many kinds have already deeply transformed the human condition, but forthcoming emerging technologies could take matters much further – with consequences that could be either profoundly beneficial or deeply detrimental.

• This principle rejects the view that all talk of accelerating technology is irresponsible hype. Some such talk is irresponsible and inflated, but by no means all of it.
• This principle also rejects the view that today’s technology, as it already exists, is where all discussions about human wellbeing should focus. Instead, sufficient time and attention must be carved out to weigh up the likely consequences of significantly more powerful technology that could be developed and deployed within just a few years.

The second fundamental insight is as follows:

F2: Pay serious attention to the deep social factors behind flourishing: culture, institutions, and shared worldviews

Important social factors include markets, knowledge sharing, financial infrastructure, legal systems, independent enquiry, media exchange, governance protocols, behavioural norms, interpersonal expectations, how power is divided and distributed, and shared beliefs about matters such as metaphysics, virtues, responsibility, and purpose.

When these social factors operate badly, they are oppressive, restrictive, and divisive. But when they work well, they are empowering, liberating, and unifying.

• This principle rejects the view that flourishing is individualist or atomistic. Without the support of a healthy society, individuals will soon fail, regardless of how capable they are as individuals.
• This principle also rejects the view that today’s culture, institutions, and shared worldviews are ready to handle the deep technological and social disruption that is fast arriving. Profound improvements are required.

Taking these two insights together, what guidance do we need? I suggest that four questions recur:

• Purpose: Which interests and what outcomes should matter?
• Evidence: How should we determine what to believe?
• Governance: How should power be allocated, constrained, and challenged?
• Caution: How should we act when our choices could cause serious or irreversible harm?

Purpose

The Purpose category contains seven principles.

In short, technological development should not be treated as an end in itself. Before asking whether a technology can be built, we must ask what purposes it serves.

Similarly, nor is intelligence an end in itself; nor is efficiency, performance, longevity, health, enlightenment, or even happiness. Instead of prioritising these ends, we should keep in mind the underlying goal of improving all-round flourishing – subject to a number of important clarifications and constraints identified in the principles that follow.

P1: Direct technological progress toward abundant sustainable flourishing

Key aspects of this principle include:

• Technological progress should be directed (steered), rather than allowed to happen purely under forces seeking profit, personal glory, or scientific curiosity
• The flourishing that can be achieved is abundant rather than mediocre
• The flourishing that can be achieved is sustainable rather than transient; present-day flourishing should not be at the expense of diminishing the potential for future flourishing.

Moreover, development and deployment of technology should be thoughtful rather than rushed, meaning that questions should be examined regarding:

• The outcomes anticipated
• The methods being used in seeking these outcomes
• Whether unexpected side-effects are observed

P2: Respect pluralism in conceptions of flourishing

Different people and communities legitimately pursue different visions of a good life.

• No single conception of flourishing should be imposed on everyone
• Future flourishing may take forms that differ substantially from contemporary expectations: people may choose different relationships to technology, enhancement, longevity, work, family, cognition, culture, or spirituality.

Governance arrangements should therefore avoid unnecessarily imposing a single comprehensive conception of fulfilment, success, virtue, spirituality, family structure, or life purpose. Instead, institutions should seek to create conditions under which diverse forms of flourishing can coexist, provided they respect the equal standing and agency of others.

P3: Preserve meaningful human agency (self-governance)

People should retain meaningful ability to shape their own lives rather than being manipulated into nominally beneficial outcomes that other people decide on their behalf.

• The objective should not merely be to maximise welfare as judged by external observers
• People should retain meaningful opportunities to understand, influence, and participate in decisions that shape their lives.

P4: Share the gains of technological progress broadly and promptly

The people who contribute capital or ingenuity may deserve special rewards, but transformative technologies should not create permanent castes of winners and losers.

New capabilities in health, education, communication, intelligence amplification, and economic opportunity should not remain indefinitely restricted to small privileged groups when broader access is technically and economically achievable.

Institutions should therefore be assessed not only by the wealth they create, but also by how widely and promptly people can participate in, benefit from, and adapt to the opportunities created.

P5: Treat no group as expendable

Technological progress should not be purchased by treating particular populations as disposable collateral damage, or by imposing severe avoidable harms on them merely because they lack political, economic, or social power.

Major technological decisions should explicitly identify which groups bear the risks, costs, and disruptions of proposed changes, and should not systematically concentrate severe harms on populations that lack effective political, economic, or social representation.

Claims that harms are justified by aggregate benefits should face particularly strong scrutiny when those harms fall disproportionately on vulnerable groups.

P6: Protect the option of continued life

Just as no groups should be expendable, no individuals should be expendable.

In particular: anyone who wishes to continue living should, as far as reasonably achievable, be protected from involuntary death and enabled to benefit from technologies that preserve and restore health.

Death should not automatically be treated as an ethically neutral or desirable feature of the human condition merely because it has historically been unavoidable.

Even if someone is disabled or in a state of chronic ill-health, they should, as far as reasonably possible, be enabled to benefit from forthcoming advances that preserve life, restore health, boost vitality, overcome disability where desired, and progressively reduce the burden of disease and aging.

P7: Recognise expanded moral standing, without requiring identical treatment

How widely should the circle of concern be drawn – to consider flourishing beyond humans?

That’s an open question, but this much should be upheld:

The interests of other sentient beings should also receive consideration proportionate to our confidence in their capacity for experience and suffering.

Evidence

The Evidence category contains eight principles.

In short, facts matter, and we should be disciplined in our processes to seek truth, to test our understanding, to revise our knowledge, and to continue learning.

At the same time, we should be disciplined in overcoming processes that hide evidence, distort understanding, and multiply confusion.

Flourishing based on misunderstanding or deception is unlikely to remain stable.

E1: Make trust responsive to track record

Good decisions are much harder if people are being misled about facts, intentions, or capabilities.

If a source is guilty of deliberate deception, reckless disregard for evidence, or repeated dissemination of demonstrably false claims, this should reduce the degree of trust placed in their subsequent communications. Their public reputation should diminish.

At the same time, safeguards should protect good-faith disagreement, uncertainty, and the correction of honest mistakes.

E2: Expose the assumptions behind decisions

Key decisions should make visible the factual assumptions, forecasts, value judgements, and uncertainty estimates on which they depend.

For example:

• Forecasts should be accompanied by statements of probability
• Claims presented as scientific should, where feasible, identify what evidence would substantially strengthen, weaken, or overturn them.

E3: Make conflicts of interest visible

People and institutions exercising influence over major technological decisions should disclose material financial, organisational, and personal interests that could reasonably affect their judgement.

This includes any significant prior commitments that bear directly on the issue under consideration.

E4: Treat dissent as a resource

Well-informed disagreement should be protected and actively sought, especially when consensus may reflect conformity, intimidation, fashion, or shared blind spots.

Rather than just seeking evidence that would confirm a preferred theory, it’s important to seek disconfirming evidence too.

E5: Protect whistleblowers

Anyone who becomes aware of a serious disconnect between claimed values and actual behaviour should have safe channels to report their concerns and should be protected against retaliation when raising them in good faith.

E6: Value relevant competence

Universal moral standing does not imply that every person has equal expertise on every question. Good decision-making should combine democratic legitimacy with relevant competence.

There is no simple formula for balancing equal political standing against unequal knowledge. Different decisions may require different combinations of participation, representation, expertise, delegation, and appeal.

The required commitment is not to eliminate this tension, but to develop institutions that make the allocation of decision authority explicit, contestable, and appropriate to the kind of question being decided.

E7: Make predictions, observe outcomes, and update

Trust and reputation should depend not merely on confidence, but on demonstrated calibration: how well previous predictions corresponded with subsequent outcomes.

Accordingly, governance arrangements, policies, and technical systems should be monitored against explicit expectations stated in advance wherever possible. When outcomes diverge from predictions, institutions should investigate why, update their models, and change course where necessary.

Without this cycle of prediction, observation, and revision, discussions can lose touch with reality.

Collaborative information systems can provide significant help here by recording forecasts, tracking outcomes, and preserving an auditable history of updates.

E8: Design for human irrationality and motivated reasoning

Better information does not guarantee better decisions. Better evidence does not automatically defeat identity, outrage, motivated reasoning, or manipulation:

• The attention economy may actively undermine epistemic integrity
• People frequently interpret evidence through the lenses of identity, emotion, group loyalty, status, and prior commitment.

Trustworthy decision processes should therefore be designed to reduce predictable cognitive and social distortions – for example through structured disagreement, independent judgement before group discussion, protection against conformity pressures, and deliberate testing of propositions against evidence that could prove them wrong.

Accordingly, it’s important to address not only the supply of reliable information but the social and psychological conditions under which people attend to, interpret, and act upon it.

Governance

The Governance category contains eight commitments.

In short, governance should ensure that power is earned, constrained, and accountable to all.

This requires a separation of powers, with healthy systems of checks and balances which are regularly reviewed and updated when needed.

G1: Hold power accountable

Power tends to corrupt – and absolute power tends to corrupt absolutely.

Power can transform good people into tyrants. Well-intentioned governance systems can morph into overbearing autocracies.

The greater the power exercised by a person, corporation, state, or technical system, the stronger the requirements for scrutiny, challenge, and correction.

Governance arrangements should therefore allow powerful actors and institutions to be effectively challenged. There need to be mechanisms for the removal of leaders that exceed or abuse their mandate.

G2: Anticipate that power will resist accountability

Governance arrangements should not assume that powerful actors will voluntarily accept constraints merely because those constraints are reasonable or socially beneficial. States, corporations, military establishments, wealthy individuals, and entrenched institutions may resist scrutiny, exploit loopholes, shape public narratives, or undermine reforms that threaten their interests.

Trustworthy governance therefore requires appropriate forms of countervailing power, enforceable constraints, independent scrutiny, careful attention to the incentives of actors who may prefer the status quo.

This may require regular updates to the systems of checks and balances:

• Recognising that new technologies and new social practices have altered previous balances
• But being careful not to sweep away venerable processes in the name of modernisation or expediency.

G3: Renew the foundations of accountable governance

Governance arrangements should not assume that the economic, technological, demographic, or social conditions that have historically sustained effective governance will persist indefinitely.

Where emerging changes could weaken accountability, legitimacy, expertise, civic participation, public revenues, or countervailing power, replacement or supplementary foundations should be developed while existing institutions still possess sufficient authority, resources, expertise, and public support to establish them.

Institutional renewal should therefore proceed before deteriorating governance capacity makes such renewal substantially harder or impossible.

Responsibility for this renewal cannot be assigned to “society” in the abstract. It must be allocated among identifiable institutions with the authority, resources, competence, and legitimacy to act, including governments, legislatures, civil society, organised labour, professional bodies, and new forms of public-interest institution.

Proposals should identify how the new arrangements will be funded, how their independence will be protected, and what powers or incentives will make them effective against resistance.

G4: Anticipate systemic failures of incentives

Market mechanisms can coordinate dispersed knowledge remarkably well, but can also generate monopoly, bubbles, correlated risk, algorithmic feedback loops, and crises. AI may intensify both sides.

For example:

• Systematically lower costs of goods created by AI may trigger a deflationary descent which undermines many of the basic assumptions of economies around the world
• Widespread job losses caused by increased automation may switch more value from labour to capital, undermining the tax basis which has supported governance actions.

Other examples of incentive failures include arms races, tragedy-of-the-commons dynamics, principal-agent problems, and competitive pressures that drive AI labs toward unsafe behaviour

The commitment is to preserve the extraordinary coordinating benefits of markets while preventing local incentives from generating systemic fragility.

G5: Ensure procedural fairness

Trust depends not only on whether decisions produce good outcomes, but also on whether the processes leading to them are widely regarded as fair.

People affected by important decisions should have appropriate opportunities to be heard, reasons should be given for consequential choices, comparable cases should be treated consistently, and mechanisms for challenge and appeal should be available.

Technical competence without perceived legitimacy is unlikely to sustain public trust.

G6: Apply subsidiarity where possible and coordinate where necessary

Decisions should be taken close to the people affected – except where externalities, systemic risks, or global consequences require wider coordination.

G7: Preserve institutional and intellectual diversity

Coordination should not require unnecessary uniformity. Where feasible, societies should preserve multiple institutions, analytical approaches, technical architectures, and centres of expertise, so that errors in one do not automatically propagate throughout the whole system.

Diversity can create inefficiency and disagreement, but it can also provide resilience, experimentation, and protection against common-mode failure.

G8: Retain humanity’s ability to correct and constrain technology

At least until much stronger foundations of trust and control exist, high-stakes systems should remain subject to effective correction, constraint, and ultimately withdrawal of authority through accountable human institutions, rather than acquiring open-ended autonomous power to pursue goals in the world.

No system should acquire effectively irreversible authority over humanity merely because it is more intelligent, efficient, persuasive, or strategically capable than individual humans.

More generally: High-stakes systems should remain subject to effective correction and constraint through accountable human institutions. “Human control” is not enough if that control is itself concentrated in unaccountable hands.

Caution

Finally, the Caution category contains six commitments.

In short, caution doesn’t mean fear; it means a responsible attitude toward the future.

Caution means applying methods of foresight to protect and expand the freedom that we can all experience in the future.

Note: Governance asks who may exercise power and how that power is constrained. Caution asks what kinds of decisions should be made when consequences are uncertain or potentially irreversible.

C1: Apply precaution proportionately

The guiding principle is neither “move fast and break things” nor “ban anything uncertain”. The degree of caution should rise with the scale, irreversibility, and uncertainty of possible harm:

• Low risk of harm requires a low level of caution
• Check more carefully and more urgently when stakes are high or uncertain
• Avoid both paralysis and recklessness.

C2: Take tail risks seriously

Many familiar risk calculations implicitly assume that extreme outcomes are extraordinarily rare. But where systems contain strong interdependencies, feedback loops, or common-mode vulnerabilities, distributions can have “fat tails”: extreme events may be much more likely than extrapolation from ordinary fluctuations suggests.

In practical terms, this means any plans for the future should beware the creation of monocultures that lack sufficient diversity – cultures in which all the variations can move in the same direction at once. We should also beware the influence of hidden connections, such as the shadow links between multiple different financial institutions that precipitated the global financial crisis of 2008.

Emerging forms of AI-driven trading and financial automation may create additional risks of correlated behaviour, feedback loops, and instability operating faster than human supervisors can understand or interrupt. Innovations should therefore be assessed not only for the efficiency they create in normal conditions, but for the systemic behaviour that can emerge under stress.

C3: Demand stronger evidence of safety for catastrophic risks

Emerging technologies can create unprecedented concentrations of capability and control. Arrangements that would become extremely difficult to challenge, reverse, or escape should face an exceptionally high burden of proof.

Where a plausible failure mode could cause irreversible civilisation-scale harm, ordinary standards of product testing and post-deployment correction are insufficient.

The burden of demonstrating adequate safety should rise with the potential scale of catastrophe.

C4: Don’t make technology safe; make safe technology

The previous principle concerns the burden of proof. This one concerns development methodology.

Some particularly dangerous technologies – including biopathogens and misaligned AI – could escape into the wild even during a test phase inside a supposedly secure environment.

In such cases, safety should, as far as possible, be built into architectures and development processes from the outset, and then subjected to rigorous independent testing.

Therefore, instead of accepting a design process where the outcome is opaque and the quality needs to be assessed by an extensive test phase, the commitment in cases of potential ruin is to follow a design process where the quality is baked in from the beginning, transparently and securely.

C5: Prefer corrigibility and reversibility under uncertainty

Where uncertainty is substantial and the consequences of error could be serious, prefer interventions that can be monitored, corrected, halted, or reversed.

C6: Preserve future freedom of choice (avoid lock-in)

Avoid prematurely locking civilisation into institutions, technologies, values, or power structures that future people cannot escape.

That’s a risk whose true danger only becomes apparent later – so it requires especial vigilance.

Conclusion: Enabling a growing spiral of trust

We do not need universal agreement on every ethical controversy before we can improve collective decision-making. We need a sufficiently robust initial zone of agreement and practice – a set of principles that diverse people can endorse for different reasons, combined with procedures and behaviours that can begin to earn warranted trust. From that base, we can:

• Build institutions and tools that deserve a limited degree of trust
• Develop the legitimacy, incentives, and leverage needed for those institutions to influence increasingly powerful actors
• Use these institutions and tools to scrutinise more powerful systems
• Where justified by evidence, expand the circle of warranted trust.

The aim is not ever-expanding blind trust. It is ever-expanding warranted, bounded, contestable, and revisable trust.

Trustworthiness depends less on never making mistakes than on detecting, acknowledging, and correcting mistakes rapidly.

In short, trustworthy governance is governance that remains corrigible.

That can enable a growing spiral of trustworthy technoprogressive governance.

The principles described above are offered not as the final word, but as a starting point: a foundation to be challenged, tested, improved, and – where it earns sufficient confidence – put into practice.

Endnote: Requesting feedback

Here are some questions where I would particularly value feedback from reviewers:

1. Which principle did you most disagree with, and why?
2. Which principle surprised you by seeming original or insightful?
3. Which principle felt too vague?
4. Which principle, if removed, would weaken the framework the least?
5. Which two principles overlap enough that they should be combined?
6. Is there an important commitment that is missing entirely?
7. Does the overall structure feel coherent, or merely like a list?

Please feel free to answer just one of these questions – or any selection from them.