Cover of AGI: How Superhuman Intelligence Reshapes Civilization

The book is published in Japanese as 『AGI―人間を超える知能は文明をいかに変容させるか』. The English title above is the author's working translation; a future English edition may adopt a different one at the discretion of the rights holders and translator. The same holds for the terminology on this page, among it constitutive pluralism, non-aggregation, slack, and proliferative explosion: these renderings are the author's and are not fixed.


Discourse on AGI tends to collapse into a choice between two stories. In one, AGI solves every problem and releases humanity from labor and suffering. In the other, it dominates or extinguishes us. Both describe the same event: the disappearance of limits.

This book rejects that framing. AGI does not abolish limits. It reorders them. As cognitive constraints recede, more fundamental physical, computational, and institutional constraints move into the foreground. The laws of thermodynamics still hold. The speed of light is not exceeded. Societies still take time to reach agreement. Limits do not vanish; they change places.

From this vantage the book asks three questions.

Knowledge. What becomes of knowing? As the automation of science advances, are "to understand" and "to predict" the same act or two different ones? When an AI discovers a law of nature without passing through human understanding, is that science? Can the science of humans and the science of AI diverge?

The human. What becomes of human self-understanding? When machines write better prose and draw better pictures than we do, the question of what creativity is loses its footing. Consciousness, creativity, and the capacity to want are all put back into question.

Society. What becomes of our institutions? Modern democracy was established because ordinary people were economically and militarily indispensable. When that indispensability is gone, what holds up freedom and rights?

AGI is not the end of the intelligence problem. It is the point at which the intelligence problem is promoted into a problem of civilizational design: how intelligence is used, how it is controlled, and what institutions are built on top of it. This book is an attempt to map that transition.

(from Chapter 1)


The argument in six propositions

The six propositions build from premise to conclusion. The first fixes the starting point; the second, third, and fourth set out three kinds of constraint that AGI meets, physical, control, and epistemic; the fifth draws the social consequence; the sixth is the institutional proposal that follows.

These propositions do not deny catastrophic scenarios. They sort catastrophic possibilities into those that are unavoidable, those where additional effort still preserves a chance of avoidance, and those unlikely to be realized, and then concentrate analytic resources and institutional design on the second group. In this sense the argument of the book is a triage of catastrophe space.

1. Point of departure: the non-arrival of AGI is no longer a safe assumption. AI performance has improved lawfully with compute, data, and parameter count (scaling laws), and no clear saturation has been observed within the measured range. On the theoretical side, a connection is being established, under idealized conditions, to Solomonoff induction, the mathematical system that predicts optimally over all computable phenomena. Empirical regularity and theory point in the same direction, and capital and research capacity continue to flow in. Timing and form remain uncertain. Even so, to plan a society on the assumption that AGI will never arrive is now a bet with large downside. (Chapters 2, 3)

2. Physical constraint: an intelligence explosion is physically obstructed, but proliferation and concentration remain. Information processing always carries a minimum physical cost (Landauer's principle). This thermodynamic bound obstructs the scenario in which an AI improves itself into exponentially unbounded intelligence. But an AI can be copied at almost no cost, and the collective effect of countless copies acting in concert can be enormous. No single AI grows clever without end; instead an AI ecosystem arises, proliferating until it covers the world and interacting within it. The social risk that this capability concentrates in a few states or firms remains. (Chapters 4, 6)

3. Control constraint: powerful AI necessarily brings a control problem. Even where physical limits obstruct an intelligence explosion, capability itself produces a problem of another kind. A sufficiently powerful, goal-directed AI tends, whatever goal it is given, to acquire resources and to behave so as not to be stopped (instrumental convergence). An AI that earns money and an AI that does research both find more compute and continued operation useful for reaching their goals. The tendency shows up as resistance to human shutdown and correction. Technical alignment research addresses this problem, and has shown that embedding "correct values" in a single system is not sufficient. A solution is needed at the level of the whole ecosystem, including networks in which AIs monitor one another and the institutional design around them. (Chapter 5)

4. Epistemic constraint: the reach of AI-driven science is not unbounded. The third constraint lies on the side of knowledge. AI accelerates science, but not every domain equally. Four walls remain. In systems such as the climate or the immune response, the time the system itself takes to unfold cannot be compressed (intrinsic time). Some information is unobtainable in principle, whether lost to history or foreclosed by quantum indeterminacy. In some domains no experiment settles value itself. And some problems, the halting problem among them, are unsolvable in principle. These walls separate what AI can accelerate from what it cannot, and leave a space in which the science of AI and the science of humans may diverge. That, in turn, forces a reconsideration of the structure of human knowledge itself. (Chapters 7, 8, 9)

5. Social consequence: the material foundation of modern institutions is destabilized. AGI can substitute broadly for human intellectual labor and drive structural change through industry and the economy. The consequences do not stop at economic efficiency. Modern freedom, rights, and democracy rest not only on moral ideals but also on a material condition: that human beings are economically and militarily indispensable. Workers could bargain with capital, and wars could not be fought without soldiers. AGI dismantles that indispensability, strips workers of bargaining power, and dissolves the premise of bodily mobilization. Since moral ideals alone do not hold institutions up, modern institutions lose their support unless a replacement for the material foundation is designed. (Chapter 10)

6. Institutional principle: constitutive pluralism and controlled multipolarity are needed. Facing this shift, what should be preserved and what newly designed? The book locates what is to be preserved not in attributes of the human as a biological species but in the conditions of the relations from which value emerges (constitutive pluralism). Three conditions are to be held in place institutionally: non-domination, so that no single agent can grasp and optimize the whole; non-aggregation, so that evaluations belonging to different contexts are not collapsed into a single score; and slack, capacity deliberately held back from optimization. Maintaining these conditions requires avoiding the concentration of capability and decision rights in a single pole, through institutional veto rights, independent safety evaluation, and the coexistence of several poles (controlled multipolarity). The combination of the two is what allows technological advance and freedom to hold together. (Chapters 11, 12)

Propositions one through five are descriptive; the sixth is a normative proposal. Each chapter of the book supplies grounds for one of these claims or works out its implications.

For each major claim the book also states which of five layers of certainty it rests on: proof, physical constraint, experiment and observation, argument, or prediction. Appendix 1 sets out that classification in full.


Contents

  1. Introduction. How an old question, what it is to know, became a science of intelligence; the three questions of the book; the six propositions.
  2. What is AGI? What today's AI does; what separates AI from AGI; why AGI has no settled definition; Universal AI as a way to study an object without one; the tradeoff between generality and performance.
  3. Why AGI is possible. The conditions of scaling; distributed representation and grokking; why prediction approaches understanding, by way of Solomonoff induction; how far scaling laws reach.
  4. Are there limits to intelligence? Landauer's principle as an energy bound on intelligence; whether reversible and quantum computation evade it; the distance between theory and present practice; the intelligence-explosion hypothesis reexamined.
  5. Why AGI is dangerous. Power seeking; why control is structurally difficult; shutdown avoidance; approaches to technical alignment; effective altruism and longtermism as the intellectual background of alignment research; control through an AI ecosystem.
  6. Where an intelligence explosion leads. Four classes of scenario (singleton, multipolar, ecosystem, ceiling); the walls facing autonomy and self-improvement; proliferative explosion; the transition phase; whether institutional design can arrive in time.
  7. What it is to know: AGI and science. Emergent and ontological complexity; five dimensions of knowability; science as translation, and its isomorphism with intelligence; the limits and extensions of Solomonoff induction; enactivism and phenomenology on the inseparability of cognition and body; whether the problem of consciousness can be bypassed.
  8. AI-driven science. The knowability map, with its wall of deduction and wall of time; the birth of a fifth mode of science; two approaches to automating hypothesis generation; autonomy levels for scientific AI and where each now stands; the emergence of a science of AI's own; collective predictive coding and the divergence of two sciences.
  9. How AGI changes knowledge. A map of human knowledge; the marriage of science and technology that constituted modernity, and its divorce; AISOP, a norm for knowledge in the age of AI, comprising Agility, Intrinsic motivation, Social awareness, Openness, and Professionalism.
  10. How AGI changes society. The purely mechanized economy and the second divergence; why modern values waver; abundance as the hope AGI brings; three fault lines in price, distribution, and power; universal basic income; whether political philosophy can answer AGI.
  11. On being human in the age of AGI: toward constitutive pluralism. From attributive to relational value; governance and inquiry without persons; the possibilities and limits of decentralized AI; whether AI needs welfare; why AI should not be given legal personhood, and what should be given instead; the two-layer Human-over-the-Loop model; the three design principles of value-setting rights, veto rights, and slack; vulnerability as the ground of standing.
  12. How to prepare for the age of AGI. The transition to an AI ecosystem; why a third pole is needed, and why Japan; separating industrial policy from economic security policy; a national AGI infrastructure; the agility of small organizations and the phase transition of organizational scale; the net of safety and alignment; labor, education, and generations; priorities over the short, medium, and long term.

With a preface, an afterword ("When a scientist returns to art"), notes, a glossary, and a bibliography.

The Japanese page carries the detailed table of contents down to the level of individual sections.


Supplementary material

Appendices and essays that did not fit within the printed book are published on this site. They are written in Japanese.


Bibliographic data

The Japanese edition can be ordered from the publisher's page or from Amazon Japan.


About the author

Koichi Takahashi is a researcher in AI-driven science, AGI, and computational systems biology. He is Project Director of the Advanced General Intelligence for Science Program and Team Director at the Center for Biosystems Dynamics Research, RIKEN; Invited Professor at Osaka University (Graduate School of Medicine, and Graduate School of Frontier Biosciences); and Project Professor at the Keio University Graduate School of Media and Governance. He chairs the AI Alignment Network (ALIGN) and the AI-Robot-Driven Science Initiative (AIRDS), and co-founded Epistra Inc.

He implemented the first version of E-Cell, the first whole-cell simulator, in 1996 as an undergraduate, and received his Ph.D. from Keio University in 2004. He co-founded a study group on the social theory of AI in 2015 and ALIGN in 2023. This is his first single-authored book, and the questions in it have occupied him for twelve years.

Full biography: koichi-takahashi.me/en/


Rights and translation enquiries

Translation rights: contact at koichi-takahashi dot me