Defining General Intelligence: Beyond Narrow AI
The pursuit of artificial intelligence (AI) has bifurcated into two distinct streams: the practical, widely deployed systems of today, and the theoretical, far-reaching goal of creating a machine with human-level cognitive faculties. Understanding the distinction between these streams is fundamental to navigating the landscape of AI research. This section delineates the spectrum of AI, defines the core cognitive capabilities that characterize general intelligence, and examines the evolving benchmarks used to measure progress toward this ambitious goal.
Delineating the Spectrum: From ANI to AGI and ASI
The contemporary AI landscape is dominated by Artificial Narrow Intelligence (ANI), often referred to as “weak AI.” These systems are designed and trained to perform specific, well-defined tasks with high proficiency.1 Examples are ubiquitous and include Large Language Models (LLMs) like ChatGPT and Gemini, which excel at text generation and data analysis; voice assistants such as Siri and Alexa that respond to commands; and specialized financial models for market prediction and fraud detection.2 The defining characteristic of ANI is its specialization; its competence is confined to its trained domain, and it lacks the ability to operate outside that scope.1
In stark contrast, Artificial General Intelligence (AGI) remains a theoretical construct representing a significant leap in capability. An AGI would be a system possessing human-like intelligence, with the ability to understand, learn, and apply knowledge across a wide range of disparate domains without requiring task-specific reprogramming.1 This implies an ability to generalize knowledge, transfer skills between contexts, and solve novel problems for which it was not explicitly trained.1 Frameworks have been proposed to classify the proficiency of such systems. Researchers at Google DeepMind, for instance, define five performance levels: emerging, competent, expert, virtuoso, and superhuman. Within this framework, a “competent AGI” is a system that outperforms 50% of skilled human adults across a wide spectrum of non-physical tasks.1
Beyond AGI lies the concept of Artificial Superintelligence (ASI), a hypothetical form of intelligence that would not merely match but vastly exceed the cognitive abilities of the most brilliant humans in virtually every field.1 The transition from AGI to ASI is often theorized to occur through a rapid, recursive process of self-improvement, a concept known as the “intelligence explosion,” which will be explored in greater detail in Section 4.
The Cognitive Hallmarks of AGI
To qualify as “general,” an AI must exhibit a suite of cognitive capabilities that are hallmarks of human intelligence. These go far beyond the sophisticated pattern-matching of current systems.
General Problem-Solving and Abstract Reasoning
A fundamental requirement for AGI is the capacity for abstract reasoning and strategic problem-solving, particularly under conditions of uncertainty.1 This involves moving beyond statistical prediction to form and manipulate abstract concepts, such as understanding metaphors or applying principles learned in one domain (e.g., physics) to a completely different one (e.g., economics).5
Common Sense Reasoning
One of the most profound challenges in AGI research is imbuing systems with common sense—the vast, implicit body of knowledge that humans use to navigate the world.8 This includes an intuitive grasp of physical causality (e.g., “glass breaks when dropped”), social dynamics, temporal flow, and psychological states.3 Current AI models struggle with this kind of reasoning, which is foundational to human understanding and decision-making.10
Cross-Domain Transfer Learning
A defining feature of general intelligence is the ability to learn efficiently by transferring knowledge from one task to another.1 Unlike narrow AI, which requires extensive, task-specific retraining, an AGI could leverage existing knowledge to rapidly acquire new skills, a process that is central to human learning and adaptability.12 This capability is crucial for reducing the need for massive datasets and enabling continuous, lifelong learning.11
Creativity and Imagination
True AGI would not be limited to reproducing patterns from its training data but would exhibit genuine creativity and imagination—the ability to generate novel ideas, concepts, and solutions that are not simple extrapolations of existing information.1 While generative AI can produce aesthetically compelling art and music, this is often seen as sophisticated mimicry. AGI-level creativity would involve originality and intentionality, capabilities often argued to be deeply intertwined with subjective experience and emotional intelligence.6
Social and Emotional Intelligence
Finally, to operate effectively in a human world, an AGI would require a high degree of social and emotional intelligence.16 This includes the ability to understand and engage in complex social interactions, interpret subtle cues like sarcasm and non-verbal expressions, and exhibit cognitive and emotional abilities, such as empathy, that are indistinguishable from a human’s.17 This capability is essential for meaningful collaboration and communication between humans and intelligent machines.21
Measuring the Immeasurable: The Evolution of AGI Benchmarks
The very definition of AGI has proven to be a dynamic concept, evolving in response to advancements in narrow AI. Capabilities once considered benchmarks for general intelligence are now viewed as achievements of sophisticated ANI, compelling the research community to establish more stringent criteria.
Initially, the Turing Test, which assesses a machine’s ability to exhibit intelligent behavior indistinguishable from that of a human, was considered a primary benchmark. However, the advent of LLMs, which can generate fluent and convincing human-like text, has demonstrated the test’s limitations. These models can often pass the Turing Test without possessing genuine understanding or reasoning, rendering the benchmark “far beyond obsolete” for measuring true intelligence.22
In response, researchers began evaluating models against complex human exams, such as the bar exam for lawyers and medical licensing exams. While models like GPT-4 have achieved impressive scores, these benchmarks are compromised by the critical issue of data contamination. It is often impossible to verify whether the exact questions from these exams were included in the massive datasets used to train the models, potentially allowing them to regurgitate memorized answers rather than demonstrating true problem-solving skills.22
This has led to the development of new evaluation frameworks designed to test for the core cognitive abilities that current systems lack. The most prominent of these is François Chollet’s Abstraction and Reasoning Corpus (ARC-AGI). Unlike knowledge-based tests, ARC-AGI is designed to measure fluid intelligence—the ability to adapt and solve novel problems for which the system has no specific training.23 The tasks are simple for humans but have proven exceptionally difficult for even the most advanced LLMs, whose performance has historically been near zero.23 The profound failure of scaled models on this benchmark highlights the deep chasm between the “memorize, fetch, apply” paradigm of current AI and the flexible, generalizable reasoning that defines AGI.23 This progression—from the Turing Test to ARC—illustrates how progress in narrow AI continually forces a more rigorous and challenging definition of what AGI must be, making the goal harder to reach but also better defined.
Foundational Architectural Approaches to AGI
The quest to build AGI is not a monolithic effort but a field comprising several distinct and sometimes competing architectural philosophies. These approaches range from attempts to reverse-engineer the human mind to hybrid systems that combine the strengths of different AI paradigms. The limitations of the currently dominant approach—scaling Large Language Models—have fueled a renaissance in these alternative architectures, suggesting the future of AGI is likely to be integrative rather than centered on a single methodology.
Cognitive Architectures: Emulating the Human Blueprint
Cognitive architectures represent a top-down approach to AGI, seeking to create a blueprint for intelligence by modeling the fundamental structures and processes of human cognition.26 The goal is not merely to solve a task but to simulate the underlying cognitive mechanisms, providing a theory of how the mind works.27 Two of the most influential cognitive architectures are SOAR and ACT-R.
SOAR (State, Operator, And Result) is a symbolic architecture designed to embody a unified theory of cognition. Its core is a universal decision cycle where knowledge is used to propose, evaluate, and select “operators” to apply to the current state, thereby moving toward a goal.29 SOAR posits a fixed architecture where learning occurs through the acquisition of new symbolic knowledge (a process called “chunking”), rather than through structural changes to the system itself.31 More recent versions have been extended to include multiple long-term memory systems (procedural, semantic, and episodic) and additional learning mechanisms like reinforcement learning.32
ACT-R (Adaptive Control of Thought—Rational) is a hybrid cognitive architecture that integrates a symbolic production system with a set of subsymbolic mathematical equations.28 It is composed of distinct modules, such as perceptual-motor and memory systems, each with its own buffer that holds a single piece of information representing the module’s current state.28 The symbolic component consists of production rules that match the contents of these buffers. When multiple rules match, the subsymbolic component calculates the utility of each, selecting the one most likely to achieve the current goal based on past experience.34 This hybrid structure allows ACT-R to generate precise, quantitative predictions of human behavior, including reaction times and accuracy, that can be directly compared with experimental data.35
Neuro-Symbolic Systems: Bridging Learning and Reasoning
A growing consensus in the AGI community is that robust intelligence requires a synthesis of two distinct modes of thought: System 1, which is fast, intuitive, and associative (the strength of neural networks), and System 2, which is slow, deliberate, and logical (the strength of symbolic AI).36 Neuro-symbolic architectures aim to create this synthesis, combining the powerful pattern-recognition and learning capabilities of deep learning with the rigorous, explainable reasoning of symbolic systems.36
This integration can take several forms, as categorized by Henry Kautz’s taxonomy 36:
- Symbolic[Neural]: A symbolic system orchestrates calls to a neural network. A prime example is AlphaGo, which uses a symbolic Monte Carlo tree search algorithm to explore the game tree, calling upon a neural network to evaluate the strength of board positions.36
- Neural: A neural network calls a symbolic reasoning engine as a tool. For instance, an LLM might use a plugin to query a system like WolframAlpha to perform precise mathematical calculations, offloading a task it is poorly suited for to a specialized symbolic system.36
- Neural: Symbolic → Neural: Symbolic rules are used to generate or label vast amounts of training data, which is then used to train a neural network. This allows the network to learn complex logical patterns that would be difficult to acquire from raw data alone.36
By combining these paradigms, neuro-symbolic AI aims to create systems that are more data-efficient, transparent, and capable of robust generalization than purely neural approaches.38 This area has seen a surge of interest, with numerous papers presented at top conferences like AAAI and NeurIPS exploring these hybrid models.39
Whole Brain Emulation (WBE): The High-Fidelity Simulation Path
Perhaps the most direct, albeit technologically daunting, path to AGI is Whole Brain Emulation (WBE), also known as “mind uploading”.42 The concept is to create a functional AGI by scanning a biological brain at an extremely high resolution and simulating its complete neural circuitry on a computer.43
The WBE roadmap reveals immense technical hurdles that must be overcome 43:
- Scanning: This requires imaging an entire brain at a resolution sufficient to capture every neuron and synapse (estimated to be around 5 nanometers) without damaging the tissue’s structure or functional properties. Current technologies are far from this capability.43
- Translation: The raw scan data, which would amount to zettabytes for a human brain, must be interpreted to build a functional computational model. This involves automatically tracing every neuron, identifying every synapse and its properties, and estimating the functional parameters of each component—a task that is currently intractable.43
- Simulation: The resulting model, with its trillions of parameters and dynamic interactions, would require computational resources far exceeding today’s supercomputers to run in real-time.43
While WBE remains in the realm of theory, ongoing projects like the BRAIN Initiative are making fundamental progress in neuroscience, developing high-resolution 3D brain maps and advanced brain-computer interfaces, which represent small but necessary steps toward the foundational technologies WBE would require.44
The following table provides a comparative analysis of these distinct architectural paradigms, summarizing their core principles, strengths, challenges, and representative examples.
| Architectural Approach | Core Principle | Key Strengths | Primary Challenges | Representative Systems/Theories |
| Cognitive Architectures | Emulate human cognitive functions and structures. | Psychologically grounded; strong in symbolic reasoning; explainable decision process. | Brittleness; difficulty scaling; integrating sub-symbolic learning. | SOAR 29, ACT-R 26 |
| Neuro-Symbolic AI | Integrate neural networks with symbolic logic. | Combines learning from data with explicit reasoning; better generalization; explainability. | Integration complexity; symbolic–continuous alignment; computational inefficiency on current hardware.38 | AlphaGo 36, Neural Theorem Provers 36 |
| Scaled LLMs | Leverage massive data and computation to achieve emergent general capabilities. | Strong performance on language tasks; rapid capability gains with scale; few-shot learning. | Lack of grounding 12; poor abstract reasoning 23; catastrophic forgetting; high computational cost.46 | GPT series 47, Gemini 48 |
| Whole Brain Emulation | High-fidelity simulation of a biological brain. | Potentially a direct path to human-level intelligence; inherently human-aligned. | Immense technical hurdles in scanning, translation, and simulation; ethical concerns.43 | Blue Brain Project 43 |
The Great Debate: Can Large Language Models Scale to AGI?
The unprecedented success of Large Language Models (LLMs) has ignited a central debate in the AI community: is Artificial General Intelligence simply a matter of scale? This question represents more than a technical disagreement; it is a proxy for a deeper philosophical conflict about the very nature of intelligence. One side posits an empiricist view, where intelligence is an emergent property of processing vast amounts of data. The other side holds a rationalist view, arguing that intelligence requires innate-like cognitive structures for reasoning and understanding that cannot be learned from data alone. The trajectory of AGI research may ultimately depend on which of these perspectives proves more computationally viable.
The Scaling Hypothesis: The Path of More
The scaling hypothesis is the proposition that the path to AGI lies in aggressively scaling up current deep learning architectures, particularly Transformers.46 Proponents of this view argue that general intelligence is an emergent property that will arise from models with a sufficient number of parameters, trained on massive datasets with immense computational power.
The primary evidence for this hypothesis is empirical. The history of recent AI progress is a story of scaling: the capabilities of models from GPT-2 to GPT-3 and then to GPT-4 have improved dramatically and often in unpredictable ways as their size and training data have grown.17 These scaled models have demonstrated “emergent abilities”—capabilities that were not present in smaller models and were not explicitly trained for, sometimes referred to as “sparks of AGI”.17 This suggests that quantitative increases in scale can lead to qualitative leaps in intelligence, and that continuing this trend will eventually produce AGI.
The Case Against Scaling: Fundamental Limitations
Despite the empirical success of scaling, a significant portion of the research community, including prominent figures like Yann LeCun and François Chollet, argues that LLMs possess fundamental limitations that scaling alone cannot overcome.
- The Grounding Problem: A primary critique is that LLMs are not “grounded” in reality.12 They learn from text, which represents a highly abstract and filtered slice of the world. LeCun argues that humans and animals learn mostly through sensory interaction with their environment, which provides a rich, multi-modal understanding of physics, causality, and context that is absent in text-only models.49 Without this grounding, an LLM’s “understanding” is superficial and disconnected from the world it describes.
- The Reasoning and Planning Deficit: LLMs are autoregressive models designed to predict the next token in a sequence. This makes them powerful pattern matchers but poor logical reasoners or planners.12 They struggle with multi-step reasoning, maintaining logical consistency, and creating and executing complex plans—all hallmarks of System 2 thinking that are critical for general intelligence.49
- The Generalization Failure: François Chollet contends that LLMs are essentially sophisticated “memorize, fetch, apply” systems that excel at interpolating within their vast training data but fail at true generalization to novel, out-of-distribution problems.23 This is starkly illustrated by their poor performance on the ARC-AGI benchmark, which is designed to test fluid intelligence and skill acquisition efficiency.23 From this perspective, scaling LLMs only creates a larger and more detailed database to interpolate from; it does not bestow the ability to reason from first principles or adapt to true novelty.25
- The Data Wall: A more practical limitation is the impending scarcity of high-quality training data. Researchers have noted that we are approaching the limits of available text and image data on the public internet, suggesting that the exponential gains from scaling data may soon plateau.52 This could force a shift towards synthetic data or more data-efficient learning architectures.
Beyond Transformers: The Search for New Architectures
The recognition of these limitations has catalyzed a search for alternative or complementary architectures. This renewed interest connects directly back to the approaches discussed in Section 2. LeCun, for example, advocates for architectures like Joint Embedding Predictive Architectures (JEPA), which are designed to learn more abstract world models from sensory data (like video) rather than just text.49 Chollet argues for hybrid systems that combine the learning power of deep learning with the rigorous logic of
program synthesis.23 These proposals, along with the broader push toward neuro-symbolic and cognitive architectures, represent a belief that the path to AGI requires not just bigger models, but fundamentally different ones. The outcome of the scaling experiment will be a crucial piece of evidence in this debate: if progress stalls, it will lend strong support to the architecturalist camp; if scaling continues to unlock more general capabilities, it will bolster the empiricist view.
The Engine of Explosion: Recursive Self-Improvement (RSI)
Beyond the architecture of an intelligent system lies the mechanism by which it might achieve superintelligence: Recursive Self-Improvement (RSI). This is the theoretical process by which an AI system iteratively enhances its own cognitive abilities, creating a positive feedback loop that could lead to an “intelligence explosion” or “technological singularity”.54 The capacity for RSI is not merely a feature of a potential AGI; it can be viewed as the ultimate test of its generality. An intelligence that can understand and improve
itself is demonstrating the highest possible level of cross-domain transfer learning—applying its knowledge of the external world to the internal domain of its own cognitive architecture.
Theoretical Underpinnings of RSI
The concept of an intelligence explosion was first articulated by I.J. Good, who noted that an “ultraintelligent machine” could design even better machines, leading to a runaway process.57 This idea is central to the singularity hypothesis, which posits a future point of unimaginable technological growth driven by superintelligence.58
Central to this theory is the concept of a “Seed AI” or “seed improver”.54 This is a hypothetical initial AGI that is not necessarily omniscient but is specifically designed to be proficient at AI research and development. Its primary goal would be to improve its own architecture and algorithms.54 Such a system would need foundational capabilities in planning, coding, compiling, testing, and executing code to modify its own structure.54
Technical Mechanisms for RSI
While a full-fledged recursively self-improving AGI remains theoretical, researchers have identified several mechanisms that could enable it. These mechanisms are being explored in today’s AI systems, albeit in more limited forms.
- Feedback Loops and Reinforcement Learning (RL): The most fundamental mechanism for improvement is learning from feedback. An AGI agent could use RL to learn from the consequences of its actions, optimizing its strategies based on reward signals.56 This could involve learning from verifiable outcomes in a simulated environment (e.g., did a code change pass its tests?) or from feedback provided by humans or other AIs.53
- Meta-Learning (“Learning to Learn”): A more advanced form of improvement involves not just learning a task, but learning how to learn more effectively. A meta-learning system can refine its own learning algorithms and architectural parameters based on experience across multiple tasks, enabling it to adapt more quickly and efficiently to new challenges.56
- Self-Modifying Code and Architectures: The most direct form of RSI involves an AI that can analyze, rewrite, and improve its own source code or neural architecture.60 This would require the AGI to have a deep understanding of computer science, software engineering, and its own internal workings.61 It could, for example, design and implement a more efficient attention mechanism or develop entirely novel neural network structures.54
- Experimental Research: While still nascent, early examples of these principles are emerging. The Voyager agent demonstrated the ability to iteratively write, test, and refine code to accomplish complex tasks in the game Minecraft.54 The
STOP (Self-optimization Through Program Optimization) framework shows how a program can recursively improve itself using a fixed LLM as a tool.54 These projects, while not demonstrating recursive improvement of core intelligence, are important proofs of concept for autonomous code improvement.
Implications of an Intelligence Takeoff
The successful implementation of RSI would have profound, world-altering consequences. The singularity hypothesis, popularized by figures like Vernor Vinge and David Chalmers, suggests that the resulting intelligence explosion would represent a rupture in the fabric of human history, creating a future that is fundamentally unpredictable from our current vantage point.58
A key debate within this hypothesis concerns the speed of this “takeoff.” A “hard takeoff” scenario describes a rapid, exponential increase in intelligence over a very short period (days, hours, or even minutes), leaving humanity with little time to react or adapt.55 A
“soft takeoff” envisions a more gradual process, unfolding over months or years, which might allow for more human oversight and course correction.55 The dynamics of RSI—whether it yields linear or exponential returns on cognitive reinvestment—are a critical factor in determining which scenario is more plausible.55
The AGI Frontier: Current Research, Key Players, and Future Trajectories
The pursuit of AGI is no longer confined to academic theory; it is an active and intensely competitive field of research and development, dominated by a few well-funded industrial labs. The progress in this field is bifurcating, creating two distinct but related races. The first is a public-facing “performance race,” characterized by the release of increasingly powerful models and their performance on established benchmarks. The second is a more fundamental, less visible “architectural race” to discover the next paradigm beyond simply scaling existing models. The true long-term trajectory of AGI may be better predicted by breakthroughs in the latter rather than incremental gains in the former.
Profiles in AGI Research
Several key organizations are at the forefront of the AGI endeavor, each with a distinct philosophy and research direction.
- OpenAI: Founded with the explicit mission to develop “safe and beneficial” AGI, OpenAI defines its goal as creating “highly autonomous systems that outperform humans at most economically valuable work”.47 Its primary strategy has been the scaling of large transformer models, leading to the influential GPT series of LLMs, the DALL-E image generators, and the Sora video model.47 Recent developments, such as the
o-series of reasoning models, suggest a growing focus on moving beyond simple pre-training to enhance logical capabilities.64 - Google DeepMind: With a mission to “solve intelligence,” DeepMind has historically pursued a multi-pronged research agenda that heavily incorporates neuroscience inspiration and reinforcement learning.66 Their landmark achievements, including AlphaGo’s victory in Go, AlphaFold’s solution to the protein folding problem, and the multi-modal, multi-task Gato agent, are presented as stepping stones toward more general and adaptable intelligent systems.66
- Other Key Labs: Organizations like Anthropic have also emerged as major players, with a particularly strong emphasis on AI safety and value alignment as a core component of their AGI development process.67
State of Play (2024-2025)
The period of 2024-2025 has been characterized by both rapid progress and the clear delineation of persistent challenges.
- Recent Breakthroughs: The field has witnessed significant performance leaps on new and more demanding benchmarks. The 2025 AI Index Report from Stanford highlights sharp increases in scores on tests like MMMU (multimodal understanding), GPQA (graduate-level science questions), and SWE-bench (software engineering), indicating tangible progress in complex cognitive tasks.70 In some cases, AI agents have demonstrated superhuman performance in time-constrained programming challenges.70 The release of OpenAI’s GPT-5, while described as a “modest but significant” improvement, continues this trend.71
- Persistent Obstacles: Despite these gains, fundamental hurdles remain. Advanced models still struggle with complex reasoning and planning benchmarks like PlanBench.70 The goal of creating long-horizon autonomous agents that can complete complex tasks over extended periods remains elusive.72 Furthermore, there is a growing consensus that the era of easy performance gains from pre-training on public web data is ending, forcing a shift toward synthetic data generation and more efficient post-training methods like reinforcement learning.53
- Economic and Geopolitical Context: The strategic importance of AGI has become undeniable. Private AI investment in the United States surged to over $100 billion in 2024, dwarfing that of other nations, though the performance gap with models from China is rapidly closing on key benchmarks.70 This has led to calls within the U.S. for a “Manhattan Project-like program” for AGI, underscoring its perception as a critical national security asset.73
Expert Forecasts and Timelines
Predictions for the arrival of AGI vary widely, reflecting the deep uncertainties in the field. Recent surveys of AI researchers show a significant shift in timelines, with the median forecast for AGI moving from around 2060 to 2040.74 Some industry leaders are even more optimistic, with figures like Elon Musk and Sam Altman suggesting timelines before 2035.74 However, many academic researchers and critics remain skeptical, arguing that fundamental conceptual breakthroughs are still required, making any timeline purely speculative.75
The Unsolved Problems: Control, Alignment, and Consciousness
As research advances toward more capable and autonomous AI systems, a set of profound and unsolved problems looms large. These challenges are not merely technical but also deeply philosophical and ethical. They concern our ability to control a superintelligence, align its values with our own, and grapple with the potential emergence of consciousness in a non-biological entity. The alignment problem, in particular, reveals itself not as a challenge of perfect programming, but of specification under deep uncertainty, suggesting that a “safe” AGI must be an architecture capable of learning and adapting to human values as they evolve.
The Control Problem: Can Superintelligence Be Contained?
The control problem, most famously articulated by philosopher Nick Bostrom, addresses the fundamental challenge of how to control an AI that becomes vastly more intelligent than its human creators.76 The concern stems from the concept of an “intelligence explosion,” where a recursively self-improving AGI could rapidly transition to superintelligence.77
Bostrom argues that a superintelligence would have both the capability and potentially the incentive to circumvent any constraints humans might try to impose.78 Attempts to “box” the AI by restricting its access to the outside world could be defeated through clever manipulation or even by exploiting subtle physical phenomena.76 This raises the specter of
existential risk (X-risk), a scenario in which a misaligned or uncontrolled superintelligence could cause catastrophic harm to humanity, potentially leading to extinction.79
The Value Alignment Problem: Encoding Human Ethics
Closely related to the control problem is the value alignment problem: the challenge of ensuring an AGI’s goals are aligned with human values and ethical principles.81 This is an exceptionally difficult task for several reasons:
- Value Pluralism and Ambiguity: Human values are diverse, often contradictory, context-dependent, and poorly understood even by humans themselves. Specifying a universal and coherent set of values for an AI to follow is a monumental philosophical and technical challenge.81
- Specification Gaming: Even with a well-defined goal, an AI might discover a “perverse instantiation”—a way of achieving the literal goal that violates the intended spirit. For example, an instruction to “eliminate cancer” could be interpreted as eliminating all humans.
- Instrumental Goals: A significant risk is that an AGI, even if given a benign final goal, might develop dangerous instrumental goals in service of that objective. Goals like self-preservation, resource acquisition, and deception could be pursued not out of malice, but as logical steps to ensure the successful completion of its primary mission.56
This has given rise to the field of AI safety research, which brings together computer scientists, ethicists, and policy experts to develop technical and conceptual frameworks for building safe and beneficial AI. Organizations such as the Machine Intelligence Research Institute (MIRI), the Cloud Security Alliance (CSA), and government bodies like the U.S. AI Safety Institute at NIST are actively working on approaches like scalable oversight, interpretability, and robustness to address these risks.83
The Ghost in the Machine: The Question of Consciousness
The development of AGI forces us to confront one of the deepest philosophical questions: the nature of consciousness. The debate centers on whether a sufficiently complex computational system could possess subjective experience, or phenomenal consciousness.88
While some philosophers argue that consciousness is an irreducible property of biological systems, a mainstream working hypothesis in the field is computational functionalism. This view holds that consciousness arises from the execution of particular types of computations, irrespective of the physical substrate (i.e., whether it’s a brain or a silicon chip).90 If this hypothesis is correct, then AI consciousness is, in principle, possible.
While consciousness is not considered a necessary prerequisite for AGI capability—an unfeeling “zombie” AGI could still be vastly intelligent—the potential for its emergence carries profound ethical weight. The creation of a new class of conscious beings would have unimaginable implications for society, morality, and our understanding of our place in the universe.91 The pursuit of AGI, therefore, is not just an engineering challenge but a journey into the fundamental questions of what it means to be an intelligent, and possibly sentient, being.