Description

This curriculum engages learners in a multi-workshop program–level examination of AI existential risk, comparable to the structured deliberations of an internal capability program focused on long-term governance, ethical specification, and cross-jurisdictional coordination in high-consequence AI development.

Module 1: Defining Existential Risk and Superintelligence in Organizational Contexts

Establishing a working definition of existential risk that aligns with enterprise risk management frameworks such as ISO 31000.
Distinguishing between narrow AI, artificial general intelligence (AGI), and superintelligence in strategic planning documents.
Mapping AI capability thresholds to potential organizational disruption scenarios in finance, defense, and healthcare sectors.
Deciding whether to classify superintelligence as a strategic risk or a speculative concern in board-level risk registers.
Integrating long-term AI risk modeling into enterprise horizon scanning and futures analysis processes.
Assessing the credibility of AI timelines provided by research labs when allocating R&D budgets.
Designing cross-functional teams to evaluate AI risk scenarios without over-relying on technical specialists.
Creating escalation protocols for AI developments that may shift risk categorization from theoretical to imminent.

Module 2: Ethical Frameworks for High-Stakes AI Decision-Making

Selecting between deontological, consequentialist, and virtue ethics models when designing AI oversight policies.
Implementing ethical review boards with authority to halt AI development projects based on moral risk assessments.
Resolving conflicts between corporate fiduciary duties and broader societal ethical obligations in AI deployment.
Translating abstract ethical principles like "beneficence" into auditable design constraints for machine learning systems.
Managing jurisdictional differences in AI ethics regulations when operating across EU, US, and Asian markets.
Documenting ethical trade-offs in AI decision logs for future legal and regulatory scrutiny.
Balancing transparency with competitive advantage when disclosing ethical risk mitigation strategies.
Training executives to recognize ethical drift in AI projects that incrementally compromise foundational principles.

Module 3: Governance Structures for Autonomous Systems

Designing human-in-the-loop, human-on-the-loop, and fully autonomous decision pathways based on risk severity.
Assigning legal accountability for AI-driven actions when no single individual can trace cause-effect chains.
Implementing circuit breakers and kill switches in autonomous systems with defined activation thresholds.
Structuring board-level AI oversight committees with technical, legal, and ethical expertise.
Determining whether AI governance should reside under compliance, risk, strategy, or a standalone function.
Creating audit trails for autonomous decisions that satisfy regulatory requirements without enabling reverse engineering.
Establishing escalation ladders for AI behaviors that fall outside predefined operational envelopes.
Defining conditions under which autonomous systems may modify their own governance parameters.

Module 4: Risk Assessment Methodologies for Superintelligence Scenarios

Adapting failure mode and effects analysis (FMEA) for AI systems with recursive self-improvement capabilities.
Quantifying uncertainty in AI risk models where historical data is absent or non-analogous.
Selecting between probabilistic risk assessment and scenario planning for low-probability, high-impact AI events.
Calibrating risk matrices to account for irreversible outcomes such as loss of human control.
Integrating expert elicitation from AI researchers into formal risk assessments despite conflicting incentives.
Stress-testing AI governance frameworks against worst-case alignment failure scenarios.
Validating risk mitigation strategies when full-scale testing would itself pose unacceptable dangers.
Updating risk profiles in response to breakthroughs in AI capabilities without triggering organizational panic.

Module 5: AI Alignment and Value Specification Challenges

Specifying human values in machine-interpretable form without oversimplifying complex moral trade-offs.
Designing feedback mechanisms that allow AI systems to refine goals without drifting from original intent.
Implementing corrigibility features that allow safe interruption without incentivizing resistance.
Choosing between single-agent alignment and multi-stakeholder value aggregation in public-facing AI.
Handling value conflicts across cultures when deploying global AI systems with normative implications.
Preventing reward hacking by designing robust objective functions resistant to specification gaming.
Testing alignment in simulated environments that adequately represent real-world complexity.
Managing the risk of value lock-in when early design decisions become entrenched.

Module 6: Regulatory and Legal Preparedness for Post-AGI Environments

Drafting contractual clauses that allocate liability for AI behaviors beyond current legal categories.
Preparing for regulatory audits of AI systems that may evolve beyond their original certified state.
Engaging with policymakers to shape legislation that balances innovation with existential risk mitigation.
Establishing legal personhood criteria for advanced AI systems in intellectual property and liability contexts.
Creating compliance architectures that adapt to rapidly changing AI regulations across jurisdictions.
Developing evidence preservation protocols for AI decision-making in anticipation of litigation.
Negotiating international treaties on AI development limits while protecting national security interests.
Designing exit strategies for AI projects that may become legally untenable due to new regulations.

Module 7: Organizational Resilience and Control Mechanisms

Implementing layered containment strategies for AI development environments to prevent unauthorized access or exfiltration.
Designing incentive structures that discourage researchers from bypassing safety protocols for performance gains.
Creating redundancy in human oversight systems to prevent single-point failures in AI monitoring.
Establishing secure communication channels for reporting AI safety concerns without career repercussions.
Conducting red team exercises to test the robustness of AI control mechanisms under adversarial conditions.
Managing supply chain risks when third-party components introduce uncontrolled AI capabilities.
Developing continuity plans for critical infrastructure that may depend on AI systems with opaque logic.
Training crisis response teams to manage AI incidents that escalate beyond technical containment.

Module 8: International Cooperation and Geopolitical Dimensions

Assessing the feasibility of AI development moratoria given asymmetric national incentives and verification challenges.
Designing information-sharing agreements on AI safety research that do not compromise strategic advantage.
Navigating dual-use dilemmas where AI safety research could also enhance offensive capabilities.
Coordinating export controls on AI hardware and software to slow uncontrolled proliferation.
Building trust between competing nations on AI risk mitigation without exposing sensitive research.
Participating in multilateral forums to establish norms for responsible AI development.
Responding to AI advancements in adversarial states that may destabilize global equilibrium.
Allocating resources to global public goods in AI safety when benefits are diffuse and delayed.

Module 9: Long-Term Stewardship and Institutional Design

Creating intergenerational governance bodies with authority to enforce AI safeguards beyond electoral cycles.
Designing institutional memory systems to preserve AI risk knowledge across leadership transitions.
Establishing funding mechanisms for AI safety research that are insulated from short-term performance pressures.
Developing succession planning for AI oversight roles that require rare technical and ethical expertise.
Balancing transparency with security in public communication about AI risks to avoid panic or complacency.
Embedding AI stewardship principles into organizational constitutions and founding documents.
Creating mechanisms for civil society input into AI governance without compromising operational security.
Planning for organizational dissolution or transformation in scenarios where AI fundamentally alters the operating environment.