Description

This curriculum spans the breadth of a multi-workshop risk governance program, equipping teams to implement controls comparable to those in formal advisory engagements for regulated AI systems.

Module 1: Defining Risk Taxonomies for ML Systems in Enterprise Contexts

Selecting risk categories (e.g., data drift, model bias, operational failure) based on business impact severity and regulatory exposure
Mapping model use cases to risk tiers (e.g., high-risk for credit scoring vs. low-risk for product recommendations)
Establishing thresholds for model risk classification that align with internal audit standards and external compliance frameworks (e.g., EU AI Act)
Integrating existing enterprise risk frameworks (e.g., COSO, ISO 31000) with ML-specific risk dimensions
Deciding whether to treat third-party models as black boxes or require full transparency from vendors
Documenting risk ownership across model lifecycle stages (development, deployment, monitoring)
Creating a centralized risk register that links models to responsible teams, risk scores, and mitigation plans
Designing escalation protocols for risk events that trigger cross-functional review (legal, compliance, engineering)

Module 2: Data Governance and Provenance for ML Pipelines

Implementing data lineage tracking from raw sources through preprocessing steps to model inputs
Enforcing schema validation and data type consistency at ingestion points to prevent silent data corruption
Assessing the risk of using proxy variables that may indirectly encode protected attributes (e.g., ZIP code as proxy for race)
Deciding on data retention policies that balance model retraining needs with privacy regulations (e.g., GDPR right to erasure)
Establishing access controls for training data based on sensitivity level and role-based permissions
Validating data representativeness across time and subpopulations to detect selection bias
Implementing data versioning to support reproducible model training and auditability
Monitoring for data leakage between training and validation sets during pipeline construction

Module 4: Model Development and Validation Standards

Requiring out-of-time validation for models used in temporal decision-making (e.g., fraud detection)
Setting minimum performance thresholds (e.g., AUC, precision-recall) that reflect business cost structures
Conducting stress testing under edge-case scenarios (e.g., market shocks, supply chain disruptions)
Implementing model cards to document performance across segments, limitations, and intended use
Requiring adversarial testing for models exposed to strategic actors (e.g., spam filters, credit applications)
Enforcing code reviews and version control for model training scripts and hyperparameter configurations
Validating that feature engineering logic does not introduce unintended bias or regulatory exposure
Establishing a model validation team with independence from development to reduce confirmation bias

Module 5: Model Deployment and Operational Risk Controls

Choosing between shadow mode and canary deployment based on risk profile and rollback complexity
Implementing circuit breakers that halt model inference upon detection of input anomalies or performance degradation
Configuring model serving infrastructure with redundancy and failover to prevent single points of failure
Enforcing API contract validation to prevent downstream systems from sending malformed or out-of-range inputs
Setting up real-time logging of prediction requests, responses, and metadata for audit and debugging
Integrating deployment pipelines with change management systems to track approvals and rollback history
Requiring signed deployment manifests to ensure only authorized model versions are promoted to production
Monitoring inference latency and throughput to detect performance degradation affecting business SLAs

Module 6: Continuous Monitoring and Model Decay Management

Defining statistical thresholds for data drift (e.g., PSI > 0.25) that trigger model review
Tracking target drift by comparing predicted vs. actual outcomes over time in production
Implementing automated alerts for sudden drops in model confidence or prediction volume
Scheduling periodic retraining cadence based on data volatility and business cycle frequency
Monitoring for concept drift using performance decay on recent samples compared to validation baseline
Logging prediction explanations to detect shifts in feature importance over time
Establishing a model refresh protocol that includes revalidation before redeployment
Correlating model performance shifts with external events (e.g., policy changes, economic indicators)

Module 7: Bias, Fairness, and Ethical Risk Mitigation

Selecting fairness metrics (e.g., equalized odds, demographic parity) based on regulatory and business context
Conducting disparity impact assessments across protected and vulnerable groups pre- and post-deployment
Implementing bias mitigation techniques (e.g., reweighting, adversarial debiasing) only when justified by risk exposure
Documenting model decisions that affect individuals (e.g., loan denials) to support explainability and appeal processes
Establishing review boards for high-risk models that make consequential decisions about people
Testing for proxy discrimination by analyzing feature importance on indirect sensitive variables
Requiring fairness testing across multiple geographies and cultural contexts in global deployments
Designing feedback loops to capture downstream outcomes and correct for unintended consequences

Module 8: Regulatory Compliance and Audit Readiness

Mapping model documentation to specific requirements in regulations (e.g., SR 11-7, GDPR, MiFID II)
Preparing model risk assessment packages for internal audit and external regulators
Implementing data subject access request (DSAR) workflows that include model inference history
Ensuring automated decision-making systems provide meaningful explanations under GDPR Article 22
Archiving model artifacts (code, data, logs) for minimum retention periods required by law
Conducting periodic compliance reviews for models operating in regulated domains (e.g., healthcare, finance)
Standardizing model documentation formats to support efficient audit sampling and review
Coordinating with legal teams to interpret emerging AI regulations and adapt governance controls

Module 9: Incident Response and Model Recall Procedures

Defining criteria for model rollback (e.g., sustained accuracy drop, bias incident, data breach)
Establishing a model incident command structure with defined roles for engineering, compliance, and communications
Implementing versioned model storage to enable rapid restoration of prior stable versions
Conducting post-mortems for model failures to identify root causes and prevent recurrence
Notifying affected stakeholders (e.g., customers, regulators) based on incident severity and contractual obligations
Updating risk registers and control frameworks based on lessons from past incidents
Testing rollback procedures in staging environments to ensure operational readiness
Logging all incident response actions for regulatory and audit purposes

Module 10: Governance Framework Integration and Scaling

Embedding ML risk controls into existing enterprise risk management (ERM) reporting cycles
Aligning model review frequency with business unit risk appetite and audit schedules
Integrating model inventory systems with IT asset management and data governance platforms
Training business unit leaders to assess model risk in capital allocation and strategic planning
Standardizing model risk scoring methodologies across departments to enable aggregation
Implementing automated policy enforcement through infrastructure-as-code and CI/CD gates
Scaling governance processes to support hundreds of models without creating bottlenecks
Establishing a Center of Excellence to maintain governance standards and support local teams