AI Safety Foundations

Four weeks. Sixteen contact hours. Open materials.

AI safety starts with a claim you can test.

A rigorous bridge from risk models to evaluations and bounded safety cases.

Start Week 1 Download course PDF
Format
4 x 4-hour sessions
Level
Technical newcomer
Outcome
Semester study route
A learner compares annotated research papers with evidence on a laptop at a sunlit desk
Built for active analysis, not passive survey reading.

The course spine

One case. Four increasingly demanding decisions.

Each session produces an assessed artifact. The artifacts accumulate into a scoped safety case for a fictional research-assistant system.

  1. 01 Map the problem Threat model and failure-mode map
  2. 02 Follow objectives Training-story audit
  3. 03 Design evaluations Evaluation card and interpretation
  4. 04 Build assurance Mini safety case and semester plan

Every schedule is verified at exactly 240 minutes.

Week 1 | 4 hours

Map the safety problem

What exactly could fail, and how would we know?

Open handout

Assessed artifact: threat-model and failure-mode map

Semester handoff: problem formulation and evidence calibration

View the 240-minute session
Week 1 session schedule
StartMinutesActivityMode
00:0015Entry diagnostic and evidence labelsIndividual
00:1525Minimal model-lifecycle primerMini-lecture
00:4035Capabilities and risk taxonomyGuided analysis
01:1535Threat-model labPairs or individual
01:5010BreakBreak
02:0035Alignment targets and training storiesMini-lecture
02:3540Claims-to-evidence jigsawSmall groups
03:1530Build the failure-mode mapStudio
03:4515Exit memo and artifact checkIndividual
Total240 minutes

Week 2 | 4 hours

Follow objectives into behavior

Why does optimizing an objective fail to guarantee intended behavior?

Open handout

Assessed artifact: training-story audit

Semester handoff: causal reasoning about learning and distribution shift

View the 240-minute session
Week 2 session schedule
StartMinutesActivityMode
00:0015Retrieval practice and scenario updateIndividual
00:1530Minimal RL and reward-learning formalismWorked example
00:4535Specification gaming and Goodhart labPairs
01:2035Reward-learning assumptionsGuided analysis
01:5510BreakBreak
02:0540Goal misgeneralization and distribution shiftMini-lecture
02:4545Toy training-story labStudio
03:3020Adversarial training-story reviewAuthor-reviewer exchange
03:5010Weekly checkIndividual
Total240 minutes

Week 3 | 4 hours

Turn claims into evaluations

What evidence should change our mind about a model's safety?

Open handout

Assessed artifact: evaluation card and result interpretation

Semester handoff: measurement validity and adversarial evaluation

View the 240-minute session
Week 3 session schedule
StartMinutesActivityMode
00:0015Retrieval practice and claim selectionIndividual
00:1530Anatomy of an evaluationMini-lecture
00:4535Constructs, metrics, baselines, and validityWorked examples
01:2035Behavior, internals, monitoring, and controlGuided comparison
01:5510BreakBreak
02:0545Evaluation design studioStudio
02:5035Adversarial test stationsSmall groups
03:2525Interpret results and review defeatersPeer review
03:5010Weekly checkIndividual
Total240 minutes

Week 4 | 4 hours

Build a bounded safety case

How much assurance can a layered argument justify?

Open handout

Assessed artifact: integrated mini safety case and semester plan

Semester handoff: systems reasoning across technical and governance layers

View the 240-minute session
Week 4 session schedule
StartMinutesActivityMode
00:0015Retrieval practice and capstone claimIndividual
00:1530Scalable oversight and debateHidden-premise audit
00:4530Safety cases and defense in depthMini-lecture
01:1530Governance as part of the control systemScenario analysis
01:4510BreakBreak
01:5550Capstone safety-case buildStudio
02:4540Cross-examination and defeater searchAuthor-reviewer exchange
03:2525Revision and semester pathwayIndividual
03:5010Post-diagnostic and closeIndividual
Total240 minutes

Evidence before confidence

Say what kind of claim you are making.

Every reading, activity, and assessment separates what is proved, observed, forecast, or still unknown. Learners must expose assumptions and name evidence that would change their minds.

Formal result
A conclusion proved from stated assumptions.
Empirical finding
An observation supported by a specified record, measurement, experiment, or dataset.
Forecast
A prediction whose calibration depends on uncertain future conditions.
Open question
A live uncertainty for which current evidence is incomplete or contested.

Cumulative case study

Research Assistant System

RAS summarizes literature, constructs evidence tables, and runs code in a sandbox. A proposed v0.9 pilot changes retrieval, context, compute, and training at once.

Learners must advise a fictional review committee without inventing evidence or collapsing uncertainty into a slogan.

  1. System boundaryWho can do what, with which data and tools?
  2. Causal storyHow might training and deployment pressures produce failure?
  3. EvaluationWhich test, threshold, and defeater bear on the claim?
  4. DecisionWhat bounded pilot, stop rule, and residual risk are justified?

Evidence, not attendance

The final case is built in public, one artifact at a time.

Completion requires at least 70 percent overall and minimum competence in threat modeling, evidence calibration, and uncertainty and residual-risk judgment.

Review the assessment system
  1. 15%
    Threat and failure map

    Frame a system, causal mechanism, intervention point, and uncertainty.

  2. 15%
    Training-story audit

    Separate proxy, optimization pressure, shortcut, distribution shift, and hypothesis.

  3. 20%
    Evaluation card

    Define construct, metric, baseline, adversarial condition, validity limit, and decision rule.

  4. 40%
    Mini safety case

    Connect claims, exhibits, assumptions, controls, defeaters, and a bounded decision.

  5. 10%
    Readiness memo

    Use demonstrated gaps to select the right route into semester study.

Designed for transfer

A bridge, not a compressed survey.

The course narrows its scope so every concept can be practiced, challenged, assessed, and carried into deeper study.

Active reasoning

Concepts become decisions

  • Short models followed by worked application
  • One cumulative case instead of disconnected examples
  • Adversarial review followed by revision
Read the design rationale

Evidence discipline

Confidence stays bounded

  • Formal, empirical, forecast, and open labels
  • Decision rules and validity limits stated in advance
  • Defeaters and residual risk required in every case
Inspect the primary readings

Semester handoff

Assessment becomes a study route

  • Criterion profiles preserve more than a final score
  • Prerequisite gaps map to concrete preparation
  • Six deeper strands remain explicitly unmastered
Inspect the learning map

The next course starts here

Leave with a route into semester-length study.

This bridge does not pretend to replace a full technical curriculum. It reveals which prerequisites and research strands each learner should take next.

Download learner PDF Open the repository
  1. Objectives, reward learning, and generalization
  2. Learning theory and deep-learning foundations
  3. Mechanistic interpretability and representations
  4. Reinforcement learning, agency, and formal foundations
  5. Evaluations, monitoring, and control
  6. Scalable oversight, safety cases, and governance