# Cosine of Thrones Final Project Overview

## One-Sentence Summary
Cosine of Thrones is a multi-agent Retrieval-Augmented Generation system that uses vector search, GraphRAG, Neo4j memory, LangGraph orchestration, and narrative evaluation to generate and critique alternate versions of Game of Thrones Season 8.

## Core Claim
The project shows that long-form narrative generation is not only a text-generation problem. It is a continuity-management, evidence-retrieval, planning, revision, and evaluation problem. A single LLM can produce an alternate ending, but the agentic pipeline improves the process by making story decisions explicit, grounding them in canon evidence, carrying memory across episodes, and using editor/checker agents to repair weak drafts.

## Project Goals
- Answer Game of Thrones questions using hybrid retrieval over vector, keyword, custom KG, and GraphRAG evidence.
- Generate a six-episode alternate Season 8 that preserves Seasons 1-7 canon.
- Track alternate-timeline memory so later episodes build on earlier generated decisions.
- Evaluate generated and human-written endings with the same Narrative Structure Score (NSS) rubric.
- Measure continuity errors and explicit finale arc-closure coverage with a deterministic story consistency checker.
- Compare retrieval, memory, planning/editing, season repair, and baseline LLM conditions through reproducible full-season ablations.

## System Architecture

![Season 8 LangGraph Workflow](ragthrones/assets/season8_langgraph.svg)

```mermaid
flowchart LR
    A["Canon corpus<br/>S1-S7 chunks"] --> B["Vector retrieval<br/>FAISS + BM25"]
    A --> C["GraphRAG / Neo4j<br/>entities, relationships, reports"]
    D["Alternate memory<br/>timeline decisions + threads"] --> E["Episode context"]
    B --> E
    C --> E
    E --> F["Season Planner"]
    F --> G["Episode Planner"]
    G --> H["Storyboard Agent"]
    H --> I["Writer Agent"]
    I --> J{"Editor / validation"}
    J -- "patch" --> K["Patch Agent"]
    K --> J
    J -- "rewrite" --> I
    J -- "pass" --> L["Memory Writer"]
    L --> M{"More episodes?"}
    M -- "yes" --> E
    M -- "no" --> N["Season Checker"]
    N --> O["Plot Board + NSS + artifacts"]
```

## Main Concepts

### Retrieval-Augmented Generation
The system retrieves canon evidence before writing or scoring. Retrieval modes include:

| Mode | Meaning |
|---|---|
| `vector` | FAISS/BM25-backed local retrieval |
| `kg` | Custom Neo4j knowledge graph only |
| `hybrid` | Vector + custom KG |
| `graphrag` | GraphRAG Neo4j only |
| `hybrid_graph` | Vector + GraphRAG |
| `hybrid_all` | Vector + custom KG + GraphRAG |

### GraphRAG
GraphRAG is used to represent canon as structured entities, relationships, text units, and community reports. In this project, the graph is queried through fixed parameterized Neo4j retrieval functions, not LLM-generated Cypher. That makes retrieval safer and easier to debug.

### Alternate Timeline Memory
Memory is the difference between generating six isolated episode summaries and generating one coherent branch. After each accepted episode, the pipeline writes:

- generated episode scene/summary
- story decisions
- open threads
- resolved threads
- linked canon entities

Before the next episode, those memory nodes are retrieved and inserted into the planner, storyboard, writer, and editor prompts.

### LangGraph Orchestration
The LangGraph wrapper makes the agent handoffs visible:

```mermaid
flowchart TD
    START(["START"]) --> initialize_run["initialize_run"]
    initialize_run --> prepare_season_plan["prepare_season_plan"]
    prepare_season_plan --> retrieve_episode_context["retrieve_episode_context"]
    retrieve_episode_context --> plan_episode["plan_episode"]
    plan_episode --> storyboard_episode["storyboard_episode"]
    storyboard_episode --> write_and_validate_episode["write_and_validate_episode"]
    write_and_validate_episode --> continuity_edit_episode["continuity_edit_episode"]
    continuity_edit_episode --> score_episode["score_episode"]
    score_episode --> write_episode_memory["write_episode_memory"]
    write_episode_memory --> persist_episode_artifacts["persist_episode_artifacts"]
    persist_episode_artifacts --> episode_router{"more episodes?"}
    episode_router -- yes --> retrieve_episode_context
    episode_router -- no --> season_resolution_check["season_resolution_check"]
    season_resolution_check --> build_plot_board["build_plot_board"]
    build_plot_board --> finalize_run["finalize_run"]
    finalize_run --> END(["END"])
```

## Pipeline Agents

| Agent / Component | Role | Output |
|---|---|---|
| Season Planner | Designs the six-episode showrunner plan, war spine, political spine, and final character states | Structured season JSON + markdown |
| Episode Planner | Turns the season plan into episode-level A/B/C plots, required beats, turns, and guardrails | Episode plan JSON + markdown |
| Storyboard Agent | Converts the episode plan into visual staging, motifs, and closing-image guidance | Storyboard JSON + visual brief |
| Writer Agent | Generates the actual episode treatment | Markdown treatment |
| Hard Validator | Checks required sections, key characters, war state, consequences, and finale closure signals | Validation issues |
| Episode Editor | Reviews the draft against plan, memory, active threads, storyboard, and canon context | Pass/patch/rewrite decision |
| Patch Agent | Applies localized repairs when the episode is mostly sound | Revised treatment |
| Continuity Editor | Blocks contradictions against irreversible prior episode states | Continuity log |
| Memory Writer | Stores accepted decisions, scenes, open/resolved threads, and canon links | Neo4j memory nodes |
| Season Checker | Reviews the full season for unresolved arcs, finale closure, and cross-episode contradictions | Season-level correction brief |
| Story Consistency Checker | Produces quote-backed continuity evidence chains, Consistency Error Density (CED), taxonomy counts, and Episode 6 arc-closure coverage | Per-episode reports + aggregate consistency summary |
| Plot Board Agent | Produces a post-generation map of beats, arcs, reveals, payoffs, and warnings | `plot_board.md/json/html` |
| NSS Agent | Scores narrative structure and canon fit | Weighted rubric JSON |

### Planning and Closure Contracts

The latest planner/writer architecture adds explicit long-horizon contracts:

- The season planner first defines unchangeable Episode 6 terminal states for the core cast.
- Backward causal ladders require Episodes 1-5 to contain direct decisions and scenes that earn those end states.
- A canon-derived Season 8 start-state ledger prevents the system from assuming the Night King or other open Season 7 arcs were resolved offscreen.
- Each episode receives an arc-closure ledger derived from the season plan, episode plan, active memory threads, and prior final-state ledgers.
- Episode 6 validation requires a concrete Final State Ledger and rejects sequel-hook language for central arcs.
- The teleplay writer receives the closure ledger and an in-run scene ledger so outcomes are dramatized without repeating prior scenes.

## Writer / Editor Loop

```mermaid
flowchart LR
    A["Writer draft"] --> B["Hard validation"]
    B --> C{"Valid enough?"}
    C -- "no" --> D["Rewrite prompt<br/>with concrete issues"]
    D --> A
    C -- "yes" --> E["Episode Editor"]
    E --> F{"Pass, patch, or rewrite?"}
    F -- "patch" --> G["Patch Agent"]
    G --> E
    F -- "rewrite" --> A
    F -- "pass" --> H["Continuity Editor"]
    H --> I{"Contradicts locked states?"}
    I -- "yes" --> G
    I -- "no" --> J["Persist episode + memory"]
```

## Evaluation and Ablation Plan

The evaluation combines benchmark comparisons, reproducible pipeline ablations, and complementary evaluators:

| Condition | Purpose |
|---|---|
| Original HBO Season 8 | Real-world canon benchmark |
| Think Story YouTube alternate ending | Human fan benchmark |
| Gemini standalone alternate ending | External model baseline |
| Baseline LLM Season 8 | Single-model generation baseline |
| Multi-agent + vector RAG | Tests agentic planning/editing with local evidence |
| Multi-agent + memory + GraphRAG | Main proposed system |
| Memory + GraphRAG with different temperature | Tests creativity/consistency tradeoff |
| Memory + GraphRAG with different LLM model | Tests model sensitivity |
| Memory + GraphRAG with Gemini generation | Tests provider/model swap in the full agentic harness |

The implemented baseline ablation matrix is:

| Ablation Run | Controlled Question |
|---|---|
| `full_hybrid_graph_memory` | How does the current full system perform with hybrid vector + GraphRAG retrieval and Neo4j memory? |
| `no_memory` | Does alternate-timeline Neo4j memory improve NSS, continuity, or closure? |
| `vector_only_memory` | Does graph retrieval improve over vector-only retrieval when memory remains enabled? |
| `graphrag_only_memory` | Is GraphRAG alone sufficient when memory remains enabled? |
| `llm_only_baseline` | How much does the full pipeline improve over a native LLM carrying continuity only through prompt context? |

The ablation harness writes `ablation_plan.json`, `ablation_results.json`, `ablation_summary.md`, per-run logs, and isolated artifacts for each condition. Memory-enabled runs use isolated memory project IDs by default to prevent contamination between conditions.

## Evaluation Agents and Metrics

No single score captures long-form narrative quality, so the project now uses three complementary evaluation layers:

| Evaluator | Type | Measures |
|---|---|---|
| NSS Agent | LLM-based weighted rubric | Episode-level narrative quality, canon fit, causality, macrostructure, and closure |
| Season Resolution Checker | LLM-based developmental editor | Full-season pass/fail, missing resolutions, cross-episode contradictions, and concrete rewrite instructions |
| Story Consistency Checker | Deterministic evidence-chain checker | Continuity errors, CED, category counts, and explicit Episode 6 arc-closure coverage |

The story consistency checker uses the following taxonomy:

- `timeline_plot_logic`
- `characterization`
- `worldbuilding_setting`
- `factual_detail`
- `narrative_style`
- `arc_closure`

Each detected issue includes a fact quote, contradiction quote, locations, reasoning, severity, and conclusion. CED reports errors per 10,000 words. Episode 6 additionally receives an arc-closure coverage report over seven required areas:

1. Westeros governance and succession
2. The North's final political status
3. The Night King / White Walker war
4. Cersei and her remaining power bloc
5. Daenerys's final state
6. Jon's final role and identity settlement
7. Tyrion, Sansa, Arya, Bran, and Jaime's ensemble final states

The native LLM baseline now runs the same season-resolution checker and story consistency checker as pipeline variants. This makes `Season Pass`, CED, and closure coverage comparable across the ablation matrix.

## NSS Rubric

NSS scores each answer or episode using weighted categories:

| Category | Weight | What It Measures |
|---|---:|---|
| Setting consistency | 2 | Correct locations, timeline, and world state |
| Character consistency | 4 | Whether behavior matches established character arcs |
| Character motivation | 4 | Whether choices have plausible reasons |
| Referencing consistency | 3 | Whether retrieved evidence is used accurately |
| Conflict / resolution linkage | 4 | Whether causes lead to payoffs |
| Theme alignment | 3 | Whether the ending fits Game of Thrones themes |
| Macrostructure cohesion | 4 | Beginning, escalation, climax, resolution |
| Creative plausibility | 4 | Whether the alternate direction feels believable |
| Closure / arc resolution | 4 | Whether finale/end states are explicit |

## Current Evaluation Results

NSS has a maximum weighted score of 160. For final-presentation results, I reran the NSS agent across the three headline candidates with one shared scoring harness: prior episodes from the same candidate season plus the current episode treatment as evidence, with no extra source transcript/script evidence for any candidate.

Fresh final-presentation NSS rerun:

| Condition | Episodes | Corrected Avg NSS | Main interpretation |
|---|---:|---:|---|
| Original HBO Season 8 | 6 | 143.17 | Highest under the shared NSS rerun; the evaluator rewards the self-contained canon candidate. |
| GoT Season 8 Redux | 10 | 119.80 | Human fan benchmark; strong early episodes, but the finale is penalized hard under the shared closure rubric. |
| Latest agentic GenAI run, `timeline_ee8b0736bfce4fe7b4fea0913908025e` | 6 | 128.00 | Beats Redux on average in the fair rerun, but trails HBO and still fails season-level closure checks. |

Corrected agentic run context:

| Run | Condition | Corrected Avg NSS | Season checker |
|---|---|---:|---|
| `timeline_ee8b0736bfce4fe7b4fea0913908025e` | Latest multi-agent + memory + GraphRAG | 128.00 | Fail: unresolved White Walker leader, Daenerys/Jon/Cersei outcomes, and succession settlement. |
| `timeline_4818fe791fe244328fefe959a40f9db9` | Highest corrected agentic run found in local artifacts | 156.00 | Fail: checker returned invalid JSON; manual review required. |
| `timeline_67601daa0dc1475d952d3a3e49e2b294` | Earlier main multi-agent + memory + GraphRAG run | 128.00 | Fail: incomplete long-term closure. |

Source files:
- `final_presentation_reevaluation.md`
- `ragthrones/eval/original_s8_treatments/season8_treatments_summary.json`
- `ragthrones/eval/human_alternate_endings/got_s8_redux/treatments_fair/*.json`
- `ragthrones/eval/alternate_s8/timeline_ee8b0736bfce4fe7b4fea0913908025e/season8_summary.json`
- `ragthrones/eval/alternate_s8/timeline_ee8b0736bfce4fe7b4fea0913908025e/season8_nss.json`
- `ragthrones/eval/final_presentation_nss_reevaluation/summary.md`
- `ragthrones/eval/alternate_s8_baseline/baseline_v1/season8_nss.json`

### Implemented Full-Season Ablation Results

The first completed five-condition ablation with the story consistency checker was `ablation_checker_v1`:

| Run | Avg NSS | Season Pass | Consistency CED | Errors | Tokens |
|---|---:|---|---:|---:|---:|
| `graphrag_only_memory` | 159.33 | Fail | 0.000 | 0 | 2,089,421 |
| `no_memory` | 158.67 | Fail | 0.000 | 0 | 1,804,412 |
| `full_hybrid_graph_memory` | 155.33 | Fail | 0.000 | 0 | 1,987,599 |
| `vector_only_memory` | 150.17 | Fail | 1.455 | 1 | 2,014,526 |
| `llm_only_baseline` | 135.50 | Not measured in this earlier run | 0.000 | 0 | 33,609 |

This run suggests:

- The multi-agent pipeline substantially improves NSS over the native LLM baseline.
- GraphRAG-only retrieval produced the highest NSS in this single run.
- Neo4j timeline memory did not improve NSS in this run; `no_memory` outscored the full system while using fewer tokens.
- Vector-only retrieval produced the lowest pipeline NSS and the only detected continuity error.
- Every pipeline variant still failed the season-resolution checker because Episode 6 left major political or character outcomes ambiguous.

These are single-run observations rather than statistically conclusive effects. Multiple seeded/repeated runs are still needed to separate component effects from generation variance.

### Closure-Aware Baseline Result

The newer `ablation_closure_v1` run added comparable baseline season checking and deterministic arc-closure coverage. Its pipeline variants aborted before generation because Neo4j was unavailable, so only its baseline result is valid:

| Run | Avg NSS | Season Pass | CED | Closure Coverage | Errors | Tokens |
|---|---:|---|---:|---:|---:|---:|
| `llm_only_baseline` | 140.17 | Fail | 10.154 | 42.9% (`3/7`) | 6 | 43,598 |

The baseline wrote locally plausible episodes but failed to land the season. Episode 6 explicitly left governance and Lannister loyalists unresolved, used provisional language such as "fragile peace," and lacked concrete final states for several central arcs. This demonstrates why episode-level NSS must be reported alongside season resolution and closure coverage.

Source files:

- `ragthrones/eval/season8_ablations/ablation_checker_v1/ablation_summary.md`
- `ragthrones/eval/season8_ablations/ablation_checker_v1/ablation_results.json`
- `ragthrones/eval/season8_ablations/ablation_closure_v1/ablation_summary.md`
- `ragthrones/eval/season8_ablations/ablation_closure_v1/ablation_results.json`

## Interpretation For Paper

The main result is not that the agentic system always writes perfect fiction. The stronger claim is that it creates an inspectable and controllable process for long-horizon narrative generation. The pipeline exposes intermediate decisions, stores continuity, retrieves evidence, and logs why drafts are rejected or repaired. This matters because Season 8 rewriting requires:

- maintaining canon constraints from seven prior seasons
- resolving political and supernatural conflicts together
- giving many ensemble characters explicit final states
- avoiding contradictions across six generated episodes
- balancing creativity with evidence-supported plausibility

The latest agentic run outperforms the Redux human fan rewrite on the fresh shared NSS rerun, but it does not outperform original HBO Season 8. The stronger project claim is therefore not "AI wrote a better Season 8 than HBO." The stronger claim is that agentic RAG creates an inspectable and controllable narrative-generation process: planning, retrieval, memory, editor critique, patching, season checking, plot boards, and traceable artifacts. The final paper should present this as a quality/control versus reliability/cost tradeoff, with an honest distinction between episode-level scores and still-imperfect season-level resolution.

## Important Limitations

- NSS is an LLM-based evaluator, so it should be treated as a structured heuristic rather than objective truth.
- The season checker is also LLM-based and can vary across runs or return malformed output.
- The deterministic consistency checker is inspectable but currently relies on pattern-based coverage rules that require continued calibration.
- Finale-level comparisons are not identical to full-season comparisons.
- Some rubric categories, especially creative plausibility, behave differently for canon episodes than for alternate endings.
- The agentic pipeline can still leave closure gaps, especially around late-season political settlement.
- GraphRAG can penalize creative lore additions that are plausible but not directly supported by retrieved canon evidence.
- Runtime and token cost are significant because the system performs many planner, writer, editor, checker, retrieval, and scoring calls.
- Graph and memory ablations require a healthy Neo4j connection; infrastructure failures must be reported separately from storytelling failures.

## Suggested Final Paper Structure

1. **Introduction**
   - Problem: long-form narrative generation is hard because it requires continuity, causality, and payoff.
   - Project: Cosine of Thrones as a multi-agent RAG system for Game of Thrones reasoning and Season 8 rewriting.

2. **Background**
   - RAG, knowledge graphs, GraphRAG, agentic workflows, memory, and LLM-as-evaluator.

3. **System Design**
   - Corpus and retrieval.
   - Vector retrieval and GraphRAG.
   - Neo4j alternate memory.
   - LangGraph orchestration.
   - Agent roles and repair loop.

4. **Implementation**
   - Key modules:
     - `ragthrones/pipelines/season8_langgraph.py`
     - `ragthrones/scripts/run_season8_langgraph.py`
     - `ragthrones/scripts/rewrite_season8_with_memory.py`
     - `ragthrones/agents/*`
     - `ragthrones/memory/*`
     - `ragthrones/retrieval/*`

5. **Experimental Setup**
   - Conditions: original, human rewrite, Gemini rewrite, baseline LLM, vector RAG, memory + GraphRAG, temperature/model variants.
   - Controls: same premise, six episodes, same evaluator, same retrieval top-k where applicable.

6. **Evaluation Metrics**
   - NSS weighted score.
   - Story consistency evidence chains and Consistency Error Density.
   - Deterministic Episode 6 arc-closure coverage.
   - LLM season-resolution pass/fail and correction briefs.
   - Repair passes.
   - Token usage and runtime.
   - Qualitative examples.

7. **Results**
   - Finale-level benchmark table.
   - Full-season ablation table.
   - NSS versus closure-coverage comparison.
   - Cost/runtime table.
   - Example of editor rejection and repair.

8. **Discussion**
   - Why memory and GraphRAG help.
   - Where GraphRAG is stricter than vector retrieval.
   - Why agentic quality comes with cost.
   - Failure cases and remaining closure issues.

9. **Conclusion**
   - Main takeaway: agentic RAG is useful for narrative generation because it makes the process explicit, reviewable, and repairable.

## Suggested 10-Slide Final Presentation

| Slide | Title | Main Point | Visual |
|---|---|---|---|
| 1 | Cosine of Thrones | Multi-agent RAG for alternate Season 8 generation | Title + system one-liner |
| 2 | Why Season 8 Is Hard | Long-form generation needs memory, causality, payoff, and closure | Problem diagram |
| 3 | System Architecture | Vector retrieval + GraphRAG + memory + agents | LangGraph SVG |
| 4 | Agent Roles | Planner, storyboard, writer, editor, patcher, checker | Agent table |
| 5 | Memory | Episodes become a persistent alternate timeline | Memory loop diagram |
| 6 | Repair Loop | Drafts are rejected, patched, or rewritten | Writer/editor Mermaid loop |
| 7 | Evaluation Design | NSS + season checker + deterministic consistency/closure metrics | Experiment and metric table |
| 8 | Ablation Results | Pipeline beats baseline on NSS, but all tested variants still struggle with finale closure | NSS, CED, closure, and cost table |
| 9 | What GraphRAG Adds | More structured canon pressure and stricter evidence grounding | Think Story vector vs graph comparison |
| 10 | Takeaways | Better control and continuity, higher cost, remaining closure gaps | Final claims + future work |

## One-Minute Talk Track

"Cosine of Thrones started as a RAG system for answering Game of Thrones questions, but the final version became a multi-agent writing and evaluation pipeline for alternate Season 8 generation. Rewriting a final season is not just a prompt engineering problem. It requires planning backward from final states, retrieving canon evidence, carrying continuity memory, editing drafts, repairing failures, and measuring whether the season actually closes its arcs. The system uses vector retrieval and GraphRAG for Seasons 1-7 grounding, Neo4j memory for alternate-timeline decisions, and LangGraph for orchestration. It evaluates every run with NSS, a season-resolution checker, and a deterministic story consistency checker that produces quote-backed errors, Consistency Error Density, and Episode 6 arc-closure coverage. In the first completed ablation, every multi-agent variant beat the native LLM baseline on NSS, with GraphRAG-only scoring highest, but every pipeline variant still failed the season-resolution checker. The result is an inspectable account of both where agentic RAG helps and where long-horizon narrative closure remains difficult."

## Commands To Reproduce Key Runs

OpenAI / main agentic run:

```bash
LLM_PROVIDER=openai \
GEN_MODEL=gpt-4o-mini \
EVAL_LLM_PROVIDER=openai \
NSS_MODEL=gpt-4o-mini \
ragthrones/myenv/bin/python -m ragthrones.scripts.run_season8_langgraph \
  --retrieval-mode hybrid_graph \
  --memory-project-id alt_s8 \
  --score-nss \
  --plot-board \
  --plot-board-html \
  --plot-board-graph \
  --recursion-limit 240
```

Gemini full pipeline run:

```bash
LLM_PROVIDER=gemini \
GEN_MODEL=gemini-2.5-flash \
EVAL_LLM_PROVIDER=openai \
NSS_MODEL=gpt-4o-mini \
ragthrones/myenv/bin/python -m ragthrones.scripts.run_season8_langgraph \
  --retrieval-mode hybrid_graph \
  --memory-project-id alt_s8_gemini_full \
  --score-nss \
  --plot-board \
  --plot-board-html \
  --plot-board-graph \
  --recursion-limit 240
```

Full five-condition Season 8 ablation:

```bash
ragthrones/myenv/bin/python -m ragthrones.scripts.run_season8_ablation \
  --run-id ablation_closure_v2
```

Resume a partial ablation while reusing completed artifacts:

```bash
ragthrones/myenv/bin/python -m ragthrones.scripts.run_season8_ablation \
  --run-id ablation_closure_v1 \
  --skip-existing
```

Dry-run the ablation plan without generating episodes:

```bash
ragthrones/myenv/bin/python -m ragthrones.scripts.run_season8_ablation \
  --run-id ablation_dry_run \
  --dry-run
```

Tests:

```bash
./run_tests.sh
```

Regenerate LangGraph assets:

```bash
ragthrones/myenv/bin/python -m ragthrones.scripts.generate_season8_langgraph
```