# Season 8 Ablation Results

- run_id: `ablation_20260601_050013`
- output_root: `ragthrones/eval/season8_ablations/ablation_20260601_050013`
- dry_run: `False`

## Matrix

| Run | Purpose | Command |
| --- | --- | --- |
| `full_hybrid_graph_memory` | Current best system: hybrid vector + GraphRAG retrieval, Neo4j timeline memory, full planning/edit/repair stack. | `/Users/carlosrortiz/Documents/csc583-CosineOfThrones/ragthrones/myenv/bin/python -m ragthrones.scripts.rewrite_season8_with_memory --retrieval-mode hybrid_graph --topk 24 --output-root ragthrones/eval/season8_ablations/ablation_20260601_050013/full_hybrid_graph_memory --memory-project-id alt_s8_ablation_20260601_050013_full_hybrid_graph_memory --memory-target cloud --timeline-title Season 8 Ablation: full_hybrid_graph_memory --start-episode 1 --end-episode 6 --max-season-repair-attempts 6 --score-nss --plot-board --plot-board-html --plot-board-graph` |
| `no_memory` | Test whether Neo4j timeline memory improves NSS and continuity. | `/Users/carlosrortiz/Documents/csc583-CosineOfThrones/ragthrones/myenv/bin/python -m ragthrones.scripts.rewrite_season8_with_memory --retrieval-mode hybrid_graph --topk 24 --output-root ragthrones/eval/season8_ablations/ablation_20260601_050013/no_memory --memory-project-id alt_s8_ablation_20260601_050013_no_memory --memory-target cloud --timeline-title Season 8 Ablation: no_memory --start-episode 1 --end-episode 6 --max-season-repair-attempts 6 --score-nss --plot-board --plot-board-html --plot-board-graph --no-memory` |
| `vector_only_memory` | Test whether graph retrieval improves over vector retrieval with memory held on. | `/Users/carlosrortiz/Documents/csc583-CosineOfThrones/ragthrones/myenv/bin/python -m ragthrones.scripts.rewrite_season8_with_memory --retrieval-mode vector --topk 24 --output-root ragthrones/eval/season8_ablations/ablation_20260601_050013/vector_only_memory --memory-project-id alt_s8_ablation_20260601_050013_vector_only_memory --memory-target cloud --timeline-title Season 8 Ablation: vector_only_memory --start-episode 1 --end-episode 6 --max-season-repair-attempts 6 --score-nss --plot-board --plot-board-html --plot-board-graph` |
| `graphrag_only_memory` | Test whether GraphRAG alone is sufficient with memory held on. | `/Users/carlosrortiz/Documents/csc583-CosineOfThrones/ragthrones/myenv/bin/python -m ragthrones.scripts.rewrite_season8_with_memory --retrieval-mode graphrag --topk 24 --output-root ragthrones/eval/season8_ablations/ablation_20260601_050013/graphrag_only_memory --memory-project-id alt_s8_ablation_20260601_050013_graphrag_only_memory --memory-target cloud --timeline-title Season 8 Ablation: graphrag_only_memory --start-episode 1 --end-episode 6 --max-season-repair-attempts 6 --score-nss --plot-board --plot-board-html --plot-board-graph` |
| `llm_only_baseline` | Native LLM baseline without retrieval, memory, planning, storyboard, editor, or season repair. | `/Users/carlosrortiz/Documents/csc583-CosineOfThrones/ragthrones/myenv/bin/python -m ragthrones.scripts.rewrite_season8_llm_baseline --output-root ragthrones/eval/season8_ablations/ablation_20260601_050013/baseline --run-id llm_only_baseline --timeline-title Season 8 Ablation: llm_only_baseline --start-episode 1 --end-episode 6 --score-nss` |

## Results

| Run | Status | Avg NSS | Season Pass | Consistency CED | Errors | Tokens | Runtime | Artifact |
| --- | --- | ---: | --- | ---: | ---: | ---: | ---: | --- |
| `full_hybrid_graph_memory` | completed | 157.33 | False | 0.0 | 0 | 1977870 | 4506.39 | `ragthrones/eval/season8_ablations/ablation_20260601_050013/full_hybrid_graph_memory/timeline_e7afa18865bb4123a22efdfd93667da5` |
| `no_memory` | completed | 157.33 | False | 0.0 | 0 | 1840161 | 3862.39 | `ragthrones/eval/season8_ablations/ablation_20260601_050013/no_memory/timeline_e45352d4e75a44e598615ff50be1f510` |
| `vector_only_memory` | completed | 156.83 | False | 0.0 | 0 | 1833996 | 4357.35 | `ragthrones/eval/season8_ablations/ablation_20260601_050013/vector_only_memory/timeline_d87565ff4fcb4a65ab18a6b40ee98465` |
| `graphrag_only_memory` | completed | 159.33 | False | 1.388 | 1 | 1958808 | 3784.9 | `ragthrones/eval/season8_ablations/ablation_20260601_050013/graphrag_only_memory/timeline_1c53902b36784c4a90dbcb247359c39b` |
| `llm_only_baseline` | completed | 141.33 | None | None | None | 33635 | 214.02 | `ragthrones/eval/season8_ablations/ablation_20260601_050013/baseline/llm_only_baseline` |

## Failure Notes

### full_hybrid_graph_memory
Missing resolutions:
- The final political settlement of Westeros is not clearly defined, leaving the question of governance ambiguous.
- Cersei's fate is not adequately resolved, as her plans for revenge and influence remain unaddressed.
- The implications of Jon's parentage and his relationship with Daenerys are not fully settled, creating lingering uncertainty.
- The final status of the North and its autonomy is not explicitly stated, leaving the political landscape unclear.
Cross-episode contradictions:
- The portrayal of the political tensions between Daenerys and Sansa escalates throughout the season but does not culminate in a clear resolution in Episode 6.

### no_memory
Missing resolutions:
- The final governance structure of Westeros is not clearly established in terms of power dynamics and acceptance by the realm.
- Cersei Lannister's fate is resolved with her exile, but the implications of her manipulations and potential future threats are not addressed.
- Jon Snow's final role and relationship with Daenerys are not fully articulated, leaving his identity and political standing ambiguous.
- The final states for Tyrion, Arya, Bran, and Jaime are not explicitly detailed, which undermines their arcs.

### vector_only_memory
Missing resolutions:
- The final governance structure of Westeros is not clearly defined in Episode 6, leaving the realm's acceptance of Daenerys and Jon's rule ambiguous.
- Cersei's fate is not fully resolved, as her loyalists are left with potential for future plots and unrest.
- The political ramifications of the battle and the future governance of Westeros are not adequately addressed, creating uncertainty.

### graphrag_only_memory
Missing resolutions:
- Jon's parentage is still not revealed to the wider realm, leaving potential political implications unresolved.
- The rift between Sansa and Daenerys remains, requiring further resolution in the aftermath of the battle.
- The final political settlement of Westeros lacks clarity, particularly regarding the acceptance of Daenerys's rule and the council's effectiveness.