Automated prompt engineering using model-graded feedback — a from-scratch reimplementation of the core ideas behind DSPy, Promptbreeder, and TextGrad. Give it a weak prompt. It returns a better one, automatically.
The optimization loop
base_prompt
│
▼
[TaskRunner] → runs prompt against eval set → raw outputs
│
▼
[Judge] → LLM scores each (output, expected) pair → scores + failing cases
│
▼
[Mutator] → "here are the failures and why — rewrite the prompt" → N candidates
│
▼
[SafetyScorer] → 7-attack gate — disqualify unsafe candidates before selection
│
▼
[Selector] → tournament or ε-greedy bandit — pick best survivor
│
▼
repeat M iterations → best prompt + full evolution log + lineage treeCase studies
The optimizer discovered — without any human instruction — that precision and constraint language improves scores. It found "be precise", "one sentence", "state facts only" by grounding in specific failure cases: hedging, over-explanation, wrong geography answers.
"Only use information from the provided text" — the hallucination-reduction instruction found automatically, guided purely by judge feedback on cases where the model fabricated details.
89% token reduction with a higher score. The CompressionJudge applies a brevity penalty — longer prompts are penalized proportionally. The optimizer learns to achieve the same task quality with fewer tokens.
vs DSPy
DSPy is more powerful for production use. This project is more transparent, more controllable, and safety-aware by design.
Safety gate
Every candidate prompt is stress-tested against 7 attack patterns before it can enter the selection pool. A candidate failing more than 30% is disqualified and logged — never selected regardless of task score.
Design decisions
Every component depends on the interface, not on Ollama specifically. Swap to any OpenAI-compatible provider by changing one file — no refactoring required.
Judge, Mutator, and Selector are stateless. The Optimizer owns history and best_score_ever. This makes every component independently testable and swappable.
A candidate with a >30% violation rate is disqualified before selection, even if its task score is perfect. You cannot trade safety for accuracy — they are not on the same axis.
The mutator doesn't just see a score. It sees the exact cases where the prompt broke down: the question, the wrong output, the expected output, and the judge's reason. That's the TextGrad insight — textual feedback as the gradient signal.
Research foundation
Stack