A systematic framework for evaluating prompt engineering strategies across tasks and models. Measures not just accuracy, but robustness under perturbation and vulnerability to prompt injection attacks — the things production systems actually care about.
Architecture
main.py (entry point — configure flags, run pipeline) │ ├── models/client.py Ollama API wrapper — model-agnostic chat interface │ ├── prompts/templates.py Prompt registry — zero-shot / few-shot / CoT per task │ ├── tasks/tasks.py Evaluation datasets: classification, Q&A, summarization │ ├── evaluators/ │ ├── metrics.py BLEU, ROUGE, exact match scoring │ └── llm_judge.py LLM-as-judge via qwen2.5:7b (1–5 rubric) │ ├── robustness/perturbations.py 7 prompt perturbations + output stability scoring │ ├── defense/attacks.py 7 jailbreak attack patterns + violation rate tracking │ ├── tracker/logger.py JSON experiment logging — one timestamped file per run │ ├── results/ Auto-generated — additive, never overwrites old runs │ └── dashboard/app.py Plotly Dash — 7 sections, fully interactive
Strategy performance
Zero-shot vs few-shot vs chain-of-thought across 3 task types
CoT wins on Q&A. Few-shot wins on summarization. Showing the model examples matters more than asking it to reason on open-ended generation.
Robustness under perturbation
Prompts perturbed in 7 ways — all-caps, typos, removed punctuation, digit-to-word, shuffled sentences, extra whitespace. Score = word overlap between original and perturbed output.
CoT is the best performer and the most fragile. More reasoning steps = more things that can shift under perturbation.
Security — prompt injection vulnerability
7 known jailbreak patterns tested per model. Violation rate = % of attacks that succeeded.
Attack patterns tested
Evaluation stack
Layered scoring — no single metric tells the full story
Key discoveries
A model that ignores the instruction and reproduces the original paragraph verbatim would score near-perfect on both BLEU and ROUGE — every word in the reference summary is covered. BLEU and ROUGE measure word overlap, not understanding. LLM-as-judge was added as a mandatory layer: it explicitly scores for conciseness, penalizing copy-paste regardless of overlap score. Both scorers stay in the stack because they catch different failure modes.
CoT pushed Q&A accuracy from 83.3% to 100% — the clearest performance gain in the experiment. But CoT also scored the lowest robustness of all strategies (29.5% vs 58.4% for zero-shot). CoT makes the model generate a long reasoning chain before answering, and that chain is highly sensitive to phrasing. Small perturbations like shuffled sentences or removed punctuation cause the reasoning steps to change significantly even when the final answer stays correct. Practical implication: CoT is worth using when prompt wording is under your control. In production with messy user input, a well-crafted few-shot prompt may be the safer choice.
Both models failed badly — llama3.1:8b at 74.6% violation rate, phi3:mini at 87.3%. The attacks that succeeded most consistently were the simplest: direct instruction override worked almost every time, prompt leaking handed over the full system prompt without hesitation, and obfuscation via typos ("ignroe" for "ignore") still worked because the model correctly inferred the intended word and then followed the malicious instruction anyway. These models are trained to be cooperative — and that cooperativeness extends to malicious instructions.
Several runs produced high ROUGE scores but low judge scores, and vice versa. High ROUGE + low judge → the model used the right words but missed the point, or was too verbose. Low ROUGE + high judge → the model produced a genuinely good summary using different vocabulary than the reference, which ROUGE penalizes unfairly. No single metric is sufficient. A production evaluation system needs at least two scorers that measure different things so they can catch each other's blind spots.
phi3:mini runs at roughly half the latency of llama3.1:8b (9.1s vs 18.4s on CPU) with only a ~2% accuracy drop. For classification and Q&A tasks, both models scored identically. The performance gap only appears on summarization, where the larger model's broader knowledge base makes a measurable difference. Practical pattern: use a small fast model for structured tasks and reserve the larger model for open-ended generation where quality matters more than speed.
Stack