Senior Prompt Engineer

Prompt engineering patterns, LLM evaluation frameworks, and agentic system design.

Table of Contents

• Quick Start
• Tools Overview

- Prompt Optimizer

- RAG Evaluator

- Agent Orchestrator

• Prompt Engineering Workflows

- Prompt Optimization Workflow

- Few-Shot Example Design

- Structured Output Design

• Reference Documentation
• Common Patterns Quick Reference

---

Quick Start

# Analyze and optimize a prompt file
python scripts/prompt_optimizer.py prompts/my_prompt.txt --analyze

Evaluate RAG retrieval quality
python scripts/rag_evaluator.py --contexts contexts.json --questions questions.json

Visualize agent workflow from definition
python scripts/agent_orchestrator.py agent_config.yaml --visualize

---

Tools Overview

1. Prompt Optimizer

Analyzes prompts for token efficiency, clarity, and structure. Generates optimized versions.

Input: Prompt text file or string

Output: Analysis report with optimization suggestions

Usage:

# Analyze a prompt file
python scripts/prompt_optimizer.py prompt.txt --analyze

Output:
Token count: 847
Estimated cost: $0.0025 (GPT-4)
Clarity score: 72/100
Issues found:
  - Ambiguous instruction at line 3
  - Missing output format specification
  - Redundant context (lines 12-15 repeat lines 5-8)
Suggestions:
  1. Add explicit output format: "Respond in JSON with keys: ..."
  2. Remove redundant context to save 89 tokens
  3. Clarify "analyze" -> "list the top 3 issues with severity ratings"

Generate optimized version
python scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt

Count tokens for cost estimation
python scripts/prompt_optimizer.py prompt.txt --tokens --model gpt-4

Extract and manage few-shot examples
python scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json

---

2. RAG Evaluator

Evaluates Retrieval-Augmented Generation quality by measuring context relevance and answer faithfulness.

Input: Retrieved contexts (JSON) and questions/answers

Output: Evaluation metrics and quality report

Usage:

# Evaluate retrieval quality
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json

Output:
=== RAG Evaluation Report ===
Questions evaluated: 50
#
Retrieval Metrics:
  Context Relevance: 0.78 (target: >0.80)
  Retrieval Precision@5: 0.72
  Coverage: 0.85
#
Generation Metrics:
  Answer Faithfulness: 0.91
  Groundedness: 0.88
#
Issues Found:
  - 8 questions had no relevant context in top-5
  - 3 answers contained information not in context
#
Recommendations:
  1. Improve chunking strategy for technical documents
  2. Add metadata filtering for date-sensitive queries

Evaluate with custom metrics
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \
    --metrics relevance,faithfulness,coverage

Export detailed results
python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \
    --output report.json --verbose

---

3. Agent Orchestrator

Parses agent definitions and visualizes execution flows. Validates tool configurations.

Input: Agent configuration (YAML/JSON)

Output: Workflow visualization, validation report

Usage:

# Validate agent configuration
python scripts/agent_orchestrator.py agent.yaml --validate

Output:
=== Agent Validation Report ===
Agent: research_assistant
Pattern: ReAct
#
Tools (4 registered):
  [OK] web_search - API key configured
  [OK] calculator - No config needed
  [WARN] file_reader - Missing allowed_paths
  [OK] summarizer - Prompt template valid
#
Flow Analysis:
  Max depth: 5 iterations
  Estimated tokens/run: 2,400-4,800
  Potential infinite loop: No
#
Recommendations:
  1. Add allowed_paths to file_reader for security
  2. Consider adding early exit condition for simple queries

Visualize agent workflow (ASCII)
python scripts/agent_orchestrator.py agent.yaml --visualize

Output:
┌─────────────────────────────────────────┐
│            research_assistant           │
│              (ReAct Pattern)            │
└─────────────────┬───────────────────────┘
                  │
         ┌────────▼────────┐
         │   User Query    │
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │     Think       │◄──────┐
         └────────┬────────┘       │
                  │                │
         ┌────────▼────────┐       │
         │   Select Tool   │       │
         └────────┬────────┘       │
                  │                │
    ┌─────────────┼─────────────┐  │
    ▼             ▼             ▼  │
[web_search] [calculator] [file_reader]
    │             │             │  │
    └─────────────┼─────────────┘  │
                  │                │
         ┌────────▼────────┐       │
         │    Observe      │───────┘
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │  Final Answer   │
         └─────────────────┘

Export workflow as Mermaid diagram
python scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid

---

Prompt Engineering Workflows

Prompt Optimization Workflow

Use when improving an existing prompt's performance or reducing token costs.

Step 1: Baseline current prompt

python scripts/prompt_optimizer.py current_prompt.txt --analyze --output baseline.json

Step 2: Identify issues

Review the analysis report for:

• Token waste (redundant instructions, verbose examples)
• Ambiguous instructions (unclear output format, vague verbs)
• Missing constraints (no length limits, no format specification)

Step 3: Apply optimization patterns

| Issue | Pattern to Apply |

|-------|------------------|

| Ambiguous output | Add explicit format specification |

| Too verbose | Extract to few-shot examples |

| Inconsistent results | Add role/persona framing |

| Missing edge cases | Add constraint boundaries |

Step 4: Generate optimized version

python scripts/prompt_optimizer.py current_prompt.txt --optimize --output optimized.txt

Step 5: Compare results

python scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json
Shows: token reduction, clarity improvement, issues resolved

Step 6: Validate with test cases

Run both prompts against your evaluation set and compare outputs.

---

Few-Shot Example Design Workflow

Use when creating examples for in-context learning.

Step 1: Define the task clearly

Task: Extract product entities from customer reviews
Input: Review text
Output: JSON with {product_name, sentiment, features_mentioned}

Step 2: Select diverse examples (3-5 recommended)

| Example Type | Purpose |

|--------------|---------|

| Simple case | Shows basic pattern |

| Edge case | Handles ambiguity |

| Complex case | Multiple entities |

| Negative case | What NOT to extract |

Step 3: Format consistently

Example 1:
Input: "Love my new iPhone 15, the camera is amazing!"
Output: {"product_name": "iPhone 15", "sentiment": "positive", "features_mentioned": ["camera"]}

Example 2:
Input: "The laptop was okay but battery life is terrible."
Output: {"product_name": "laptop", "sentiment": "mixed", "features_mentioned": ["battery life"]}

Step 4: Validate example quality

python scripts/prompt_optimizer.py prompt_with_examples.txt --validate-examples
Checks: consistency, coverage, format alignment

Step 5: Test with held-out cases

Ensure model generalizes beyond your examples.

---

Structured Output Design Workflow

Use when you need reliable JSON/XML/structured responses.

Step 1: Define schema

{
  "type": "object",
  "properties": {
    "summary": {"type": "string", "maxLength": 200},
    "sentiment": {"enum": ["positive", "negative", "neutral"]},
    "confidence": {"type": "number", "minimum": 0, "maximum": 1}
  },
  "required": ["summary", "sentiment"]
}

Step 2: Include schema in prompt

Respond with JSON matching this schema:
summary (string, max 200 chars): Brief summary of the content
sentiment (enum): One of "positive", "negative", "neutral"
confidence (number 0-1): Your confidence in the sentiment

Step 3: Add format enforcement

IMPORTANT: Respond ONLY with valid JSON. No markdown, no explanation.
Start your response with { and end with }

Step 4: Validate outputs

python scripts/prompt_optimizer.py structured_prompt.txt --validate-schema schema.json

---

Reference Documentation

| File | Contains | Load when user asks about |

|------|----------|---------------------------|

| references/prompt_engineering_patterns.md | 10 prompt patterns with input/output examples | "which pattern?", "few-shot", "chain-of-thought", "role prompting" |

| references/llm_evaluation_frameworks.md | Evaluation metrics, scoring methods, A/B testing | "how to evaluate?", "measure quality", "compare prompts" |

| references/agentic_system_design.md | Agent architectures (ReAct, Plan-Execute, Tool Use) | "build agent", "tool calling", "multi-agent" |

---

Common Patterns Quick Reference

| Pattern | When to Use | Example |

|---------|-------------|---------|

| Zero-shot | Simple, well-defined tasks | "Classify this email as spam or not spam" |

| Few-shot | Complex tasks, consistent format needed | Provide 3-5 examples before the task |

| Chain-of-Thought | Reasoning, math, multi-step logic | "Think step by step..." |

| Role Prompting | Expertise needed, specific perspective | "You are an expert tax accountant..." |

| Structured Output | Need parseable JSON/XML | Include schema + format enforcement |

---

Common Commands

# Prompt Analysis
python scripts/prompt_optimizer.py prompt.txt --analyze          # Full analysis
python scripts/prompt_optimizer.py prompt.txt --tokens           # Token count only
python scripts/prompt_optimizer.py prompt.txt --optimize         # Generate optimized version

RAG Evaluation
python scripts/rag_evaluator.py --contexts ctx.json --questions q.json  # Evaluate
python scripts/rag_evaluator.py --contexts ctx.json --compare baseline  # Compare to baseline

Agent Development
python scripts/agent_orchestrator.py agent.yaml --validate       # Validate config
python scripts/agent_orchestrator.py agent.yaml --visualize      # Show workflow
python scripts/agent_orchestrator.py agent.yaml --estimate-cost  # Token estimation