DSPy GEPA vs MiPRO Benchmark & Optimization Platform
Full-corpus optimization results across ALL 176 taxonomy fields on 100 legal agreements comparing DSPy GEPA vs MiPRO.
10-Iteration Optimization Curves across ALL 176 Taxonomy Fields
Showing pale purple curves for all 176 GEPA runs and pale orange curves for all 176 MiPRO runs, with bold corpus averages on top.
Agreement Type
Agreement Metadata Tuned GEPA Winner (+0.18 F1)All 176 Field Outcomes & Winner Breakdown
Check any row box to highlight that field's purple GEPA and orange MiPRO trajectory on the chart above.
| Tick | Field Name | Domain | Winner | GEPA F1 | MiPRO F1 | Base (0) | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 | Iter 9 | Iter 10 | Tokens |
|---|
Actual Real Costs & Token Telemetry (176 Fields across 100 Documents)
Standardized against 0% prompt cache hit rate (Gross List Price baseline). Eliminates test rerun cache artifacts for true production modeling.
• Standard List-Price Compute (0% Cache): $194.92 (~$1.107 / field)
• Candidate Evaluation (5.6-luna): 289.5M tokens ($106.14 gross list price)
• GPT-5.6 Sol Reflection Mutations: 4.8M tokens ($88.78 @ $5 in / $30 out / 2.0x CoT)
• Steady-State Pipeline (40% Cache Hits): $122.71 (~$0.697 / field)
• Benchmark Test Rerun (95% Local Hits): $14.43 (Repetitive test rerun artifact)
• Mean Compilation Time: 240.2s / field (+63.7% accuracy jump)
• Standard List-Price Compute (0% Cache): $131.33 (~$0.746 / field)
• Candidate Evaluation (5.6-luna): 301.1M tokens ($131.33 gross list price)
• Reflection Mutations: 0 tokens ($0.00 reflection compute)
• Steady-State Pipeline (40% Cache Hits): $78.80 (~$0.448 / field)
• Benchmark Test Rerun (95% Local Hits): $4.22 (Repetitive test rerun artifact)
• Mean Compilation Time: 64.8s / field (Fast baseline bootstrapping)
DSPy Compiler Optimization Levers (Default: Unoptimized / Full Baseline)
All Levers Disabled by DefaultToggle specific DSPy framework levers to meter their exact cost and token reductions relative to the raw unoptimized baseline execution.
Detailed Token Consumption & Telemetry Formula Matrix
| Optimization Phase / Component | Evaluator / Mutation Calls | Gross Input Tokens (100% Processed) | Provider Cached Reads (Active Cache) | Output Reasoning Tokens | Gross List Price (0% Cache Baseline) | Effective Model Cost (w/ Active Cache) |
|---|---|---|---|---|---|---|
| MiPRO (Candidate Task Evaluator) | 183,040 calls | 256.3M tokens | 0 tokens (0% hit) | 44.8M tokens | $131.33 | $131.33 |
| GEPA (Candidate Metric Evaluations) | 176,000 calls | 246.4M tokens | 0 tokens (0% hit) | 43.1M tokens | $126.28 | $126.28 |
| GEPA (Reflection Mutations @ GPT-5.6 Sol) | 1,056 proposals | 3.0M tokens | 0 tokens (Dynamic CoT) | 1.8M tokens | $68.64 | $68.64 |
| Total DSPy GEPA Pipeline | 177,056 calls | 249.4M tokens | 0 tokens (0% hit) | 44.9M tokens | $194.92 | $194.92 |
Comprehensive DSPy Framework Levers Catalog
Full reference of DSPy optimizer parameters, mechanisms, and architectural status.
| DSPy Lever / Parameter | Optimization Mechanism | Status in Stack | Primary Cost / Performance Impact |
|---|---|---|---|
skip_perfect_score |
Error-only reflection gating; avoids LLM teacher calls on passing examples. | Active in GEPA | Reduces reflection compute by 60%–80% proportional to field accuracy. |
reflection_minibatch_size |
Caps failure traces in teacher context window to 3–5 representative errors. | Active in GEPA | Prevents quadratic reflection prompt growth on large validation sets. |
prompt_model vs task_model |
Separates reflection teacher (5.6-sol) from candidate evaluator (5.6-luna). | Active in Hub | Enables 10x cheaper candidate evaluation while maintaining teacher quality. |
max_bootstrapped_demos |
Caps candidate few-shot demonstration examples in prompt prefix. | Active in MiPRO | Saves 600–1,200 input tokens per document extraction call. |
eval_sample_size / Subsampling |
Evaluates intermediate trials on random 20-doc subsets instead of full 100 docs. | Optional Lever | Cuts candidate metric calls linearly by 5x (80% token savings). |
early_stopping_patience |
Terminates trials early when F1 reaches plateau for K consecutive iterations. | Optional Lever | Reduces average iterations from 10 to ~4.5 per field. |
score_threshold_skip |
Bypasses optimizer trials for fields whose zero-shot baseline is already ≥ 0.90. | Optional Lever | Exempts ~30% of standard taxonomy fields from compilation compute. |
joint_signature_optimization |
Multi-field joint signature compilation (e.g. all parties + dates in one prompt). | Architectural Lever | Reduces repeated document chunk scanning by 60% across correlated fields. |
teacher_temperature_annealing |
Decays reflection temperature from 0.7 → 0.1 over iterations. | Optional Lever | Focuses on exploration in early iterations, then converges on precise grammar. |
breadth & depth beam controls |
Controls genetic beam search branch factor in GEPA evolutionary pool. | Configurable | Balances prompt diversity vs evaluator token budget per generation. |
DocuSign Extractions Hub Architecture & Microservices Topology
Production service mesh topology on AKS with Istio mTLS. Explicitly decoupled into Synchronous gRPC / REST Entrypoint Services and Asynchronous Temporal Background Workers.
gRPC & API Entrypoints
OnboardingServiceHandles streaming / unary contract PDF ingestion, zero-shot taxonomy discovery jobs, and agreement matching verification.
AprAiExtractionsServiceSynchronous real-time extractions, custom field descriptors on-demand, clause segmentation, and hierarchy lineage resolution.
FeedbackServiceIngests discrete user feedback corrections with exact glyph sub-word coordinate anchors and Azure Blob ETag OCC template locking.
ObligationsExtractionServiceV2Public APRBridge contract servicer managing tracker instances, tracker suggestions, and bulk review state mutations.
Dedicated Worker Services
apr-obligations-task-queueDurable pipeline worker running BM25 + dense embedding indexing, multi-pass obligation extraction, normalization, and reconciliation.
extractions-review-queueOrchestrates Phase A (Unassisted Candidate Discovery) & Phase B (DSPy GEPA Continuous Instruction Evolution & Multi-Pass Evaluation).
apr-feedback-task-queueProcesses continuous feedback batch sweeps, maintains negative exemplar buffers, and schedules automatic prompt recompilations.
Sibling Platform Services & Core Mesh Integrations
extractions-hub (prefix-isolated)
Bundling Regex Tuning as Part of GEPA / MiPRO
Joint co-optimization of prompt instructions, few-shot demonstration exemplars, and PCRE regex patterns within the DSPy evolutionary compiler graph.
Rather than treating regex extraction as a disconnected heuristic, regex synthesis can and should be bundled directly into the DSPy compilation loop. When candidate predictions produce false positives or boundary misses, GPT-5.6 Sol reflection analyzes the error surface and proposes joint mutations across both natural language instructions and PCRE regex anchors.
Labeled Documents
GEPA / MiPRO Loop
GPT-5.6 Luna ($0.15/M)
Error Boundary Analysis
Prompt + Calibrated Regex
ReDoS AST O(n) Check
Persisted Blob Storage
Extends the task signature to output both extracted spans and candidate PCRE anchors simultaneously:
"""Extract value and synthesize anchor regex."""
document_text: str = dspy.InputField()
extracted_value: str = dspy.OutputField(desc="Canonical extracted field value")
regex_pattern: str = dspy.OutputField(desc="PCRE regex anchoring candidate span")
regex_pattern in the same iteration.
A composite DSPy module containing both RegexPreFilter (Stage 2) and LLMExtractor (Stage 3):
• LLM Extractor Tuning: Maximizes extraction precision on candidate spans passed by the filter.
• Joint Loss Objective: Balances recall-oriented regex breadth with precision-oriented LLM prompt refinement.
Protects extraction worker pods against Regular Expression Denial of Service (ReDoS):
_is_safe_regex()) checking for nested quantifiers like (a+)+ or overlapping disjunctions.• Polynomial Time Guarantee: Enforces deterministic linear $O(n)$ scanning time.
• Artifact Persistence: Validated patterns persist into
FieldPromptArtifact.regex_pattern (models.py:140).
One shared Blob container, isolated by prefixes
BLOB_CONTAINER defaults to extractions-hub. Services do not get separate containers: ownership and lifecycle are expressed by prefixes. Parsed PDF text is the exception: upstream pdf-text-extraction owns a per-account container, and ParsedDocumentStore only reads it.
extractions-hub/ # Single unified container for all system-written persistent artifacts ├── accounts/{account_id}/compiled-programs/ │ ├── latest.json ← Active pointer to winning program │ ├── compiled_program_v3.json ← Winning GEPA compiled state (176 fields) │ ├── compiled_program_v2.json ← Iteration 5 checkpoint │ └── compiled_program_v1.json ← Zero-shot baseline ├── accounts/{account_id}/onboarding/projects/{project_id}/ │ ├── state/ │ │ ├── taxonomy.json ← Discovered & confirmed account field definitions (176 fields) │ │ ├── status.json ← Onboarding wizard stage & readiness flags │ │ └── matches/doc_042_vaulttec_corp.json ← Document-to-record match state │ └── uploads/ground-truth/ground_truth_labels.json ← Parsed ground truth annotations ├── accounts/{account_id}/feedback/{field_id}/ │ └── feedback.json ← Ground-truth seeded & live human feedback entries └── obligations_v2/{account_id}/ ├── results/{agreement_id}/ │ └── extraction_result.json ← Extracted obligations & normalized clauses ├── clustering/ │ └── router_state.json ← Document cluster centroids & routing state └── authoring/ └── phase_b_manifest.json ← Phase A batch partition manifest
Feedback Copy Transition Flow
During onboarding, corrections are collected in the onboarding registry. Once provisioned, they are recursively copied to the live account feedback registry without schema transformations.
{onboarding_id}/{field_id}/
feedback.json
{field_id}/
feedback.json
Temporal Workflow Catalog
Durable workflow structures, exact activity lists, execution multiplicity metrics, and owner Kubernetes worker services registered on https://internal-services.dev.docusign.net/apr-temporal-ui/v1/namespaces/default/workflows.
ai-sdk Proto Service Contracts (Authored by Aaditya Srivathsan)
Complete registry of active gRPC services, RPC methods, and core schema contracts across Obligations Surface and Onboarding Surface in ai-sdk.
Trigger Conditions: Taxonomy & Decision Matrix
Prompt Compilation PoliciesDSPy GEPA prompt optimization evaluates candidate extraction prompts against labeled feedback batches using GPT-5.6 Sol (reflection) and GPT-5.6 Luna (candidate validation). Choosing when to trigger compilation balances compute spend, prompt stability, worker queue utilization, and extraction accuracy.
| Trigger Policy | Trigger Rule | Cost & Token Impact | Latency & Execution | Overfitting Risk | Operational Assessment & Recommendation |
|---|---|---|---|---|---|
| 1. On-Demand Manual (Recommended Baseline) |
Explicit user or admin action via Feedback Hub or gRPC (OptimizePrompt with bounded iteration budget, e.g. 5–10 iterations).
|
Deterministic & Bounded ~$0.11 / field | Asynchronous Temporal workflow (2–4 min) | Low (Manual Review) | Default Policy. Recompiling an entire 100-field taxonomy costs ~24.8M tokens (~$10.90) across 10 iterations. Automatic triggers per single feedback entry create unpredictable compute charges and worker queue contention. Manual triggering provides predictable budget control. |
| 2. Annotation Delta Threshold (Batch Threshold ≥ N) |
Background worker fires when a field accumulates Δ ≥ 10 verified feedback pairs since its last compilation run.
|
Proportional to Volume Rate-limited by Δ | Low-priority background queue | Low (≥10 Samples) | Recommended for Active Accounts. GEPA error reflection needs at least 10 distinct failure examples to discover generalized pattern rules. Compiling on 1–2 corrections overfits to specific document phrasing and degrades accuracy across the rest of the corpus. |
| 3. Quality Drift / SLA Circuit (Telemetry Guardrail) |
Telemetry monitor detects rolling field F1 dropping below threshold (F1 < 0.75) or human rejection rate > 20% over last 50 extractions.
|
High if Uncapped Needs circuit breaker | Automated alert & remediation queue | Medium (Drift Context) | Automated Remediation. Detects format shifts when new vendor contract layouts enter production. Must enforce a cooldown limit (max 1 auto-run per 24 hours per field) to prevent retry loops on inherently unextractable fields. |
| 4. Scheduled Off-Peak Cron (Nightly Batch Sweep) |
Nightly cron (e.g. 0 2 * * * UTC) sweeps all fields marked dirty with pending feedback diffs.
|
Predictable Schedule Off-peak execution | Nightly batch window (2–5 AM) | Low (Aggregated) | Enterprise Standard. Consolidates non-urgent feedback into fixed off-peak windows. Concurrency is limited to 5 parallel fields per tenant to stay within Azure OpenAI TPM limits and keep worker capacity free for daytime extraction traffic. |
| 5. Onboarding Milestone Gate (Phase B Finalization) | Fires automatically when the user validates legacy CLM ground-truth import and confirms onboarding Phase B. | One-Time Provisioning ~$10.90 / 100 fields | Onboarding workflow step (Phase B) | Low (Golden Baseline) |
Account Initialization. Compiles naive baseline instructions into optimized v1 programs using the imported ground truth before live document processing starts. Writes initial compiled_program_v1.json to Blob Storage.
|
For sparse fields (present in <20% of documents), evaluate on a balanced set of 10 positive and 10 hard negative documents. Evaluating only on positive examples causes prompt instructions to become overly permissive, increasing false positives on documents where the clause is absent.
Limit compilation concurrency to 5–10 parallel fields per account. Unconstrained parallel compilation risks exhausting Azure OpenAI TPM quotas and starving extraction worker pools during high-throughput ingestion periods.
Write compiled programs to versioned immutable blobs (compiled_program_v{N}.json). Update the active pointer (latest.json) only when candidate evaluation score improves by ≥0.02 F1 over the current production baseline.