Cómo rediseñamos la selección de currículums con un flujo de trabajo nativo en IA

This post is about a small experiment: hooking our Notion hiring database up to Claude Code and running a dynamic workflow that dispatches 100+ AI agents in parallel to read resumes, score them against a consistent rubric, and cross-check each other's judgments — producing a ranked shortlist we could act on immediately.

The whole thing cost $65 and ran in about 13 minutes across 115 candidates. But more interesting than the cost were the methodological questions it surfaced — when to use a fleet of agents instead of one, how to prevent AI score inflation, and what it means to encode "excellence" into something a machine can actually execute.

1. What Is a Dynamic Workflow

Let's start with the concept, because it's the foundation everything else rests on.

Most AI use today follows a prompt-and-reply pattern: send a message, get an answer, iterate. This works well for one-off tasks, but it gets awkward when you need to do the same thing to 115 objects — you're either copying and pasting 115 times, or asking a single conversation to process them sequentially, which gets slower and noisier as it goes.

A dynamic workflow is a different model: code that orchestrates a fleet of AI agents. Its defining properties are:

• Deterministic control flow + AI judgment, kept separate. Loops, dispatch, aggregation, and quota enforcement are handled by code (reproducible, auditable); subjective judgment (is this resume strong enough?) is delegated to AI agents.
• Fan-out parallelism. A single parallel(...) call can spin up dozens or hundreds of independent agents simultaneously, each working on its own slice without contaminating the others.
• Multi-stage pipelines. The output of one stage feeds the next. Code handles filtering, ranking, and deduplication between stages.
• Structured output. Each agent returns JSON conforming to a schema — not freeform chat text — so downstream code can consume it directly.

An analogy: a single conversation is like consulting one expert for an afternoon. A dynamic workflow is like assembling a 125-person review panel, issuing each member a rubric and one candidate file, running all reviews in parallel, cross-checking the top results, and aggregating into a ranked list — with the assembly, dispatch, and aggregation logic baked into the script.

Resume screening is a natural fit for this pattern: high volume, uniform criteria, subjective judgment, fairness requirement.

For a deeper technical introduction: A harness for every task: dynamic workflows in Claude Code

2. The Hiring Workflow: Goals and Design

The problem

We had a concrete pain point: over a hundred candidates sitting in "initial review" status in our Notion hiring database, with no realistic way to process them manually without standard drift — the bar you apply at resume 80 is almost never the same as the bar at resume 5.

I wanted to test one specific idea: can we abstract "what great looks like in the AI-agent era" into a machine-executable, human-readable rubric, and then run all 115 candidates through it at the same calibration?

The goal was explicitly not to have AI make hiring decisions. The goal was:

1. Compress 115 candidates into a ranked, justified shortlist so human attention goes to the people who actually deserve it.
2. Make the criteria transparent and iterable — if the output is wrong, you change a Markdown file, not code or instinct.

Three key design decisions

Decision 1: Criteria and code are completely separate

Evaluation criteria live in standalone Markdown files (criteria/), not embedded in the workflow code. Anyone — including non-technical teammates — can change screening behavior by editing these files:

criteria/
├── 00-philosophy.md          Overall philosophy: what we're hiring for + the "raise the bar" rule
├── 01-pedigree.md            Strong academic / early foundation    (weight 20%)
├── 02-ai-agent-fluency.md    AI-native capability                  (weight 35%)
├── 03-grit-problem-solving.md  Problem-solving & overcoming difficulty  (weight 30%)
├── 04-talent-lens.md         Top-talent signal                     (weight 15%)
└── scoring.md                Scoring formula + grade bands + 5% quota rule

These four dimensions are our "excellence standard for the AI-agent era, v0.1". The thinking behind each:

• AI-native capability carries the highest weight (35%). In 2026, whether someone genuinely uses agentic tools like Claude Code as a core part of how they work is a major productivity divider. We specifically penalize keyword-stuffing — listing "Claude Code" without verifiable project evidence is treated as a weak signal.
• Hard evidence of problem-solving (30%). We look for "scar tissue": things built independently from scratch, narratives of overcoming real obstacles — not tutorial-level reproductions.
• Strong foundation (20%). Academic background serves as a proxy for raw potential — it's a signal, not a requirement. A selective-university degree paired with mediocre output gets penalized; a self-taught builder with no prestigious credential but real shipped work gets a boost.
• Top-talent signal (15%). This dimension is deliberately subjective. The prompt asks: would a team like Anthropic's or a founder like Musk immediately want to reach out? It captures agency, taste, and velocity that the other three dimensions don't.

Decision 2: Encode "raise the bar" as a hard constraint, not a slogan

scoring.md includes a firm rule: candidates reaching the top tier (S) must be ≤ 5% of the full pool. After all scoring is complete, the code applies a global cap: even if many candidates technically score in the S range, only the top 5% are allowed through. This directly fights a known failure mode — AI scoring is naturally lenient. Without a hard constraint, it will grade half the pool as "excellent."

Decision 3: Add adversarial review to catch inflated scores

Scoring alone isn't enough. A single scoring agent can get pulled along by impressive-sounding keywords — "published in top journal," "built my own framework." So the top-ranked candidates go through a second pass: a panel of "devil's advocate" agents whose explicit job is to argue against "this person deserves a top-tier rating" and push scores down wherever the evidence doesn't fully support them.

The workflow

Setup 📋 Notion hiring database — Notion CLI pull → one structured data file per candidate

AI Phase 1: Scoring (115 agents in parallel)

• Reads 6 criteria MD files + that candidate's data file
• Actively visits GitHub / portfolio links to verify evidence
• Outputs structured JSON: 4-dimension scores + reasoning + highlights + risk flags

Code Deterministic Synthesis

• Computes weighted totals
• Global rank sort, calculates 5% quota slots
• Selects top candidates for adversarial review queue

AI Phase 2: Adversarial Review (Agents in parallel)

• "Devil's advocate" persona reviews each top candidate
• Argues against top-tier designation
• Pushes scores down where evidence is insufficient

Code Deterministic Verdict

• Re-sorts using calibrated scores
• Enforces 5% hard cap
• Assigns final grade bands: S / A / B / C / D

Output: Ranked Report Structured Markdown with per-candidate scores, reasoning, and adversarial review verdict

Blue stages (scoring / review) are AI. Gray stages (synthesis / verdict) are code. This split is intentional: anything mathematical — weighting, ranking, quota enforcement — goes to code for reproducibility; anything requiring judgment — is this person strong enough? — goes to AI.

3. What We Saw: Results and Insights

All candidates below have been anonymized. We describe the type of work, not names or identifying details.

What we ran

Distribution

The 5% quota (5 slots) went completely unused — it wasn't the quota that blocked anyone; the absolute score threshold did. No one cleared the A-band floor on their own. More on why that's actually a useful signal below.

What the top of the ranking looked like (anonymized)

Without exception, the highest-ranked candidates were people who had actually built agents — not people who had heard about AI:

• #1: A graduate student who built a Claude-Code-style multi-agent workbench from scratch — including agent main loop, tool call parsing, context compression, sub-agent spawning, and safety gates. All verifiable code, not descriptions.
• #2: Another graduate student who had deployed a real, publicly accessible multi-agent system (vertical-domain application), with academic output layered on top.
• Further down: someone who wrote an agent orchestration engine in Go from scratch; someone who shipped a lightweight coding agent by studying Claude Code's architecture; someone who independently built a game with a local LLM in seven days using AI tooling throughout.

What they had in common: their strong signals almost never appeared in the resume body — they were in GitHub repos and portfolios. This is exactly why each scoring agent was instructed to actively visit links and verify evidence rather than just reading the resume text.

Three insights

Insight 1: Adversarial review genuinely caught inflated scores

The clearest example was the top two candidates. After the scoring phase, both had weighted totals around 82 points — enough to push into the A band and graze the S threshold. After adversarial review, both landed around 75 points, with very specific reasoning:

"Built a verifiable multi-agent workbench — AI-native capability is a hard signal. But the project is ~3 weeks old, single contributor, 0 stars, no tests. Conceptually a reimplementation, not original problem-solving. Almost no supporting evidence beyond the degree line: a solid high-potential candidate, but not exceptional."

"A genuine, verifiable AI-native builder. But the claimed top-journal publication appears only in recruiter notes, with no independently verifiable source. Core system backend is private; individual contribution cannot be confirmed. Using unverified academic credentials to reach for top tier is keyword-driven score inflation."

This is exactly what the design was meant to do: it didn't dismiss these candidates — it walked scores back to what the evidence can actually support. A single scoring agent can get carried away; a separate agent panel whose job is to push back deflates that reliably.

Insight 2: S:0 / A:0 is not a bug — it's a mirror

The first instinct is to ask whether the bar was set wrong. But looking at the pool honestly:

• A large fraction of candidates had very sparse resumes — key dimensions (AI experience, verifiable work) simply absent.
• Many applicants for Agent Engineer roles had zero evidence of agentic tool use and no GitHub link.
• The pool also contained recruiter business emails and LinkedIn system notifications — these were correctly identified as irrelevant and scored 0, which incidentally revealed our hiring database needed cleaning.

In other words, a strict rubric cleanly separated signal from noise. The real builders (top 6) and "accomplished generalists" (middle tier) ended up in clearly distinct places. That's the point — miss a few rather than inflate everyone.

This also surfaces an open question worth discussing: is the current A-band threshold (78 points) too harsh for candidates who are students with strong GitHub records but no professional track record yet? Interestingly, the adversarial review agents themselves described the top two as "high-potential candidates" — but the weighted score kept them in the B band. Whether to relax that threshold for high-potential early-career candidates is a judgment call best made after we see the actual interview quality from the B group. The good news: that change is one number in one Markdown file. No code required.

Insight 3: "Criteria as code" makes disagreement productive

Conversations about hiring standards usually stay vague — "we want people with drive," "someone who can figure things out." Because this rubric is written down with weights and anchoring examples, the conversation immediately gets concrete: "Should AI capability be 35% or 40%?" "How much does an outlier builder without a prestigious degree actually gain?" "Should the quota be 5% or 8%?" — every disagreement corresponds to a specific line in a Markdown file that can be changed, versioned, and debated. The standard becomes an asset you maintain, not a consensus you repeat in every meeting.

4. Cost and ROI

Exact spend

We used Claude Opus 4.8 (top tier). Precise breakdown by token category:

Eso equivale a aproximadamente $0.57 por candidato.

Un hallazgo contraintuitivo: las escrituras de caché son la partida más grande

La suposición natural es que, como los 115 agentes están leyendo los mismos 6 archivos de criterios, el almacenamiento en caché de prompts debería ayudar mucho. No es así, al menos no de la manera que uno esperaría.

El almacenamiento en caché de prompts funciona por coincidencia exacta de prefijo, y cada sesión de agente es independiente. 125 agentes significan 125 sesiones independientes — cada una con una descripción de tarea distinta (datos de un candidato diferente) — así que una caché escrita por el agente A no puede ser reutilizada por el agente B. El caching sí ayuda dentro de la propia ejecución multi-turno de cada agente (leer criterios → visitar GitHub → visitar portafolio → generar salida, releyendo contenido anterior en cada ronda).

Esto revela una disyuntiva arquitectónica: el paralelismo fan-out multiplica los costos de escritura de caché (cada agente construye su propia caché), pero a cambio obtienes un juicio aislado, no contaminado, y eliminas la acumulación cuadrática de contexto propia del procesamiento secuencial. Para tareas sensibles a la calidad del juicio, esa disyuntiva vale la pena.

Cómo pensar sobre el ROI

Comparación directa con la revisión manual: un gerente de contratación leyendo un currículum con cuidado, revisando el GitHub y escribiendo notas — conservadoramente entre 5 y 10 minutos por candidato. En 115 candidatos, eso son entre 10 y 19 horas de trabajo enfocado, con los estándares desviándose todo el tiempo.

Este workflow entregó:

Pero el ROI más importante es el de la atención: redirigió el enfoque humano lejos de los 83 candidatos claramente no aptos y hacia los 6 constructores genuinos en la parte superior. Eso es lo de mayor valor que puede aportar una selección inicial.

¿Podría ser más barato?

Sí, pero probablemente no haga falta. Si esto se convirtiera en una operación de alta frecuencia y alto volumen (cientos de candidatos al día), la optimización práctica sería:

• Usar Sonnet para la fase de puntuación, y Opus solo para la revisión adversarial — probablemente una reducción de costos del 70 al 80% con una pérdida mínima de calidad.
• O usar un modelo más económico para una primera pasada rápida, y luego Opus para la evaluación detallada de los mejores candidatos.

Pero la contratación es de baja frecuencia, alto riesgo y difícil de deshacer. Por $65 para procesar todo un pipeline con auditabilidad completa y criterios iterables, la conclusión es clara: usa el mejor modelo. No sacrifiques la calidad del juicio por ahorros marginales de costo.

El panorama más amplio

Lo verdaderamente emocionante de este experimento no es que "la IA pueda seleccionar currículums" — esa no es una idea nueva. Es que el modelo de workflow dinámico — código que orquesta una flota de agentes de IA — hace que ciertas categorías de trabajo sean estructurables, reproducibles e iterables por primera vez.

La contratación es solo el punto de entrada. El mismo patrón — criterios como archivos legibles + evaluación paralela fan-out + revisión adversarial + agregación determinista — se traslada a cualquier dominio donde necesites tomar decisiones subjetivas consistentes y de alto volumen: moderación de contenido, revisión de código, clasificación de comentarios de usuarios, análisis competitivo, debida diligencia.

La rúbrica es v0.1. No es perfecta. Pero ahora es un activo versionado, debatible y mejorable — no un acuerdo implícito que vive en la cabeza de alguien. Ese cambio, más que cualquier resultado individual, es de lo que realmente trataba este experimento.