Full-time

Member of Technical Staff, Evals

San Francisco, California, United States · On-site

USD 200,000 - 300,000

AI Agents | LLM Evaluation | San Francisco

How do you know when an AI agent is genuinely ready for production?

Not just performing well on benchmarks, but consistently completing real tasks, retrieving the right information and avoiding mistakes that could cost customers money.

This is an opportunity to take ownership of that challenge, building the evaluation infrastructure that determines whether AI agents are reliable enough to deploy.

You'll join an early-stage AI company developing autonomous agents for logistics operations, tackling complex back-office workflows including appointment scheduling, driver support and load management.

Their mission is to build hallucination-free AI for environments where accuracy matters. They're developing agents that operate across text, email and voice, with the longer-term ambition of automating hundreds of operational workflows.

The opportunity

As a Member of Technical Staff focused on Evals, you'll define how agent quality is measured, tested and continuously improved.

This goes well beyond running standard LLM benchmarks.

You'll build the evaluation infrastructure used across the company, from engineers developing agents to researchers training proprietary models, and eventually customers validating their own workflows.

The challenge is translating complex, real-world interactions into measurable outcomes that accurately reflect whether an agent can do its job.

An agent might complete a task successfully once, but can it do so consistently? Is its response grounded in the right information? What happens when conversations become unpredictable or tools return unexpected results?

You'll be responsible for answering those questions with reliable evaluation systems.

Your technical focus:

  • Build evaluation frameworks measuring task accuracy, pass^k consistency and retrieval quality across individual agents and end-to-end workflows.

  • Develop golden datasets, expert-validated graders and LLM-as-judge systems, including calibration, inter-rater agreement and error severity classification.

  • Implement automated regression testing in CI, with trace-level analysis identifying failures across conversations, tool calls and retrieval.

  • Create simulated users and adversarial test environments for multi-turn text and voice agents, including difficult edge cases drawn from real interactions.

A particularly interesting part of this role is connecting offline evaluations with what actually happens in production.

You'll analyse production traces, monitor drift and determine whether improvements in evaluation metrics translate into better task completion and automation rates.

You'll also help develop self-service evaluation tools, allowing internal teams and customers to create test cases, review failures and validate agents without relying on an engineer for every change.

Working alongside the research team, you'll evaluate proprietary model checkpoints and fine-tunes against frontier models, helping determine where improvements are meaningful and where further work is needed.

What you'll bring

You'll have experience deploying LLM-powered products into production and, importantly, building the evaluation systems that determine whether those products are ready to ship.

You'll understand how to turn messy real-world conversations and documents into reliable ground truth, particularly when there isn't a single correct answer.

Experience developing LLM-as-judge frameworks, evaluating agentic workflows and validating automated graders against human assessments would be particularly relevant.

Strong Python engineering skills, applied statistics and experience integrating evaluations into CI pipelines would also be valuable.

Perhaps you've built simulated user environments, adversarial testing systems or evaluation infrastructure for conversational AI products.

What matters most is that you've taken responsibility for model or agent quality in a production environment, not simply reported benchmark scores.

Why consider it?

You'll own a core technical capability rather than contribute to an isolated evaluation project.

The infrastructure you build will influence every agent release, how the company measures reliability and how its own models are developed.

It's an opportunity to establish evaluation standards from the ground up, with direct responsibility for the systems that determine what ships.

Package and location

Base salary: $200,000–$300,000
Equity: Meaningful equity package
Location: San Francisco, fully onsite

If you've built production LLM systems and want to own the evaluation infrastructure behind reliable AI agents, we'd be happy to discuss the opportunity.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 200,000 - 300,000

Location

San Francisco, California, United States · On-site

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

Full-time

Member of Technical Staff, Evals

San Francisco, California, United States · On-site

USD 200,000 - 300,000

AI Agents | LLM Evaluation | San Francisco

How do you know when an AI agent is genuinely ready for production?

Not just performing well on benchmarks, but consistently completing real tasks, retrieving the right information and avoiding mistakes that could cost customers money.

This is an opportunity to take ownership of that challenge, building the evaluation infrastructure that determines whether AI agents are reliable enough to deploy.

You'll join an early-stage AI company developing autonomous agents for logistics operations, tackling complex back-office workflows including appointment scheduling, driver support and load management.

Their mission is to build hallucination-free AI for environments where accuracy matters. They're developing agents that operate across text, email and voice, with the longer-term ambition of automating hundreds of operational workflows.

The opportunity

As a Member of Technical Staff focused on Evals, you'll define how agent quality is measured, tested and continuously improved.

This goes well beyond running standard LLM benchmarks.

You'll build the evaluation infrastructure used across the company, from engineers developing agents to researchers training proprietary models, and eventually customers validating their own workflows.

The challenge is translating complex, real-world interactions into measurable outcomes that accurately reflect whether an agent can do its job.

An agent might complete a task successfully once, but can it do so consistently? Is its response grounded in the right information? What happens when conversations become unpredictable or tools return unexpected results?

You'll be responsible for answering those questions with reliable evaluation systems.

Your technical focus:

  • Build evaluation frameworks measuring task accuracy, pass^k consistency and retrieval quality across individual agents and end-to-end workflows.

  • Develop golden datasets, expert-validated graders and LLM-as-judge systems, including calibration, inter-rater agreement and error severity classification.

  • Implement automated regression testing in CI, with trace-level analysis identifying failures across conversations, tool calls and retrieval.

  • Create simulated users and adversarial test environments for multi-turn text and voice agents, including difficult edge cases drawn from real interactions.

A particularly interesting part of this role is connecting offline evaluations with what actually happens in production.

You'll analyse production traces, monitor drift and determine whether improvements in evaluation metrics translate into better task completion and automation rates.

You'll also help develop self-service evaluation tools, allowing internal teams and customers to create test cases, review failures and validate agents without relying on an engineer for every change.

Working alongside the research team, you'll evaluate proprietary model checkpoints and fine-tunes against frontier models, helping determine where improvements are meaningful and where further work is needed.

What you'll bring

You'll have experience deploying LLM-powered products into production and, importantly, building the evaluation systems that determine whether those products are ready to ship.

You'll understand how to turn messy real-world conversations and documents into reliable ground truth, particularly when there isn't a single correct answer.

Experience developing LLM-as-judge frameworks, evaluating agentic workflows and validating automated graders against human assessments would be particularly relevant.

Strong Python engineering skills, applied statistics and experience integrating evaluations into CI pipelines would also be valuable.

Perhaps you've built simulated user environments, adversarial testing systems or evaluation infrastructure for conversational AI products.

What matters most is that you've taken responsibility for model or agent quality in a production environment, not simply reported benchmark scores.

Why consider it?

You'll own a core technical capability rather than contribute to an isolated evaluation project.

The infrastructure you build will influence every agent release, how the company measures reliability and how its own models are developed.

It's an opportunity to establish evaluation standards from the ground up, with direct responsibility for the systems that determine what ships.

Package and location

Base salary: $200,000–$300,000
Equity: Meaningful equity package
Location: San Francisco, fully onsite

If you've built production LLM systems and want to own the evaluation infrastructure behind reliable AI agents, we'd be happy to discuss the opportunity.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 200,000 - 300,000

Location

San Francisco, California, United States · On-site

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

Full-time

Member of Technical Staff, Evals

San Francisco, California, United States · On-site

USD 200,000 - 300,000

AI Agents | LLM Evaluation | San Francisco

How do you know when an AI agent is genuinely ready for production?

Not just performing well on benchmarks, but consistently completing real tasks, retrieving the right information and avoiding mistakes that could cost customers money.

This is an opportunity to take ownership of that challenge, building the evaluation infrastructure that determines whether AI agents are reliable enough to deploy.

You'll join an early-stage AI company developing autonomous agents for logistics operations, tackling complex back-office workflows including appointment scheduling, driver support and load management.

Their mission is to build hallucination-free AI for environments where accuracy matters. They're developing agents that operate across text, email and voice, with the longer-term ambition of automating hundreds of operational workflows.

The opportunity

As a Member of Technical Staff focused on Evals, you'll define how agent quality is measured, tested and continuously improved.

This goes well beyond running standard LLM benchmarks.

You'll build the evaluation infrastructure used across the company, from engineers developing agents to researchers training proprietary models, and eventually customers validating their own workflows.

The challenge is translating complex, real-world interactions into measurable outcomes that accurately reflect whether an agent can do its job.

An agent might complete a task successfully once, but can it do so consistently? Is its response grounded in the right information? What happens when conversations become unpredictable or tools return unexpected results?

You'll be responsible for answering those questions with reliable evaluation systems.

Your technical focus:

  • Build evaluation frameworks measuring task accuracy, pass^k consistency and retrieval quality across individual agents and end-to-end workflows.

  • Develop golden datasets, expert-validated graders and LLM-as-judge systems, including calibration, inter-rater agreement and error severity classification.

  • Implement automated regression testing in CI, with trace-level analysis identifying failures across conversations, tool calls and retrieval.

  • Create simulated users and adversarial test environments for multi-turn text and voice agents, including difficult edge cases drawn from real interactions.

A particularly interesting part of this role is connecting offline evaluations with what actually happens in production.

You'll analyse production traces, monitor drift and determine whether improvements in evaluation metrics translate into better task completion and automation rates.

You'll also help develop self-service evaluation tools, allowing internal teams and customers to create test cases, review failures and validate agents without relying on an engineer for every change.

Working alongside the research team, you'll evaluate proprietary model checkpoints and fine-tunes against frontier models, helping determine where improvements are meaningful and where further work is needed.

What you'll bring

You'll have experience deploying LLM-powered products into production and, importantly, building the evaluation systems that determine whether those products are ready to ship.

You'll understand how to turn messy real-world conversations and documents into reliable ground truth, particularly when there isn't a single correct answer.

Experience developing LLM-as-judge frameworks, evaluating agentic workflows and validating automated graders against human assessments would be particularly relevant.

Strong Python engineering skills, applied statistics and experience integrating evaluations into CI pipelines would also be valuable.

Perhaps you've built simulated user environments, adversarial testing systems or evaluation infrastructure for conversational AI products.

What matters most is that you've taken responsibility for model or agent quality in a production environment, not simply reported benchmark scores.

Why consider it?

You'll own a core technical capability rather than contribute to an isolated evaluation project.

The infrastructure you build will influence every agent release, how the company measures reliability and how its own models are developed.

It's an opportunity to establish evaluation standards from the ground up, with direct responsibility for the systems that determine what ships.

Package and location

Base salary: $200,000–$300,000
Equity: Meaningful equity package
Location: San Francisco, fully onsite

If you've built production LLM systems and want to own the evaluation infrastructure behind reliable AI agents, we'd be happy to discuss the opportunity.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 200,000 - 300,000

Location

San Francisco, California, United States · On-site

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

techire AI website footer image