Full-time

Research Engineer, Benchmarks

San Francisco, California, United States · Remote

USD 100,000 - 230,000

Frontier AI Agents | Benchmarking & Evaluations

A benchmark is only useful if it actually tells you something meaningful about a model's capabilities.

As AI agents become more capable, evaluating them is becoming increasingly difficult. It's one thing to measure performance on a predefined task, but quite another to determine whether an agent can reliably handle complex, real-world workflows.

This is an opportunity to tackle that problem, building rigorous evaluation benchmarks that help frontier AI labs understand what their agents can actually do, where they fail and how their capabilities are improving.

You'll join a well-funded AI startup building the data infrastructure and evaluation systems supporting the development of frontier AI models.

Backed by $16M in funding, including Y Combinator, they're developing a platform used by frontier AI labs, startups and Fortune 500 companies.

The opportunity

As a Research Engineer, you'll take ownership of developing high-quality benchmarks for evaluating frontier agents across domain-specific tasks.

This isn't simply about creating datasets and running existing evaluation frameworks.

You'll be investigating what makes a benchmark genuinely useful, from designing realistic tasks and reliable scoring methods to understanding how benchmark results translate into real-world performance.

One of the more interesting challenges is developing evaluations for complex workflows where success isn't always straightforward to measure.

You'll work with subject-matter experts to understand how tasks are performed in different domains, then translate those workflows into reproducible environments that meaningfully test agent capabilities.

The goal is to create benchmarks that frontier AI researchers can trust, rather than another collection of scores that look impressive but reveal very little.

Your technical focus:

  • Design and develop agent benchmarks that evaluate realistic, domain-specific workflows, working alongside subject-matter experts to define meaningful tasks.

  • Build infrastructure for reliably running models and agents through benchmark environments, ensuring evaluations are reproducible and consistent.

  • Develop scoring methods and analyses to understand benchmark difficulty, reliability, edge cases and model failure modes.

  • Investigate how benchmark performance correlates with real-world evaluations and produce technical reports that make results credible and interpretable.

You'll have significant ownership over the entire process, from initial task design and environment development through to evaluation, analysis and documentation.

A particularly important part of the role is understanding why a benchmark works.

Can an agent achieve a high score without genuinely completing the intended task? Are there inconsistencies in the evaluation? Does the benchmark discriminate meaningfully between different model capabilities?

You'll need to think critically about these questions and build evaluation systems that stand up to scrutiny from experienced AI researchers.

What you'll bring

You'll have strong software engineering skills, particularly Python, alongside experience working with Docker and Linux environments.

Ideally, you'll have worked on agent environments, model evaluations or benchmarking systems and understand the challenges involved in assessing modern AI capabilities.

Particularly relevant experience includes:

  • Developing benchmarks, agent environments or evaluation infrastructure.

  • Understanding benchmark design, task realism, scoring reliability and failure analysis.

  • Researching model limitations or evaluating agent performance across complex tasks.

  • Publishing technical research, papers or detailed engineering articles on benchmarks, evaluations or related ML topics.

You don't necessarily need years of experience or an extensive publication record.

What's more important is your ability to investigate unfamiliar technical problems, understand how real workflows operate and build robust systems that provide meaningful results.

Why consider it?

You'll join a team of approximately 25 people, including researchers with publications at ICLR and NeurIPS, international Olympiad medallists and experienced AI startup founders.

The company is growing, with an opportunity to influence how its benchmarks are designed, evaluated and used by frontier AI teams.

You'll have genuine ownership of technically demanding work, with the freedom to investigate problems rather than simply implement predefined solutions.

Compensation and location

Base salary:

  • San Francisco / major US cities: $135,000–$230,000

  • Singapore: $100,000–$175,000

  • Rest of world: $100,000–$175,000

Location: San Francisco or Singapore, with remote options available for candidates who can overlap 70–80% with either office's working hours.

Visa sponsorship and relocation support are available for suitable candidates.

Benefits include fully covered medical, dental and vision insurance for US and Singapore employees, access to AI development tools, office meals and an annual travel budget.

The interview process involves two technical interviews and a 2–3 day work trial.

If you've been working on agent evaluations or benchmarking and want to help define how frontier AI capabilities are measured, we'd be happy to discuss the role.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 100,000 - 230,000

Location

San Francisco, California, United States · Remote

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

Full-time

Research Engineer, Benchmarks

San Francisco, California, United States · Remote

USD 100,000 - 230,000

Frontier AI Agents | Benchmarking & Evaluations

A benchmark is only useful if it actually tells you something meaningful about a model's capabilities.

As AI agents become more capable, evaluating them is becoming increasingly difficult. It's one thing to measure performance on a predefined task, but quite another to determine whether an agent can reliably handle complex, real-world workflows.

This is an opportunity to tackle that problem, building rigorous evaluation benchmarks that help frontier AI labs understand what their agents can actually do, where they fail and how their capabilities are improving.

You'll join a well-funded AI startup building the data infrastructure and evaluation systems supporting the development of frontier AI models.

Backed by $16M in funding, including Y Combinator, they're developing a platform used by frontier AI labs, startups and Fortune 500 companies.

The opportunity

As a Research Engineer, you'll take ownership of developing high-quality benchmarks for evaluating frontier agents across domain-specific tasks.

This isn't simply about creating datasets and running existing evaluation frameworks.

You'll be investigating what makes a benchmark genuinely useful, from designing realistic tasks and reliable scoring methods to understanding how benchmark results translate into real-world performance.

One of the more interesting challenges is developing evaluations for complex workflows where success isn't always straightforward to measure.

You'll work with subject-matter experts to understand how tasks are performed in different domains, then translate those workflows into reproducible environments that meaningfully test agent capabilities.

The goal is to create benchmarks that frontier AI researchers can trust, rather than another collection of scores that look impressive but reveal very little.

Your technical focus:

  • Design and develop agent benchmarks that evaluate realistic, domain-specific workflows, working alongside subject-matter experts to define meaningful tasks.

  • Build infrastructure for reliably running models and agents through benchmark environments, ensuring evaluations are reproducible and consistent.

  • Develop scoring methods and analyses to understand benchmark difficulty, reliability, edge cases and model failure modes.

  • Investigate how benchmark performance correlates with real-world evaluations and produce technical reports that make results credible and interpretable.

You'll have significant ownership over the entire process, from initial task design and environment development through to evaluation, analysis and documentation.

A particularly important part of the role is understanding why a benchmark works.

Can an agent achieve a high score without genuinely completing the intended task? Are there inconsistencies in the evaluation? Does the benchmark discriminate meaningfully between different model capabilities?

You'll need to think critically about these questions and build evaluation systems that stand up to scrutiny from experienced AI researchers.

What you'll bring

You'll have strong software engineering skills, particularly Python, alongside experience working with Docker and Linux environments.

Ideally, you'll have worked on agent environments, model evaluations or benchmarking systems and understand the challenges involved in assessing modern AI capabilities.

Particularly relevant experience includes:

  • Developing benchmarks, agent environments or evaluation infrastructure.

  • Understanding benchmark design, task realism, scoring reliability and failure analysis.

  • Researching model limitations or evaluating agent performance across complex tasks.

  • Publishing technical research, papers or detailed engineering articles on benchmarks, evaluations or related ML topics.

You don't necessarily need years of experience or an extensive publication record.

What's more important is your ability to investigate unfamiliar technical problems, understand how real workflows operate and build robust systems that provide meaningful results.

Why consider it?

You'll join a team of approximately 25 people, including researchers with publications at ICLR and NeurIPS, international Olympiad medallists and experienced AI startup founders.

The company is growing, with an opportunity to influence how its benchmarks are designed, evaluated and used by frontier AI teams.

You'll have genuine ownership of technically demanding work, with the freedom to investigate problems rather than simply implement predefined solutions.

Compensation and location

Base salary:

  • San Francisco / major US cities: $135,000–$230,000

  • Singapore: $100,000–$175,000

  • Rest of world: $100,000–$175,000

Location: San Francisco or Singapore, with remote options available for candidates who can overlap 70–80% with either office's working hours.

Visa sponsorship and relocation support are available for suitable candidates.

Benefits include fully covered medical, dental and vision insurance for US and Singapore employees, access to AI development tools, office meals and an annual travel budget.

The interview process involves two technical interviews and a 2–3 day work trial.

If you've been working on agent evaluations or benchmarking and want to help define how frontier AI capabilities are measured, we'd be happy to discuss the role.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 100,000 - 230,000

Location

San Francisco, California, United States · Remote

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

Full-time

Research Engineer, Benchmarks

San Francisco, California, United States · Remote

USD 100,000 - 230,000

Frontier AI Agents | Benchmarking & Evaluations

A benchmark is only useful if it actually tells you something meaningful about a model's capabilities.

As AI agents become more capable, evaluating them is becoming increasingly difficult. It's one thing to measure performance on a predefined task, but quite another to determine whether an agent can reliably handle complex, real-world workflows.

This is an opportunity to tackle that problem, building rigorous evaluation benchmarks that help frontier AI labs understand what their agents can actually do, where they fail and how their capabilities are improving.

You'll join a well-funded AI startup building the data infrastructure and evaluation systems supporting the development of frontier AI models.

Backed by $16M in funding, including Y Combinator, they're developing a platform used by frontier AI labs, startups and Fortune 500 companies.

The opportunity

As a Research Engineer, you'll take ownership of developing high-quality benchmarks for evaluating frontier agents across domain-specific tasks.

This isn't simply about creating datasets and running existing evaluation frameworks.

You'll be investigating what makes a benchmark genuinely useful, from designing realistic tasks and reliable scoring methods to understanding how benchmark results translate into real-world performance.

One of the more interesting challenges is developing evaluations for complex workflows where success isn't always straightforward to measure.

You'll work with subject-matter experts to understand how tasks are performed in different domains, then translate those workflows into reproducible environments that meaningfully test agent capabilities.

The goal is to create benchmarks that frontier AI researchers can trust, rather than another collection of scores that look impressive but reveal very little.

Your technical focus:

  • Design and develop agent benchmarks that evaluate realistic, domain-specific workflows, working alongside subject-matter experts to define meaningful tasks.

  • Build infrastructure for reliably running models and agents through benchmark environments, ensuring evaluations are reproducible and consistent.

  • Develop scoring methods and analyses to understand benchmark difficulty, reliability, edge cases and model failure modes.

  • Investigate how benchmark performance correlates with real-world evaluations and produce technical reports that make results credible and interpretable.

You'll have significant ownership over the entire process, from initial task design and environment development through to evaluation, analysis and documentation.

A particularly important part of the role is understanding why a benchmark works.

Can an agent achieve a high score without genuinely completing the intended task? Are there inconsistencies in the evaluation? Does the benchmark discriminate meaningfully between different model capabilities?

You'll need to think critically about these questions and build evaluation systems that stand up to scrutiny from experienced AI researchers.

What you'll bring

You'll have strong software engineering skills, particularly Python, alongside experience working with Docker and Linux environments.

Ideally, you'll have worked on agent environments, model evaluations or benchmarking systems and understand the challenges involved in assessing modern AI capabilities.

Particularly relevant experience includes:

  • Developing benchmarks, agent environments or evaluation infrastructure.

  • Understanding benchmark design, task realism, scoring reliability and failure analysis.

  • Researching model limitations or evaluating agent performance across complex tasks.

  • Publishing technical research, papers or detailed engineering articles on benchmarks, evaluations or related ML topics.

You don't necessarily need years of experience or an extensive publication record.

What's more important is your ability to investigate unfamiliar technical problems, understand how real workflows operate and build robust systems that provide meaningful results.

Why consider it?

You'll join a team of approximately 25 people, including researchers with publications at ICLR and NeurIPS, international Olympiad medallists and experienced AI startup founders.

The company is growing, with an opportunity to influence how its benchmarks are designed, evaluated and used by frontier AI teams.

You'll have genuine ownership of technically demanding work, with the freedom to investigate problems rather than simply implement predefined solutions.

Compensation and location

Base salary:

  • San Francisco / major US cities: $135,000–$230,000

  • Singapore: $100,000–$175,000

  • Rest of world: $100,000–$175,000

Location: San Francisco or Singapore, with remote options available for candidates who can overlap 70–80% with either office's working hours.

Visa sponsorship and relocation support are available for suitable candidates.

Benefits include fully covered medical, dental and vision insurance for US and Singapore employees, access to AI development tools, office meals and an annual travel budget.

The interview process involves two technical interviews and a 2–3 day work trial.

If you've been working on agent evaluations or benchmarking and want to help define how frontier AI capabilities are measured, we'd be happy to discuss the role.

Contact Techire AI for a confidential conversation.

All applicants will receive a response.

Salary

USD 100,000 - 230,000

Location

San Francisco, California, United States · Remote

Contract

Full-time

Your consultant

Kieran Newton

Your consultant

Kieran Newton

techire AI website footer image