AI Quality Analyst
<h3>This is a remote position.</h3><div><b><b>AI Quality Analyst</b></b><p>Reporting to Manager, Quality Engineering & AI Validation, focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. Builds benchmark scenarios, defines scoring rubrics, evaluates business usefulness, and identifies failure patterns that conventional pass or fail software testing would not catch.</p><h3><b>Key Responsibilities</b></h3><h3>AI Output Evaluation</h3><ul><li>Design and execute structured evaluations for AI-enabled features and workflows.</li> <li>Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk.</li> <li>Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations.</li> </ul><h3>Benchmark & Rubric Development</h3><ul><li>Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards.</li> <li>Develop rubrics that allow quality to be measured consistently across releases and changes.</li> </ul><h3>Workflow & Model Change Validation</h3><ul><li>Compare performance across prompt versions, workflow revisions, tools, and models.</li> <li>Support release decisions with evidence on quality regression or improvement.</li> </ul><h3>Business & Domain Partnership</h3><ul><li>Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts.</li> <li>Help define human-review thresholds and escalation patterns for higher-risk use cases.</li> </ul><h3>Production Feedback</h3><ul><li>Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time.</li> </ul><br><h3>Requirements</h3><h3><b>Required Qualifications</b><ul><li>4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles.</li> <li>Strong writing, analysis, and pattern-recognition skills.</li> <li>Experience evaluating outputs against nuanced criteria rather than only binary correctness.</li> <li>Ability to work with structured rubrics, scenario libraries, and evidence-based reviews.</li> <li>Comfort collaborating across Engineering and business teams.</li> <li>Experience with finance, accounting, FP&A, transaction services, or business process design preferred.</li> </ul></h3><h3>·Bachelor's degree preferred.</h3><h3>You Are<ul><li>Thoughtful, precise, and highly discerning.</li> <li>Strong at spotting subtle output problems others miss.</li> <li>Comfortable with ambiguity but disciplined in scoring and documentation.</li> <li>Focused on trust, usefulness, and business reality.</li> </ul><br></h3><h3>Benefits</h3><div>Salary plus performance-based bonus.<div>Actual compensation packages are determined by evaluating a wide array of factors unique to each candidate, including but not limited to skill set, years and depth of experience, education, certifications, cost of labor, and internal equity.<br></div></div></div><p>Originally posted on <a href="https://himalayas.app">Himalayas</a></p>
Apply with uptayn.
Sign in free to open the apply link, get this role scored against your CV, and track your application.