APPLYDJINN
Vacancy catalog
SoftServe
Open role>14 days

AI Senior QC

SoftServeChile
Work model
Hybrid
Experience
5+ years
Employment
Full Time
Compensation
Not disclosed
Technology signal
4 tags

Technology context

4

Parsed from the vacancy text; ordered by relevance to this role.

AILLMMachine LearningRAG

Full listing

Role description

About the role

SoftServe is delivering the AI Student Tutor education company - a Canvas-integrated, context-aware AI tutor and learning coach. The product routes student conversations through a single-assistant orchestration layer to specialized AI capabilities (tutoring, content generation, safety/distress handling), and must meet strict pedagogical and safety data requirements ahead of a production pilot launch.

The Senior QC Engineer owns quality across this engagement, working within the SoftServe delivery team alongside AI/ML engineering, application engineering, product, and data engineering, and coordinating directly with client's Product, SME, IT, and Security/Compliance stakeholders through UAT and pilot-readiness activities.

Responsibilities

  • Own creation of the Acceptance Testing Plan (ATP), defining acceptance criteria, test types, roles and responsibilities, test data, evidence expectations, defect severity definitions, retest process, and formal sign-off requirements, in coordination with Covista.
  • Design and execute the overall test strategy spanning functional, integration, regression, load/performance, accessibility, safety, and pedagogical-quality testing.
  • Test the AI agent application itself, including single-assistant routing/orchestration logic, prompt governance and version-tracking, persona-adaptive behavior, content-validation and grounding/citation pipelines, and resistance to identity-override and prompt-injection attempts.
  • Run safety scenario testing across low, medium, high, and critical distress-handling cases, measuring detection accuracy and false-negative/false-positive rates.
  • Drive UAT: build test scripts, execute with client's Product, SMEs, IT, Security/Compliance, and pilot stakeholders; manage defect triage, disposition, and re-test.
  • Serve as the primary client-facing point of contact for quality matters: communicate testing status, defect trends, and pilot-readiness risk directly to client's Product, SMEs, and other UAT stakeholders.
  • Jointly verify and reproduce reported bugs with Covista during UAT and Hypercare, confirming severity, scope, and expected vs. actual behavior before logging.
  • Translate verified client-reported issues into clear, actionable defect write-ups (repro steps, environment, evidence, severity) and hand them off to the Europe-based delivery team, working within the daily US/Europe overlap window to ensure timely triage and unblock next-day resolution.

Requirements

  • Proven experience leading QA/QC for complex, multi-workstream software programs, including authoring test strategies and acceptance test plans from scratch.
  • Experience testing AI/LLM-based or conversational agent systems - prompt-driven behavior, RAG/grounding validation, model-output quality rubrics, and safety/guardrail testing.
  • Familiarity with LMS integration testing (Canvas/LTI, SSO, roster sync) is a plus.
  • Strong written documentation skills for test plans, traceability matrices, defect reports, and readiness packages.
  • Strong client-facing communication skills, comfortable running bug-verification calls/sessions directly with client stakeholders and writing defect descriptions clear enough for a remote team to act on without follow-up questions.
  • Availability during the US - Europe daily overlap window to support live client bug verification and same-day handoff to the Europe-based team.