Job Title: AI Quality Analyst
Location: Mississauga
Duration: 6 Months
Work Model: Onsite-Hybrid

Job Summary
The AI Quality Analyst is a critical role responsible for ensuring the performance, safety, and reliability of our cutting-edge AI/ML models. You will be at the forefront of our development lifecycle, designing and executing comprehensive evaluation strategies to identify model weaknesses, potential biases, and critical edge cases. This role requires a blend of analytical rigor, technical aptitude, and a deep curiosity for how AI models behave in real-world scenarios. You will not just find bugs, but provide the actionable insights that drive model improvement and guide our research and development efforts.

Responsibilities

• Evaluation Strategy & Benchmark Development: Design, develop, and maintain a comprehensive suite of test cases and evaluation benchmarks. Proactively identify potential model failure points, including edge cases, adversarial inputs, and sources of bias.
• Error Analysis & Failure Triage: Conduct systematic error analysis to categorize model failures and identify underlying patterns. Triage defects, prioritize them based on severity and impact, and work with the development team to ensure resolution.
• Data Sourcing & Curation: Source, curate, and manage high-quality datasets for model evaluation and testing. This includes performing data annotation and validation to ensure the integrity of our ground-truth data.
• Exploratory & Adversarial Testing (Red Teaming):Perform unscripted, exploratory testing to discover unexpected model behaviors. Participate in red teaming exercises to intentionally challenge our models and identify potential safety and security vulnerabilities.
• Test Environment Management: Set up, maintain, and troubleshoot testing and demonstration environments to ensure a stable and reliable evaluation pipeline.
• Reporting & Insights: Analyze and synthesize test results into clear, actionable reports for both technical and non-technical stakeholders. Translate complex findings into concrete recommendations for model improvement.

• Process Improvement: Actively participate in post-hoc evaluation reviews and contribute to the continuous improvement of our testing methodologies, tools, and overall quality assurance processes.

Qualifications and Skills

• Proven experience in a quality assurance, testing, or data analysis role, preferably within the AI/ML domain.
• A deep understanding of the machine learning lifecycle and the common failure modes of AI models.
• Hands-on experience with data annotation, data validation, and managing large datasets.
• Meticulous attention to detail and a methodical approach to problem-solving.
• Strong analytical skills with the ability to identify patterns in data and draw meaningful conclusions.
• Expertise with industry-standard test automation tools and libraries (e.g., Selenium, Playwright, Cypress, REST-assured).
• Experience in testing across different platforms (e.g. comprehensive testing of mobile Android/iOS and web applications)
• Experience with bug tracking systems (e.g., Jira) and test case management tools.
• Scripting skills (e.g., Python) for test automation and data manipulation.
• (Preferred) experience in testing AI systems, including evaluating agentic responses, model performance metrics, and data integrity.
• Excellent communication skills, with the ability to clearly document bugs and articulate complex technical issues.
• Familiarity with computer vision or other specific AI domains relevant to our work.

 
 

 
 
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws.This position's pay range is $48.00/hr - $51.00/hr.