About the Role:
We're seeking a sharp, systems-thinking Risk Analyst with a Data Annotation focus to join our Product Management team. You'll help evaluate and scale labeled data workflows for the next generation of risk and compliance assessment frameworks.
You'll work closely with product managers, engineers, and risk domain experts to design scalable assessment frameworks, map regulatory obligations to annotation workflows, and embed high-quality risk logic into how our platform handles training data and supervised learning. Ideal candidates bring experience in information security, audit, legal, compliance, and cloud risk, plus an emerging understanding of AI workflows and data governance.
This role supports our AI-powered risk products by providing high-quality annotations and evaluations of AI-generated outputs. You'll focus on tactical annotation work while applying risk and compliance domain expertise to ensure accuracy and consistency in our training datasets.

Key Responsibilities

• Perform detailed annotations of AI-generated outputs, evaluating accuracy, correctness, and compliance alignment
• Review datasets after new deployments and report on accuracy metrics
• Test updated control sets and validate annotation quality
• Report annotation conflicts, edge cases, or unclear guidelines to improve data quality standards
• Apply knowledge of IT compliance frameworks (SOC 1, SOC 2, ISO 27001) to annotation tasks
• Collaborate with product managers and AI teams to maintain annotation integrity and consistency

Required Qualifications

• 2–3 years of experience in IT audit, legal, risk management, compliance, or related fields (Big Four or consulting firm experience preferred)
• Knowledge of IT compliance standards including SOC 2, ISO 27001, Pen Tests etc.
• Understanding of IT infrastructure risks and control assessments
• Strong attention to detail with ability to follow complex instructions and deliver results effectively
• Good communication skills to collaborate with cross-functional teams and report findings clearly

Nice-to-Have Qualifications

• Exposure to IT audit, legal or internal audit processes
• Familiarity with data annotation or AI/ML workflows
• Understanding of GDPR, NIST, or other regulatory frameworks
• Background in data quality or governance practices

Why Join Certa?
At Certa, you'll play a critical role in building AI-powered risk intelligence products that transform how organizations assess and manage third-party risk. Join us in creating the future of compliance automation.