MagicSchool is a premier generative AI platform for teachers that has attracted over 6 million educators globally in just over two years since its launch. The company is on a mission to create a positive social impact through innovative educational technology. They are seeking a Senior LLM Quality Analyst to join their Trust, Safety, and Quality team to enhance the quality of their AI outputs for educational purposes.
Role Overview
As a Senior LLM Quality Analyst, the professional will focus on ensuring that the AI outputs meet high standards suitable for both educators and students. This role requires a blend of data analysis, prompt engineering, and practical teaching experience. The analyst will be instrumental in designing experiments, executing prompt strategies, and maintaining performance dashboards while ensuring the AI aligns with educational requirements.
Responsibilities
The job involves various responsibilities critical to maintaining and improving AI quality:
Design and Execute Prompt Engineering Experiments: Collaborate with the product manager, data scientist, and team lead to craft prompt strategies and conduct appropriate experiments.
Maintain Response Quality Reporting and Dashboards: Create reports and dashboards for performance metrics over time, utilizing tools like Metabase to provide stakeholder visibility into response quality metrics.
Evaluate and Improve AI Outputs: Design evaluators and test cases to assess AI outputs from an educator's perspective and work cross-functionally to address identified quality issues.
Stay Current on LLM Best Practices: Continual learning in prompt engineering and evaluations to drive the highest quality outcomes.
Required Skills and Qualifications
To be a successful candidate for this role, applicants are expected to possess
Strong analytical skills that enable proficiency in working with spreadsheets and data analysis tools.
Experience with Business Intelligence (BI) tools, with familiarity in Metabase being advantageous.
A sound understanding of basic statistics and SQL for data processing.
Excellent organizational skills with attention to detail and an ability to maintain proper documentation.
A critical thinking mindset to develop insightful experiments and draw conclusions from analytics.
A team-first attitude, promoting collaboration and effective communication within the team environment.
Candidates with the following skills will have an advantage:
Experience with analytical Python and tools like pandas or Jupyter notebooks.
At least 5 years of professional experience in data analysis roles, possibly including team lead experience.
Previous work or familiarity with LLMs or AI products, as well as any exposure to K-12 teaching or the EdTech industry.
A strong passion for utilizing technology to solve educational problems is highly valued.
Compensation and Benefits
The salary range for this position varies between $120,000 and $185,000, depending on experience and skill level. The company offers numerous benefits to ensure employees feel supported and well taken care of:
Unlimited time off to promote a work-life balance.
Flexible remote working options while cultivating a culture built on relationships, trust, communication, and collaboration.
A choice of employer-paid health insurance plans including dental and vision.
Generous stock options which vest over four years, along with a 401k match and a monthly wellness stipend.
Company Values
MagicSchool is dedicated to its core values, which are foundational to its operating culture:
Educators are Magic: They believe that educators are pivotal to the educational process.
Community: They emphasize building a community that supports each other.
Innovation: They are driven by a desire to modernize the educational system using AI technology.
Diversity: They strive to embrace diversity of thought and backgrounds.
Excellence: They commit to providing the highest quality service for educators and students.
By applying for this position, candidates will not only be contributing to a mission-driven team but also working on technologies that directly influence educational outcomes. Interested applicants are encouraged to submit their applications before November 2, 2025, to be considered within the priority deadline. Late submissions will also be reviewed as the hiring process continues beyond this date.
This job offer was originally published on himalayas.app
This job offer summary has been generated using automated technology. While we strive for accuracy, it may not always fully capture the nuances and details of the original job posting. We recommend reviewing the complete job listing before making any decisions or applications.
