Sanctuary Computer is seeking a Senior Data Engineer to join their dynamic team, focusing on the AI and health and wellness space. This opportunity allows for remote work from anywhere globally, with a strong preference for candidates near New York City for potential collaboration. The company emphasizes a collaborative and inclusive culture, aiming at innovative projects including AI-driven solutions for a diverse client base, such as Google and MIT.
Key Responsibilities
The Senior Data Engineer will be responsible for several core activities, notably in designing and managing data pipelines, orchestrating workflows, and executing complex data integrations. Here’s a detailed breakdown of the main responsibilities:
Monitoring and maintaining data pipelines. This includes troubleshooting errors and addressing format drift to ensure the smooth operation of data flows.
Extracting and enriching data from diverse sources, actively using web crawlers and scrapers to gather insights.
Reprocessing large datasets within batch workflows, validating the integrity and accuracy of the data.
Designing and integrating new data sources into existing pipelines to enhance functionality and usability.
Aligning extracted data with the core data model to ensure consistency across applications.
Contributing to code reviews, delivering constructive feedback, and ensuring adherence to best practices in data engineering.
Participating in project management tasks by tracking team velocity, project scope, budget, and timelines, and negotiating project adjustments when needed.
Required Skills
For candidates considering this role, the following skills and competencies are critical:
Extensive experience with Python, emphasizing senior-level proficiency.
History of working with workflow orchestration tools like Prefect or Airflow.
Knowledge of ETL processes and data transformation essentials for Large Language Models (LLMs), such as OpenAI and Claude.
Competence in API interfacing, especially using libraries like Instructor or LiteLLM.
Practical experience with prompt engineering methodologies.
At least 5 years of backend experience using frameworks such as Ruby on Rails, Python Django, or Node Express.
Experience in developing native applications is a plus.
Nice to Have Skills
While the following skills are not essential, they will enhance a candidate's desirability:
Familiarity with Google Cloud Platform (GCP), particularly Cloud Run and Cloud Tasks.
Understanding of search technologies, including vector databases for semantic search.
Experience in client-facing roles, working closely with customers to define requirements and technical solutions.
Background in product management or engineering management, helping set technical directions and mentor team members.
Previous experience working in a startup environment is advantageous.
A diverse set of interests or hobbies outside of work, such as design, art, or music, is considered a positive attribute.
Compensation
The salary structure for this role is competitive:
Hourly Rate: Between $125 to $175.
Annual Salary: Ranges between $150,000 to $200,000 based on experience.
Additionally, for candidates demonstrating the right level of expertise and commitment, equity options may also be available, enhancing overall compensation potential.
Interview Process
Candidates interested in this position can expect a multi-step interview process that begins with an introductory call with team members. Subsequently, suitable candidates will participate in a technical exercise to demonstrate their skills. It is advisable to include a Loom recording in your application to showcase your skill set, which can substantially increase your chances of standing out.
Conclusion
The role of Senior Data Engineer at Sanctuary Computer is not just about technical skills; it's also about fitting into a vibrant, creative team that values collaboration and innovation. Candidates who seek to join a forward-thinking company at the intersection of AI and health & wellness and who possess a passion for data engineering are highly encouraged to apply.
This job offer summary has been generated using automated technology. While we strive for accuracy, it may not always fully capture the nuances and details of the original job posting. We recommend reviewing the complete job listing before making any decisions or applications.
