Thinkgrid Labs: Fabric Data Engineer (Remote)

Related keywords: remote job microsoftdata engineer remote jobremote job us

Overview

Thinkgrid Labs is at the forefront of innovation and technology, specializing in creating bespoke web, mobile, cloud applications, data platforms, and AI solutions. The company is expanding its data practice by standing up Microsoft Fabric for a health insurer in the US. They are looking for a Fabric Data Engineer who will play a crucial role in fast and reliable Bronze-layer ingestion of data.

Job Title and Location

The position is for a Fabric Data Engineer who will work remotely. The working hours for this role are from 2 PM IST to 11 PM IST, providing flexibility for those who wish to work from various parts of the world.

Required Experience and Education

Candidates should have between 6 to 10 years of experience in data engineering. Additionally, at least 2 years of hands-on experience with Azure data platforms, specifically Microsoft Fabric (including Mirroring, Lakehouse, and Warehouse) is mandatory. A Bachelor's or Master's degree in Computer Science, Health Informatics, or Business is expected. Preferred certifications include DP-600, DP-203, and DP-500.

Essential Skills

To be successful in this role, candidates need to possess several key skills:

• Strong SQL Server/T-SQL knowledge, with hands-on experience in CDC (Change Data Capture) or replication patterns.
• Proficiency in setting up and tuning Fabric Mirroring to land data into OneLake/Lakehouse.
• Ability to detect and communicate schema changes safely, which includes understanding contracts, tests, alerts — aimed at minimizing downstream breakage.
• A mindset focused on high-volume ingestion, designing for throughput, resiliency, and backpressure retries, which includes idempotency and partitioning strategies.
• Experience in programming languages such as Python, Scala, and Spark for building ingestion frameworks and conducting data quality checks.
• A commitment to operational excellence, adding observability (metrics, logging, alerts), and documenting processes.
• Understanding of data security principles, especially regarding PII (Personally Identifiable Information) and PHI (Protected Health Information).

Job Responsibilities

As a Fabric Data Engineer, the individual will have diverse responsibilities, including:

• Configure Fabric Mirroring from SQL Server and other relational sources into OneLake, tuning schedules, snapshots, retention, and throughput.
• Define structured organization of Lakehouse folder structures, establishing naming and tagging conventions for efficient Bronze ingestion.
• Implement CDC for handling both soft and hard deletes, along with managing late-arriving data and backfills with reliable watermarking.
• Create resilient data ingestion pipelines using Fabric Data Factory and notebooks, incorporating retries and dead-lettering for fault tolerance.
• Automate detection and management of schema drift while ensuring that all stakeholders receive pertinent change notes and guardrails.
• Optimize performance and costs by fine-tuning batch sizes, partitions, parallelism, and capacity usage to balance speed, reliability, and budget.
• Instrument observability features, ensuring that data quality tests are in place, along with setting simple SLOs (Service Level Objectives) for ingestion health.

Collaboration and Documentation

The role requires active collaboration with various teams, including the Fabric Platform Architect, on domains such as security and workspaces. Candidates must be skilled in documenting pipelines, SLAs (Service Level Agreements), and creating runbooks to streamline operations and knowledge transfer.

Benefits

The position offers several perks, including:

• A 5-day work week contributing to work-life balance.
• 100% remote setup allowing for a flexible work culture and international exposure.
• An opportunity to work on mission-critical healthcare projects that significantly impact providers and patients globally.

Conclusion

Overall, the Fabric Data Engineer position at Thinkgrid Labs is an excellent opportunity for seasoned data engineers interested in advancing their careers while working on innovative projects with substantial global impact. The successful candidate will not only enhance their technical skill set but also contribute positively to the healthcare sector through their work.

This job offer was originally published on weworkremotely.com