Data Engineer III (Databricks)

Related keywords: entry level remote jobremote job entry leveldata entry remote job

This page contains product affiliate links.

Overview of the Position

LivePerson, a leading player in the realm of enterprise conversations, is seeking a Data Engineer III specializing in Databricks to join their innovative team. With their Conversational Cloud platform utilized by major brands such as HSBC and Virgin Media, LivePerson is at the forefront of revolutionizing customer interactions. This position will play a crucial role in enhancing data workflows by migrating existing data processing ecosystems from Hadoop to Databricks on GCP.

Key Responsibilities

Assessment and Planning

The Data Engineer will undertake a comprehensive review of the current Hadoop infrastructure to understand the existing systems better. This includes analyzing Spark and MapReduce jobs and evaluating the compatibility of Java and Scala codebases with Databricks. A significant aspect of this role is to propose a detailed migration plan, including timelines and milestones.

Code Migration

The individual will be tasked with refactoring Spark jobs to optimize them for Databricks. In addition, they will work to migrate or rewrite MapReduce jobs using the Spark DataFrame/Dataset API. The migration process will also involve updating Java and Scala codes to align with the Databricks' runtime environment.

Testing and Validation

Post migration, the Data Engineer will need to develop both unit and integration tests to ensure that the new systems perform at least at the same level as the current setup. Performance metrics comparison before and after the migration will be crucial, alongside implementing robust error handling and logging mechanisms in Databricks.

Optimization and Performance Tuning

An essential aspect of the role is to fine-tune Spark configurations for better performance on Databricks. The Engineer will also optimize the data ingestion and transformation processes, ensuring workflows are efficient and scalable.

Deployment and Documentation

Finally, once migrated jobs are ready, the Engineer will deploy them to production in Databricks. Thorough documentation of all changes, configurations, and processes is critical, as is providing knowledge transfer to internal teams when necessary.

Required Skills and Qualifications

The ideal candidate for this position will possess:

• 6+ years of experience in Data Engineering, specifically with a focus on creating data pipelines, data platforms, and ETL processes, particularly on Hadoop and Databricks.
• Strong expertise in Databricks, including knowledge of Spark and Delta Lake, preferably on GCP.
• A robust understanding of the Hadoop ecosystem, with significant experience in Spark, MapReduce, and HDFS.
• Proficiency in Scala and Java, with a strong command of SQL.
• In-depth understanding of data engineering best practices, optimization techniques, and knowledge around data governance, data modeling, and enterprise-scale data lakehouse platforms.
• Familiarity with testing frameworks like Great Expectations is desirable.

Minimum Qualifications

Candidates should ideally hold a Bachelor's degree in Computer Science or a related field. A Certified Databricks Engineer will be preferred.

Employment Benefits

LivePerson provides competitive benefits which include:

• Medical, Dental, and Vision insurance.
• Vacation time and paid holidays.
• Access to internal professional development resources for continuous learning and growth.

Work Environment and Culture

LivePerson is recognized as a remote-friendly workplace, emphasizing an inclusive culture that fosters collaboration, innovation, and connection among team members. The company values diversity, encouraging unique contributions, and providing opportunities for professional growth. They have been recognized by various organizations for their innovative and beloved work environment.

Commitment to Equal Opportunity

LivePerson is committed to being an Equal Opportunity Employer, welcoming applicants from diverse backgrounds without regard to age, gender identity, race, sexual orientation, or other characteristics. This commitment extends to accommodations for applicants with disabilities, ensuring that everyone has equal access to opportunities.

Conclusion

This is a fantastic opportunity for an experienced Data Engineer to contribute to a cutting-edge company while enhancing their professional skills in a supported environment. LivePerson not only offers a competitive salary but also a culturally robust workplace that recognizes the importance of diversity and innovation. A significant emphasis is placed on professional development and the well-being of their team members, making it an ideal place for individuals looking to grow their careers in data engineering.

This job offer was originally published on himalayas.app