Databricks Developer (Java & Apache Spark) with IRS MBI Clearance required

Related keywords: remote job java developergovernment remote jobdevops remote job

Job Overview

The job opening is for a Databricks Developer specializing in Java and Apache Spark. The position is fully remote, with specific requirements that candidates must meet to be considered. IRS MBI Clearance is mandatory for all applicants. The role focuses on designing and building data processing solutions that can handle large-scale data in a secure environment.

Employer Information

The employer for this position is 3M Consultancy. This organization is likely known for its specialization in data engineering and technology solutions, particularly in government and enterprise spaces.

Contract Type

This role is classified as a full-time position, which typically means a standard work schedule with associated benefits.

Job Responsibilities

As a Databricks Developer, the individual will be expected to tackle several vital tasks:

• Build and maintain data pipelines using Apache Spark on Databricks.
• Write clean and efficient Java code (specifically Java 8 or later) adhering to coding best practices.
• Work with large structured and semi-structured data sets and ensure they are tuned for performance, stability, and cost-effectiveness.
• Collaborate with other teams to understand their data requirements and deliver appropriate solutions.
• Ensure that all data handling conforms to security and governance standards.
• Investigate and resolve any issues that arise in production environments.

Required Skills

The position requires a robust skill set and experience in various technical areas:

Active Federal Background Clearance: Candidates are expected to possess active federal clearance, which is essential for roles handling sensitive data.

Educational Background: A bachelor's degree in Computer Science or a related field is essential.

Experience: 8+ years of hands-on experience in data engineering or big data development is required. This suggests that candidates should not only have theoretical knowledge but practical experience working on large data systems.

Technical Skills:

• Java Development: Proficiency with Java 8 or newer, including experience with Streams, Lambdas, and object-oriented design principles.
• Apache Spark: Candidates should have hands-on experience with Spark Core, Spark SQL, and DataFrames along with a strong understanding of RDDs. Proficiency in performance tuning Spark jobs is also critical.
• Big Data Tools: Familiarity with tools like HDFS, Hive, or HBase is essential, as well as knowledge of streaming tools like Kafka. Experience with cloud-based data lakes and data formats like Parquet or ORC is advantageous.
• ETL & Data Processing: Strong experience in building both batch and real-time ETL pipelines is needed. Candidates should possess skills in data cleansing, transformation, and enrichment.
• DevOps & Deployment: Familiarity with running Spark on YARN, Kubernetes, or cloud platforms and knowledge of CI/CD tools (e.g., Jenkins, GitHub Actions) is expected.
• Testing & Code Quality: Proficiency in unit testing using tools such as JUnit or TestNG, and experience with mocking frameworks like Mockito.

Desired Traits

Candidates for this role are expected to thrive in Agile teams and maintain clear documentation of their work. The ability to remain calm and composed while troubleshooting production issues is deemed a valuable trait.

Nice to Have Skills

While not mandatory, the following skills can provide an edge in the application process:

• Experience with Scala or Python in Spark
• Familiarity with Databricks or Google Dataproc
• Understanding of Delta Lake or Apache Iceberg
• Prior background in data modeling and performance design.

Conclusion

In summary, the Databricks Developer position offers an exciting opportunity for seasoned professionals to engage with large-scale data processing in a secure remote environment. Candidates should carefully evaluate their qualifications against the outlined requirements and responsibilities to determine their fit for this role. Individuals meeting these criteria are encouraged to apply, particularly if they possess the essential IRS MBI clearance.

This job offer was originally published on himalayas.app