- Only applicants local to either the Branchville, NJ; Short Hills, NJ; or Charlotte, NC metro areas will be considered, due to the hybrid on-site requirements of this role.
- Due to the nature of this opportunity, visa sponsorship or application through a 3rd party/employer will not be considered (e.g H1-B visa).
Data Engineer (Azure / Databricks / Medallion Architecture)
Optomi, in partnership with a flagship insurance organization, is seeking a Data Engineer to join their centralized Data & Analytics team. This role is ideal for a hands-on data engineering professional experienced in building cloud-based data pipelines using Databricks, PySpark, and Medallion Architecture. The right candidate will bring deep Databricks expertise and a passion for designing scalable, modern data lakehouse solutions that support analytics, reporting, and machine learning initiatives.
What the right candidate will enjoy!
- The opportunity to build and optimize Azure + Databricks pipelines supporting enterprise-scale analytics and data products!
- A technically modern environment leveraging Delta Lake, Medallion Architecture, and real-time ingestion!
- A highly collaborative team culture with visibility across business units and the chance to influence future data strategy!
Experience of the right candidate:
- Bachelor’s Degree or equivalent hands-on experience in Computer Science, Data Engineering, or related field
- 4+ years of experience in data engineering, preferably within cloud environments.
- Deep experience with Databricks, including Autoloader, Delta Lake, and Medallion Architecture patterns.
- Strong proficiency in PySpark, SparkSQL, Python, and SQL for building scalable data processing pipelines.
- Direct experience with Azure services such as ADLS, ADF (orchestration), Event Hubs or Kafka, and Azure DevOps/GitHub for CI/CD.
- Solid understanding of data warehousing, dimensional modeling, and modern lakehouse concepts.
- Experience supporting data science teams, ML operationalization, or Azure ML is a plus.
- Nice to have: familiarity with Informatica, Netezza, or P&C insurance data.
- Excellent communication skills with the ability to collaborate across technical and business stakeholders.
Responsibilities of the right candidate:
- Design, build, and maintain Databricks-based ingestion and transformation pipelines, leveraging PySpark and Delta Lake.
- Implement Medallion Architecture (Bronze/Silver/Gold) for scalable, governed data products.
- Develop and optimize real-time and batch pipelines using Autoloader, ADF, and Azure services.
- Collaborate with business units to gather requirements and deliver high-quality, reusable data assets.
- Support analytics, BI, and data science teams through reliable data delivery and documentation.
- Apply best practices for CI/CD, version control, and environment management using Azure DevOps or GitHub.
- Troubleshoot and optimize data pipelines for performance, quality, and reliability.
- Contribute to data governance, documentation, and standards that support the organization's transition to a modern data mesh/product model.
