About the job
Databricks Developer – Medicare
- Advantage (Payer)
- Work Model: Remote (U.S.-based candidates only)
- Level: Senior
- Domain: Healthcare Payer / Medicare Advantage
Overview
- Our client, a leading Medicare Advantage health plan, is seeking an experienced Databricks Developer to build and
- operate data solutions on the Databricks lakehouse platform, supporting claims, eligibility, and member/provider
- data used across Medicare Advantage reporting and compliance. This is a Databricks platform-focused role: the
- ideal candidate lives inside Databricks day to day — notebooks, Delta Live Tables, Workflows, Unity Catalog, and
- cluster/job tuning — rather than a general data engineer who happens to use Databricks as one of several tools.
- Candidates should be comfortable working independently in a remote setup and available to start promptly.
Key Responsibilities
- Build, deploy, and maintain data pipelines natively within Databricks, using notebooks, Delta Live Tables
- (DLT), and Databricks Workflows/Jobs for orchestration.
- Develop and optimize Spark (PySpark/Scala/SQL) logic for processing large-scale Medicare Advantage
- claims (EDI 837/835), eligibility, and provider data within the Databricks environment.
- Manage and evolve the Delta Lake medallion architecture (bronze/silver/gold) to ensure clean, reliable,
- well-governed data.
- Configure and maintain Unity Catalog for data governance, lineage, and access control across PHI/HIPAA-
- regulated datasets.
- Tune cluster configurations, job performance, and cost efficiency (autoscaling, cluster policies, Photon,
- job/task-level optimization) within Databricks.
- Use Databricks SQL and SQL warehouses to support ad hoc analysis, validation, and reconciliation of
- Medicare Advantage claims and eligibility data.
- Monitor pipeline health, troubleshoot job failures, and resolve performance issues directly within the
- Databricks platform (Jobs UI, cluster logs, DLT event logs).
- Manage code and deployments via Databricks Repos and CI/CD integration (Databricks Asset Bundles, Azure DevOps/Git).
- Apply PHI/HIPAA-compliant data handling and role-based access practices throughout all Databricks workspace and pipeline configurations.
- Document Databricks pipeline design, workspace structure, and job configurations.
Required Experience & Skills
- 5+ years of hands-on Databricks experience — this is a Databricks-first role; deep platform fluency is the primary requirement.
- Strong command of Spark (PySpark/Scala) and Databricks SQL for pipeline development.
- Hands-on experience building and managing Delta Live Tables (DLT) pipelines.
- Practical experience with Unity Catalog for data governance, access control, and lineage.
- Experience with Databricks Workflows/Jobs for orchestration and scheduling.
- Demonstrated ability to tune cluster and job performance (autoscaling, cluster policies, Photon, cost optimization) within Databricks.
- Experience with Databricks Repos and Git-based CI/CD for notebook/pipeline deployment.
- 3–10+ years of healthcare payer domain experience, ideally with Medicare Advantage data (claims ingestion, eligibility logic, EDI 837/835, CMS-aligned reporting).
- Working knowledge of PHI/HIPAA-regulated data handling and role-based security within a data platform.
- Comfortable working within an Azure or equivalent cloud environment (ADLS, workspace/networking basics) as the underlying infrastructure for Databricks.
- Strong problem-solving skills and ability to work independently in a remote environment.
- U.S.-based, remote-capable (U.S. resident, citizen, or Green Card holder).
Nice to Have
- Databricks Certified Data Engineer (Associate or Professional).
- Experience with MLflow or Databricks’ ML/AI tooling.
- Exposure to Azure Data Factory, Power BI, or Snowflake in a supporting/downstream capacity.
- Experience migrating legacy ETL workloads into Databricks.
- Prior experience in a client-facing or consulting environment.
- Experience with EHR-adjacent systems (e.g., Epic Clarity) or provider-side data in addition to payer data.
Ideal Candidate Profile
“Senior Databricks Developer who lives in the platform day to day — Delta Live Tables, Unity Catalog, Workflows,
and Spark performance tuning — with real Medicare Advantage claims and eligibility experience, and a track
record of running governed, production-grade lakehouse pipelines end to end within Databricks.
