Overview: Join a dynamic team to architect and implement scalable enterprise data platform solutions utilizing Azure Databricks and related Azure services. This role is ideal for an experienced engineer with a strong background in data engineering and cloud technologies.
Key Responsibilities:
Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services
Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads
Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance
Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns
Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities
Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices
Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables
Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation
Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture
Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure
Required Qualifications:
At least 8-10 years of experience in designing and implementing enterprise data platforms
At least 5 years of hands-on Azure Databricks experience in enterprise environments
Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake
Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security
Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL)
Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained access controls
Experience designing and working with enterprise security architectures including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security
Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards
Strong understanding of data modeling techniques including dimensional modeling, Data Vault, medallion architecture, and enterprise data warehouse concepts
Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking
Strong SQL programming skills and proficiency in Python and PySpark development
Experience with enterprise data quality frameworks and operational dashboards
Strong soft skills and ability to mentor within a team structure
Applicant Notices & Disclaimers
For information on benefits, equal opportunity employment, and location-specific applicant notices, click here
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws.This position's pay range is $70.00/hr - $79.00/hr.
