Design & Deployment: Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints.
Configuration Management: Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations.
Performance Tuning: Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations.
Capacity Planning: Forecast and plan for future Databricks resource needs based on platform usage trends.
Cost Optimization: Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances).
Troubleshooting: Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems.
2. Data Governance & Unity Catalog:
Unity Catalog Design: Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations.
Fine-Grained Access Control (FGAC): Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required.
Data Lineage & Audit: Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access.
Managed Tables & Volumes: Advise and govern the usage of managed and external tables/volumes within Unity Catalog.
Azure Entra ID Integration: Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization.
SCIM (System for Cross-domain Identity Management): Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks.
Azure Key Vault Integration: Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations.
Access Control Lists (ACLs): Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments.
Compliance & Auditing: Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity.
4. Networking & Connectivity:
Secure Network Design: Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs).
Firewall Rules: Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources.
DNS & Routing: Ensure proper DNS resolution and routing for Databricks components and integrated services.
5. Data Integration & Orchestration (Azure Data Factory - ADF):
ADF Integration Patterns: Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines.
Automated Deployments: Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps.
API Management: Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed.
6. Azure Storage (ADLS Gen2) Integration:
Storage Account Design: Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies.
Mount Points & Credential Passthrough: Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals.
7. Operations, Monitoring & Reliability:
Monitoring & Alerting: Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization).
Disaster Recovery & Business Continuity: Contribute to and validate disaster recovery and business continuity plans for the Databricks platform.
Patching & Upgrades: Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities.
Service Health: Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks.
8. Documentation & Best Practices:
Architectural Documentation: Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform.
Best Practices: Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams.
Knowledge Transfer: Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage.
9. Collaboration & Stakeholder Management:
Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues.
Act as a subject matter expert for all Databricks-related queries from internal teams.
Participate in architectural review boards and provide technical leadership.
Key Skills & Qualifications:
Deep Expertise in Azure Services: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure Networking (VNets, NSGs, Private Endpoints), Azure Entra ID (formerly Azure AD), Azure Monitor, Log Analytics.
Security: Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security. Experience with SCIM.
Networking: Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure.
Scripting/Automation: Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks.
CI/CD: Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions).
Problem-Solving: Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues.
Communication: Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences.
Certifications (Preferred): Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Professional/Associate.
Applicant Notices & Disclaimers
For information on benefits, equal opportunity employment, and location-specific applicant notices, click here
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws.This position's pay range is $55.00/hr - $60.71/hr.
