Databricks Architect/Admin

Position Title: Databricks Architect / Admin
Employment Type:
Location: Hartford, CT (Hybrid – 3 Days Onsite / 2 Days Remote)
Schedule: Monday–Friday, 8:00 AM – 5:00 PM
Duration: 6-month Contract-to-Hire
Team Size: 10–12 members (Onshore + Offshore)
Work Environment: Agile, daily stand-up at 9:00 AM, ServiceNow workflow management
Interview Process: 1–2 rounds (possible third round), panel with hiring manager and team members; technical + behavioral focus, no formal assessment.
Travel: Minimal (commute only)

A senior Databricks architect with deep platform administration experience who can function as a strategic advisor rather than a day-to-day operator. The ideal candidate combines strong Databricks architecture knowledge with AWS infrastructure expertise, cost optimization skills, Linux administration, and enterprise-scale data platform experience.

Position Overview

The Databricks Architect/Admin is a senior individual contributor responsible for architecting, governing, optimizing, and supporting the enterprise Databricks platform. This role is primarily consultative with selective hands-on responsibilities and serves as the subject matter expert for platform architecture, administration, governance, cost optimization, and enterprise integrations.

The environment is already established; the selected candidate will provide strategic guidance, optimization recommendations, and ongoing support rather than building from scratch. This position is a backfill for a retiring resource and requires deep expertise to guide engineering teams and platform direction.

The role requires a strong combination of:

• Databricks architecture expertise
• Advanced administration knowledge
• Cloud platform experience (AWS)
• Unix/Linux administration
• Cost management and optimization skills
• Data engineering and integration experience
• Strategic advisory capability

---

Key Responsibilities

Platform Architecture & Design

• Architect and govern enterprise Databricks environments including:
  • Workspace topology
  • Unity Catalog structures
  • Access control frameworks
• Define and enforce:
  • Cluster standards
  • Runtime versions
  • Instance pool strategies
  • Auto-scaling policies
• Design scalable and performant pipelines using:
  • Delta Live Tables (DLT)
  • Databricks Workflows
  • Structured Streaming
• Define architectural standards for:
  • Delta Lake formats
  • Partitioning
  • Z-ordering
  • OPTIMIZE/VACUUM strategies
• Lead integration architecture with upstream platforms:
  • Fivetran
  • Ab Initio

---

Databricks Consulting & Advisory

• Serve as the primary Databricks consultant across teams
• Provide guidance on platform best practices
• Support technical decision-making
• Act as the escalation point for Databricks platform issues
• Mentor teams on platform usage and optimization
• Recommend new Databricks capabilities and roadmap enhancements

---

Cost Management & Optimization

A major focus area during the first 3–6 months.

Responsibilities include:

• Own DBU consumption reporting and tracking
• Identify optimization opportunities across:
  • Interactive clusters
  • Jobs
  • SQL Warehouses
• Detect inefficiencies such as:
  • Excessive DBU consumption
  • Looping jobs
  • Misconfigured workloads
• Build cost attribution/chargeback models
• Support:
  • Capacity planning
  • Contract forecasting
  • Vendor discussions
• Collaborate on monthly cost analyses

---

Unix/Linux Infrastructure & Operations

• Administer and troubleshoot Linux environments supporting Databricks
• Manage:
  • Init scripts
  • Cluster lifecycle processes
  • Shell scripting
• Build Bash and Python automation solutions
• Support:
  • Monitoring
  • Log aggregation
  • Maintenance workflows
• Manage:
  • File systems
  • Permissions
  • Data movement operations
• Perform EC2/VM diagnostics with infrastructure teams

---

Governance, Security & Compliance

• Design governance frameworks in Unity Catalog
• Implement:
  • Data lineage
  • Tagging
  • Access auditing
• Partner with cybersecurity teams
• Support:
  • Secrets management
  • Network isolation
  • Encryption
• Maintain audit documentation and compliance artifacts

---

Automation, AI & Machine Learning

• Implement automation for:
  • Cluster lifecycle management
  • Scheduling
  • Alerting
  • Self-healing workflows
• Leverage:
  • AutoML
  • MLflow
  • Model Serving
• Integrate:
  • Databricks Assistant
  • GitHub Copilot
• Support scalable feature engineering and ML deployment pipelines
• Evaluate emerging:
  • AI capabilities
  • Generative AI tools
  • LLM-integrated workflows

---

Required Qualifications

• 7+ years in data engineering or platform engineering
• Minimum 4+ years of hands-on Databricks implementation experience
• Deep expertise in:
  • Unity Catalog
  • Delta Lake
  • Databricks Workflows
  • Delta Live Tables
  • SQL Warehouses
• Strong Unix/Linux skills:
  • Shell scripting
  • Process management
  • File systems
  • Cron
• Python and PySpark expertise
• AWS or cloud platform experience:
  • EC2
  • Storage
  • Networking
  • IAM/security
• SQL expertise for reporting and analysis
• Experience designing scalable and cost-efficient enterprise platforms
• Experience with automation frameworks
• Familiarity with AI/ML ecosystem
• Oracle database experience
• Git and CI/CD knowledge
• ServiceNow and Jira exposure
• Strong communication skills

---

Preferred Qualifications

• MLflow deployment and model registry experience
• Generative AI tools:
  • LangChain
  • LlamaIndex
  • RAG workflows
• Workflow orchestration:
  • Apache Airflow
  • Databricks Workflows
• Fivetran integration expertise
• Ab Initio administration or development
• Data governance platforms:
  • Collibra
  • Alation
• Experience in regulated industries:
  • Insurance
  • Financial services
• Terraform/Ansible experience
• Disaster recovery and resiliency planning

---

Core Technical Skills

Databricks Ecosystem

• Databricks
• Unity Catalog
• Delta Lake
• Delta Live Tables
• Databricks Workflows
• SQL Warehouses
• MLflow
• AutoML

Cloud & Infrastructure

• AWS
• EC2
• S3
• Networking
• Private Links
• Firewalls

Programming & Automation

• Python
• PySpark
• Bash/Shell scripting
• SQL
• Git

Integration Technologies

• Fivetran
• Ab Initio
• Oracle

Platforms & Operations

• ServiceNow
• Jira
• Linux/Unix

---

First 3–6 Month Priorities / Challenges

Primary focus areas:

1. Cost optimization and Databricks usage tracking
2. Stabilization of Databricks onboarding application (currently MVP)
3. Identification of billing discrepancies and inefficiencies
4. Strategic capacity planning and vendor support
5. Providing platform guidance and consultation to teams

---

Hiring Notes

• Hartford candidates strongly preferred
• Exceptional candidates from alternate locations may be considered
• Budget is below current market expectations
• Hiring manager noted limited flexibility but open to strong profiles

 
 
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position’s starting pay is: $ 45.00/daily.