Job Title – Lead Cloud Data Engineer – Data Mesh & AI
Location – San Francisco, CA (Onsite)
Duration – 6+ Months
 
Job Description:
Qualifications:
We are seeking an experienced Lead Data Engineer to join our team and drive the development of modern data platforms and AI-augmented solutions. This role is critical to supporting ongoing operations of Large Data Warehouse, architecting the Large Data Hub , and preparing for the enterprise data mesh modernization program.
 
Responsibilities:
Data Engineering & Pipeline Development:

• Develop and maintain end-to-end ETL/ELT pipelines for data ingestion from multiple sources (databases, APIs, streaming), transformation, quality validation, data modeling, and consumption layer optimization using AWS services (Glue, EMR, Lambda, Kinesis, Step Functions)
• Implement data governance policies, metadata management, data lineage tracking, fine-grained access controls, and automated data quality frameworks with validation rules, anomaly detection, and monitoring/alerting mechanisms

 
Data Mesh Architecture:

• Design and implement Data Mesh architecture on AWS with federated governance, defining domain boundaries, data product specifications, and self-serve infrastructure patterns across Databricks, Starburst, Collibra, and Immuta platforms
• Develop reusable frameworks, accelerators, and self-serve tools enabling domain teams to independently publish, discover, and consume data products while ensuring performance optimization and cost efficiency

 
DevSecOps & Infrastructure:

• Integrate DevSecOps practices including CI/CD pipelines, infrastructure as code (Terraform/CloudFormation), automated security scanning, compliance validation, and disaster recovery strategies

 
AI-Augmented Development:

• Design and build agentic AI solutions, RAG pipelines, and intelligent automation using LLMs, orchestration frameworks, and AI-powered development tools (Copilot, Claude Code) to enhance platform capabilities and accelerate SDLC

 
Technical Leadership & Collaboration:

• Provide L3 technical support and mentor engineering teams on data mesh principles and modern data technologies
• Collaborate with stakeholders to translate business requirements into technical solutions
• Lead architecture design reviews and create comprehensive technical documentation

 
Required Qualifications:
Data Mesh Architecture & AWS (5+ years required):

• Deep experience designing distributed data architectures on AWS (S3, Glue, Lake Formation, EMR, Redshift)
• Hands-on implementation of data mesh principles including domain-oriented data ownership, data as a product, and federated computational governance

 
Modern Data Platform Technologies:

• Proficiency: Databricks (Unity Catalog, Delta Lake), Starburst/Trino (federated query), Collibra (data governance), or Immuta (dynamic data access control)
• Ability to architect multi-technology solutions

 
DevSecOps & Infrastructure as Code:

• Strong experience with CI/CD pipelines (GitLab/GitHub Actions, Jenkins)
• Containerization expertise (Docker/ECS)
• Infrastructure automation (Terraform, CloudFormation)
• Security compliance frameworks for AWS GovCloud environments

 
AI-Augmented Development & Engineering:

• Demonstrated use of generative AI tools for accelerating SDLC activities including code generation, testing, and documentation
• Experience designing or implementing agentic AI solutions using Amazon Bedrock or similar frameworks

 
Leadership & Soft Skills:

• Proven ability to translate business requirements into technical solutions
• Experience leading technical design sessions and mentoring engineering teams
• Strong communication skills to convey complex architectural concepts to both technical and non-technical stakeholders
• Experience working across hybrid cloud/on-premises environments

 
 
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's pay range is $70.00/hr - $90.00/hr.