Platform Engineer IV
Spectraforce
Seatac, Washington
2 months ago
Job Description
Job Description:
Job Title: Senior Data Architect (Azure Databricks)
Location: 08+Months
Duration: Remote (but has to be PST or MST)
Role Summary
We are seeking a talented and experienced Senior Data Architect to join our Cloud Center of Excellence. The candidate will work with our DataOps team to design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment. The ideal candidate must have a strong background in data engineering and infrastructure, with specific experience in related Azure services, including Azure Databricks. The position requires expertise in both architecture and the ability to perform hands-on engineering tasks at a senior level. Relevant certifications such as Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position.
Key Duties
• Azure Databricks Infrastructure Deployment:
o Design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment using a Hub and Spoke Architecture. Experience with data mesh architecture using Azure Databricks, including creating decentralized data ownership, establishing domain-oriented data products, and ensuring data interoperability across various domains is also a plus.
o Configure and optimize Azure Databricks environments for performance, scalability, and security.
o Develop and maintain automated deployment scripts using Terraform for Databricks infrastructure.
o Implement best practices for Databricks cluster management, including autoscaling, cluster policies, and job scheduling.
o Troubleshoot and resolve issues related to Databricks clusters, including performance bottlenecks, connectivity problems, and configuration errors.
o Set up monitoring and alerting for Databricks clusters to ensure proactive identification and resolution of issues.
o Collaborate with data engineering teams to integrate Databricks with other Azure services (e.g., Azure Data Lake Storage).
o Set up cost management dashboards, alerts, and budgets to control Databricks costs and ensure efficient utilization.
o Train data teams on how to effectively use Databricks for their data processing and analytics needs.
• Infrastructure as Code (IaC):
o Develop and maintain IaC scripts using Terraform to automate the provisioning and management of cloud infrastructure.
o Implement infrastructure changes and updates through version-controlled IaC repositories.
o Ensure compliance with security and governance policies in IaC deployments.
o Collaborate with DevOps and cloud engineering teams to streamline infrastructure deployment processes.
o Conduct code reviews and ensure adherence to Terraform best practices.
o Troubleshoot and resolve issues related to Terraform deployments.
• Training & Documentation:
o Work with data teams to train them on incorporating Databricks best practices into their solutions & workflows.
o Create and maintain documentation for Databricks and DataOps best practices.
Required Job-Specific Skills & Education
• At least 5 years of proven experience in deploying and managing Azure Databricks clusters and workspaces.
• Expert-level proficiency in Azure Databricks is required for this position.
• Experience using Databricks asset bundles for efficient management and deployment of Databricks resources.
• Strong understanding of Databricks architecture, including cluster management, job scheduling, and workspace configuration.
• Experience with Databricks integration with Azure services (e.g., Azure Data Lake Storage & Data Lake Medallion Architecture).
• Extensive experience with IaC tools, particularly Terraform and ability to write and maintain Terraform scripts for cloud infrastructure deployment.
• Experience in troubleshooting and resolving Terraform deployment issues.
• Experience with CI/CD pipelines and automation tools such as Azure DevOps and GitHub.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position.
• Candidates must work Pacific Standard Time hours. East Coast & Central Time Zone candidates will not be considered.
Preferred Experience:
• Developed, deployed, and optimized AI and machine learning solutions on the Databricks platform, utilizing frameworks such as Spark MLlib and TensorFlow to enhance data processing efficiency and predictive analytics performance.
• Certifications including Databricks Certified Generative AI Engineer Associate and/or Microsoft Azure AI Engineer Associate.
About Us: Established in 2004, SPECTRAFORCE® is one of the largest and fastest-growing diversity-owned staffing firms in the US. The growth of our company is a direct result of our global client service delivery model that is powered by our state-of-the-art A.I. proprietary talent acquisition platform, robust ISO 9001:2015/ISO 27001 certified processes, and strong and passionate client engaged teams. We have built our business by providing talent and project-based solutions, including Contingent, Permanent, and Statement of Work (SOW) services to over 140 clients in the US, Canada, Puerto Rico, Costa Rica, and India. Key industries that we service include Technology, Financial Services, Life Sciences, Healthcare, Telecom, Retail, Utilities and Transportation. SPECTRAFORCE is built on a concept of “human connection,” defined by our branding attitude of NEWJOBPHORIA®, which is the excitement of bringing joy and freedom to the work lifestyle so our people and clients can reach their highest potential. Learn more at: http://www.spectraforce.com
Benefits: SPECTRAFORCE offers ACA compliant health benefits as well as dental, vision, accident, critical illness, voluntary life, and hospital indemnity insurances to eligible employees. Additional benefits offered to eligible employees include commuter benefits, 401K plan with matching, and a referral bonus program. SPECTRAFORCE provides unpaid leave as well as paid sick leave when required by law.
Equal Opportunity Employer: SPECTRAFORCE is an equal opportunity employer and does not discriminate against any employee or applicant for employment because of race, religion, color, sex, national origin, age, sexual orientation, gender identity, genetic information, disability or veteran status, or any other category protected by applicable federal, state, or local laws. Please contact Human Resources at LOA@spectraforce.com if you require reasonable accommodation.
California Applicant Notice: SPECTRAFORCE is committed to complying with the California Privacy Rights Act (“CPRA”) effective January 1, 2023; and all data privacy laws in the jurisdictions in which it recruits and hires employees. A Notice to California Job Applicants Regarding the Collection of Personal Information can be located on our website. Applicants with disabilities may access this notice in an alternative format by contacting NAHR@spectraforce.com.
LA County, CA Applicant Notice: If you are selected for this position with SPECTRAFORCE, your offer is contingent upon the satisfactory completion of several requirements, including but not limited to, a criminal background check. We consider qualified applicants with arrest or conviction records for employment in accordance with all local ordinances and state laws, including the Los Angeles County Fair Chance Ordinance for Employers (FCO) and the California Fair Chance Act (FCA). The background check assessment will consider whether a criminal history could reasonably have a direct, adverse impact on the job-related safety, security, trust, regulatory compliance, or suitability for this role. Such findings may result in withdrawal of a conditional job offer.
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position’s starting pay is: $80.00/hr.
Job Title: Senior Data Architect (Azure Databricks)
Location: 08+Months
Duration: Remote (but has to be PST or MST)
Role Summary
We are seeking a talented and experienced Senior Data Architect to join our Cloud Center of Excellence. The candidate will work with our DataOps team to design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment. The ideal candidate must have a strong background in data engineering and infrastructure, with specific experience in related Azure services, including Azure Databricks. The position requires expertise in both architecture and the ability to perform hands-on engineering tasks at a senior level. Relevant certifications such as Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position.
Key Duties
• Azure Databricks Infrastructure Deployment:
o Design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment using a Hub and Spoke Architecture. Experience with data mesh architecture using Azure Databricks, including creating decentralized data ownership, establishing domain-oriented data products, and ensuring data interoperability across various domains is also a plus.
o Configure and optimize Azure Databricks environments for performance, scalability, and security.
o Develop and maintain automated deployment scripts using Terraform for Databricks infrastructure.
o Implement best practices for Databricks cluster management, including autoscaling, cluster policies, and job scheduling.
o Troubleshoot and resolve issues related to Databricks clusters, including performance bottlenecks, connectivity problems, and configuration errors.
o Set up monitoring and alerting for Databricks clusters to ensure proactive identification and resolution of issues.
o Collaborate with data engineering teams to integrate Databricks with other Azure services (e.g., Azure Data Lake Storage).
o Set up cost management dashboards, alerts, and budgets to control Databricks costs and ensure efficient utilization.
o Train data teams on how to effectively use Databricks for their data processing and analytics needs.
• Infrastructure as Code (IaC):
o Develop and maintain IaC scripts using Terraform to automate the provisioning and management of cloud infrastructure.
o Implement infrastructure changes and updates through version-controlled IaC repositories.
o Ensure compliance with security and governance policies in IaC deployments.
o Collaborate with DevOps and cloud engineering teams to streamline infrastructure deployment processes.
o Conduct code reviews and ensure adherence to Terraform best practices.
o Troubleshoot and resolve issues related to Terraform deployments.
• Training & Documentation:
o Work with data teams to train them on incorporating Databricks best practices into their solutions & workflows.
o Create and maintain documentation for Databricks and DataOps best practices.
Required Job-Specific Skills & Education
• At least 5 years of proven experience in deploying and managing Azure Databricks clusters and workspaces.
• Expert-level proficiency in Azure Databricks is required for this position.
• Experience using Databricks asset bundles for efficient management and deployment of Databricks resources.
• Strong understanding of Databricks architecture, including cluster management, job scheduling, and workspace configuration.
• Experience with Databricks integration with Azure services (e.g., Azure Data Lake Storage & Data Lake Medallion Architecture).
• Extensive experience with IaC tools, particularly Terraform and ability to write and maintain Terraform scripts for cloud infrastructure deployment.
• Experience in troubleshooting and resolving Terraform deployment issues.
• Experience with CI/CD pipelines and automation tools such as Azure DevOps and GitHub.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position.
• Candidates must work Pacific Standard Time hours. East Coast & Central Time Zone candidates will not be considered.
Preferred Experience:
• Developed, deployed, and optimized AI and machine learning solutions on the Databricks platform, utilizing frameworks such as Spark MLlib and TensorFlow to enhance data processing efficiency and predictive analytics performance.
• Certifications including Databricks Certified Generative AI Engineer Associate and/or Microsoft Azure AI Engineer Associate.
About Us: Established in 2004, SPECTRAFORCE® is one of the largest and fastest-growing diversity-owned staffing firms in the US. The growth of our company is a direct result of our global client service delivery model that is powered by our state-of-the-art A.I. proprietary talent acquisition platform, robust ISO 9001:2015/ISO 27001 certified processes, and strong and passionate client engaged teams. We have built our business by providing talent and project-based solutions, including Contingent, Permanent, and Statement of Work (SOW) services to over 140 clients in the US, Canada, Puerto Rico, Costa Rica, and India. Key industries that we service include Technology, Financial Services, Life Sciences, Healthcare, Telecom, Retail, Utilities and Transportation. SPECTRAFORCE is built on a concept of “human connection,” defined by our branding attitude of NEWJOBPHORIA®, which is the excitement of bringing joy and freedom to the work lifestyle so our people and clients can reach their highest potential. Learn more at: http://www.spectraforce.com
Benefits: SPECTRAFORCE offers ACA compliant health benefits as well as dental, vision, accident, critical illness, voluntary life, and hospital indemnity insurances to eligible employees. Additional benefits offered to eligible employees include commuter benefits, 401K plan with matching, and a referral bonus program. SPECTRAFORCE provides unpaid leave as well as paid sick leave when required by law.
Equal Opportunity Employer: SPECTRAFORCE is an equal opportunity employer and does not discriminate against any employee or applicant for employment because of race, religion, color, sex, national origin, age, sexual orientation, gender identity, genetic information, disability or veteran status, or any other category protected by applicable federal, state, or local laws. Please contact Human Resources at LOA@spectraforce.com if you require reasonable accommodation.
California Applicant Notice: SPECTRAFORCE is committed to complying with the California Privacy Rights Act (“CPRA”) effective January 1, 2023; and all data privacy laws in the jurisdictions in which it recruits and hires employees. A Notice to California Job Applicants Regarding the Collection of Personal Information can be located on our website. Applicants with disabilities may access this notice in an alternative format by contacting NAHR@spectraforce.com.
LA County, CA Applicant Notice: If you are selected for this position with SPECTRAFORCE, your offer is contingent upon the satisfactory completion of several requirements, including but not limited to, a criminal background check. We consider qualified applicants with arrest or conviction records for employment in accordance with all local ordinances and state laws, including the Los Angeles County Fair Chance Ordinance for Employers (FCO) and the California Fair Chance Act (FCA). The background check assessment will consider whether a criminal history could reasonably have a direct, adverse impact on the job-related safety, security, trust, regulatory compliance, or suitability for this role. Such findings may result in withdrawal of a conditional job offer.
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position’s starting pay is: $80.00/hr.