Job Title: Data Engineer IV Location: Remote in the EST Time zone Duration: 25 Months
Position Overview
We are seeking seven highly skilled Data Engineer IV professionals to lead the design, development, testing, and optimization of enterprise data solutions within the Azure ecosystem. This role requires deep expertise in Databricks, Azure Data Factory, Spark, SQL, and Power BI, along with strong leadership capabilities and the ability to work independently in a fast-paced Agile environment.
The ideal candidate will be responsible for building and supporting scalable data pipelines that move data from source systems into Azure Data Lake environments for reporting, analytics, and business intelligence. Candidates must have extensive experience working with Databricks, Unity Catalog, Delta Tables, and Power BI reporting solutions while ensuring data quality through effective unit and system testing practices.
All candidates must demonstrate strong communication skills, technical leadership experience, and the ability to guide engineering efforts across multiple workstreams.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks and Azure Data Factory.
Develop Spark-based data transformation and processing solutions for large-scale data environments.
Move and transform data from various source systems into Azure Data Lake environments.
Utilize Databricks Unity Catalog and Delta Tables to manage and optimize enterprise data assets.
Develop and support Power BI datasets, semantic models, reports, dashboards, and visualizations.
Partner with business stakeholders to understand reporting and analytics requirements.
Perform unit testing, integration testing, and system testing to ensure complete and accurate data processing.
Validate that all required files and transactions are successfully captured and processed.
Troubleshoot and resolve data pipeline and reporting issues.
Participate in Agile ceremonies and contribute to sprint planning, backlog grooming, and delivery activities.
Lead technical discussions, mentor team members, and promote engineering best practices.
Required Qualifications/Must-Have Skills
Databricks: 5+ years of experience
Databricks Certification required
Strong experience with Databricks architecture, optimization, and development
Azure Data Factory: 5+ years of experience
SQL: 5+ years of strong hands-on development experience
Power BI Development: 3+ years of experience creating enterprise dashboards, reports, and data models
Strong Spark programming experience
Experience with Azure Data Lake environments
Experience working with Unity Catalog and Delta Tables
Strong understanding of data integration, transformation, and validation processes
Experience performing unit testing and system testing of data solutions
Strong verbal and written communication skills
Strong understanding of Agile development methodologies
Proven ability to function as a technical lead and mentor junior team members
Preferred Qualifications
Experience within large enterprise data environments
Azure cloud certifications
Experience with CI/CD and DevOps practices for data engineering
Knowledge of data governance, data quality, and security best practices
Leadership Expectations
Lead and influence technical solution design and implementation.
Provide guidance and mentorship to other engineers.
Drive best practices related to data engineering, testing, and reporting.
Communicate effectively with technical and business stakeholders.
Take ownership of deliverables and provide proactive issue resolution.
Top Required Skills
Databricks (5+ years) - Certification Required
Azure Data Factory (5+ years)
Strong SQL Development (5+ years)
Power BI Development
Spark Programming
Unity Catalog & Delta Tables
Azure Data Lake
Agile Methodology
Strong Communication Skills
Technical Leadership Experience
Applicant Notices & Disclaimers
For information on benefits, equal opportunity employment, and location-specific applicant notices, click here
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws.The pay rate for this position is $51.00/hr.
