Sr. Data Engineer – MS Fabric, Azure
Spectraforce
Chicago, Illinois
Remote
3 hours ago
Job Description
Position: Sr. Data Engineer – MS Fabric, Azure
Location: Remote
Duration: 7 Months
Responsibilities:
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position’s starting pay is: $ 60.00/hr.
Location: Remote
Duration: 7 Months
Responsibilities:
- Design and build end-to-end data platforms using Microsoft Fabric
- Lakehouse, Warehouse, OneLake, Dataflows Gen2
- Develop and optimize Spark workloads using PySpark and SparkSQL
- Develop MLOps pipelines for Advanced Analytics & AI
- Build scalable ETL/ELT pipelines using:
- Azure Data Factory (ADF)
- MS Fabric Data pipeline
- Dataflow gen 2
- SSIS (on-prem, Azure-SSIS IR, and migration scenarios)
- Implement data modeling patterns:
- Medallion (Bronze / Silver / Gold)
- Dimensional modeling (Star/Snowflake)
- Different data file management experience – Parquet, JSON, XML
- Integrate Microsoft Purview for:
- Data cataloging & classification
- Automated data lineage (ADF, Fabric, SQL, ADLS)
- Enforce data security and access controls:
- RBAC, column-level security, masking
- Fabric & Purview policy alignment
- Optimize performance, reliability, and cost across Fabric capacities
- Implement CI/CD and IaC for data pipelines and governance artifacts
- Partner with security, compliance, and BI teams to ensure trusted data delivery
- Microsoft Fabric (Must-Have)
- Fabric Data Engineering workloads
- Lakehouse & Warehouse
- OneLake architecture
- Fabric pipelines & notebooks
- Capacity planning and performance optimization
- Advanced PySpark (joins, windows, UDFs, optimization)
- Strong SparkSQL
- Strong MLOps & Feature Engg.
- Partitioning strategies, shuffle tuning, caching
- Large-scale data processing (TB+)
- Azure Data Platform
- Azure SQL Database / SQL Server
- Azure Data Factory (ADF)
- SSIS / Azure-SSIS Integration Runtime
- ADLS Gen2
- Data Lineage & Security (Microsoft Purview)
- Purview data catalog & scanning
- Automated lineage across ADF, Fabric, SQL, ADLS
- Business glossary management
- Integration with Azure RBAC & security policies
Applicant Notices & Disclaimers
- For information on benefits, equal opportunity employment, and location-specific applicant notices, click here
At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position’s starting pay is: $ 60.00/hr.