Senior Data Engineer (Consultant, 100% Remote)

Key Responsibilities: Data Pipeline Design & Development -Design and build scalable batch and real-time data pipelines using Microsoft Fabric and Snowflake. -Develop data processing logic using Spark and Python. -Work extensively with Delta and Parquet data formats. -Build and integrate API-based data ingestion and consumption pipelines. -Develop real-time streaming solutions using Kafka, Azure Event Hub, or similar technologies. CI/CD, DevOps & Engineering Standards -Implement and maintain CI/CD pipelines to deploy data solutions across multiple environments. -Apply DevOps best practices for release management, environment promotion, and rollback strategies. -Follow coding standards, design principles, and version control practices using GitHub. -Ensure high code quality, reusability, and long-term maintainability. Operations, Monitoring & Resilience -Monitor data pipelines and jobs to ensure reliability and performance. -Proactively handle failures, implement retries, alerting, and recovery mechanisms. -Perform root cause analysis and continuously improve system resilience. -Maintain low-level design documentation, operational runbooks, and technical artefacts. Security, Compliance & Cost Optimisation -Implement access controls, encryption, data masking, and governance policies. -Ensure compliance with data protection regulations such as GDPR. -Design cost-optimised data pipelines and warehouse solutions. -Monitor and optimise compute and storage usage to improve efficiency.