MediaRadar, Inc. is an industry leader in marketing intelligence, empowering critical marketing and sales decisions. The Data Engineering Lead will be responsible for designing scalable systems, integrating complex Machine Learning models, and leading a team towards agile practices while maintaining high coding involvement.

Responsibilities

Architect and implement complex, end-to-end data pipelines using Azure Databricks and PySpark

Design, build, and maintain a scalable data architecture using the Medallion Architecture (Bronze/Silver/Gold layers)

Optimize Apache Spark jobs, tune Databricks units, and define cluster policies to minimize compute costs

Proactively audit and refactor pipelines every 3-6 months to maintain effectiveness and reduce cloud costs

Implement caching strategies (e.g., broadcast joins) and manage performance impact

Develop a proactive monitoring and alerts framework to ensure 99.9% reliability and mitigate system issues before they impact end-users

Build an end-to-end Data Validation Framework (e.g., Great Expectations) to enforce data accuracy and consistency

Minimize job failure rates and ensure data is available in the Gold layer within the required 24-hour turnaround time

Architect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing complex analytical queries

Lead a lean team toward cross-trained agility, moving away from 'siloed specialists'

Manage sprint cycles, conduct code reviews, and guide the team on best engineering practices (including CI/CD)

Anticipate future data needs and design High-Velocity Architecture that is highly scalable and manageable to handle sudden volume increases (e.g., double the data from new sources like paid social/CTV)

Collaborate with ML teams to integrate automated model orchestration into robust ETL pipelines

Collaborate with the offshore team lead to facilitate seamless knowledge transfer and operational continuity across time zones

Establish clear communication protocols, standardized documentation, and robust feedback loops to ensure alignment on project goals

Act as the primary bridge between teams to mitigate bottlenecks and maintain high-quality delivery standards

Qualification

PythonPostgreSQLPgvectorAzure DatabricksPySparkDelta LakeDockerGitAzure DevOpsCI/CDApache AirflowAzure Kubernetes ServiceCross-training AdvocacyAgile Team Management

Required

10+ years of experience in Data or Software Engineering with deep codebase involvement

3+ years as a Technical Lead managing agile teams

Proven ability to lead lean, high-impact teams while maintaining high individual output

Experience with cross-training advocacy and scaling data processing through automation

Core: Python, PostgreSQL + pgvector

Big Data: Azure Databricks, PySpark, Delta Lake

DevOps: Docker, Git, Azure DevOps, CI/CD

Preferred

Workflow Orchestration: Experience with Apache Airflow

Containerization: Familiarity with Azure Kubernetes Service (AKS)