Databricks PlatformLead

Job Title: Senior Data Platform Engineer (Mgr/Sr Mgr grade)
Location: Navi Mumbai
Function: Information Management / IT


Key Accountabilities
1. Platform Strategy & Technical Leadership
Own the enterprise data platform strategy and roadmap, guiding the transition from legacy platforms to Microsoft Fabric, Databricks and/or Snowflake.
Act as the primary technical decision-maker for data platform tools, standards and architecture choices.
Lead architecture and design reviews, establishing engineering quality standards.
Represent data platform engineering in leadership forums and vendor discussions.
Stay current with data integration, automation and emerging data platform technologies and apply them where they deliver business value.
2. Project Leadership & Cross-Functional Delivery
Lead major data platform projects end-to-end, coordinating across engineering, analytics, IT and business teams.
Serve as the technical lead for enterprise data migration programs.
Follow a value-first delivery approach by prototyping early, reducing data debt and ensuring operational readiness.
Create and maintain architecture documents, runbooks, technical documentation and project status updates.
3. Platform Architecture & Migration
Design and implement enterprise Lakehouse, Data Lake and Data Warehouse architectures.
Build and maintain enterprise Medallion Architecture (Bronze–Silver–Gold).
Lead migration and decommissioning of SAP BW and on-premise SQL platforms, including historical data loads, validation and cutover.
Define and enforce platform standards, reusable templates and engineering guardrails.
Design integration patterns for Palantir Foundry and Aera within the Fabric/Databricks ecosystem.
4. Data Engineering & Pipeline Development
Build enterprise-grade data pipelines using Microsoft Fabric Pipelines, Azure Data Factory and Databricks.
Integrate SAP, CRM, ERP, IoT and SQL data sources using appropriate extraction and integration patterns.
Implement real-time data pipelines using Eventstream and/or Event Hubs.
Own transformation logic using Delta Live Tables, dbt and Fabric Notebooks.
Reduce data duplication through virtualization and data-sharing patterns such as Shortcuts and Delta Sharing.
Own pipeline monitoring, SLAs, alerting and incident response.
5. DataOps, Automation & Engineering Standards
Define CI/CD standards using Azure DevOps and Databricks Asset Bundles.
Implement automated data testing frameworks using DLT checks, Great Expectations and dbt tests.
Continuously improve platform performance, cost efficiency, reliability and observability.
Establish documentation standards and leverage AI-powered tools to streamline engineering documentation.
6. Data Products, Governance & Observability
Design and deliver governed data products with clear ownership, SLAs and data contracts.
Implement automated data quality controls across ingestion and transformation layers.
Build and maintain enterprise data catalog and lineage using Microsoft Purview and Unity Catalog.
Deploy data observability capabilities covering schema drift, data freshness, volume anomalies, lineage and cost monitoring.
Define systems of record and enforce appropriate RBAC, RLS and data masking controls.
7. AI & Intelligence Platform Enablement
Provide high-quality, governed and AI-ready data for ML, RAG and agentic AI workloads.
Build data preparation pipelines supporting preprocessing, embeddings and retrieval use cases.
Integrate Palantir Foundry and its ontology with Fabric/Databricks.
Champion AI-powered engineering tools such as GitHub Copilot, Databricks AI Assistant and Azure AI.
Candidate Profile
Education
Required:
Bachelor's degree in Computer Science, Software Engineering, Information Systems, Data Engineering or a closely related technical discipline.
Preferred:
Master's degree in Computer Science, Data Science or Information Management.
Equivalent professional experience and relevant certifications may be considered in lieu of an advanced degree.
Experience
Required:
10+ years of hands-on experience in data engineering, data platforms or data architecture.
3–5 years at a senior/principal technical level, owning platform strategy and enterprise-scale technical direction.
Proven experience leading complex, multi-workstream data platform projects end-to-end across engineering, analytics, IT and business teams.
Experience owning and delivering an enterprise data platform strategy, including architecture decisions, technology selection and engineering standards.
Hands-on experience with Microsoft Fabric, Databricks and/or Snowflake in enterprise Lakehouse environments.
Strong experience designing enterprise data architectures covering Lakehouse, Data Lake, Data Warehouse and Medallion architectures.
Proven experience executing on-premise-to-cloud migrations, including source mapping, historical data loads, validation and cutover.
Experience establishing data governance foundations covering catalog, lineage, data quality, access controls and ownership models.
Strong track record delivering multiple data integration patterns including ETL/ELT, replication, virtualization and streaming/event-based pipelines.
Demonstrated experience with pipeline monitoring, data observability and incident management, including SLA ownership and root-cause analysis.
Experience defining engineering standards and reusable frameworks for broader data engineering organizations.
Preferred:
Experience working in manufacturing, chemicals or process-industry environments.
Experience designing and delivering self-service or federated data products.
Exposure to Palantir Foundry, including ontology, pipelines and AIP integration.
Experience enabling data science/ML teams through feature pipelines, MLOps and model-scoring pipelines.
Preferred Certifications
Databricks Certified Data Engineer Professional
Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
Microsoft Certified: Azure Data Engineer Associate (DP-203)

Show more Show less