Job Summary
We are seeking a highly skilled and hands-on Databricks Solution Architect with deep expertise in designing, building, and optimizing Lakehouse-based Data Engineering, Machine Learning, and AI solutions on the Databricks platform. This contract role is based in Singapore, offering flexibility to work on-site with enterprise clients or remotely within the region.
The ideal candidate will have strong experience in architecting modern data platforms, implementing scalable data pipelines, enabling MLOps, and delivering AI/LLM use cases using Databricks technologies. You will lead technical solution design, mentor engineering teams, and collaborate closely with business stakeholders to deliver secure, scalable, and high-performance data solutions.
Key Responsibilities
- Design, architect, and implement enterprise-scale Databricks Lakehouse solutions for analytics, data engineering, machine learning, and AI workloads.
- Develop scalable batch and streaming data pipelines using Delta Live Tables (DLT), Apache Spark, Spark SQL, Python, and Scala.
- Build, optimize, and maintain Delta Lake architectures for high-performance data processing.
- Configure and administer Databricks Workspaces, Unity Catalog, SQL Warehouses, clusters, and job orchestration.
- Design secure, governed, and compliant data platforms using Unity Catalog, access controls, and data governance best practices.
- Implement Infrastructure as Code (IaC) using Terraform and automate deployments through Git-based CI/CD pipelines.
- Optimize ETL, ELT, streaming, and machine learning workloads for performance, scalability, cost efficiency, and reliability.
- Build and operationalize MLOps pipelines using MLflow, model registry, experiment tracking, and automated model deployment.
- Develop AI and Generative AI solutions leveraging DBRX, Mosaic AI, Retrieval-Augmented Generation (RAG), and LLM frameworks.
- Integrate enterprise AI services such as OpenAI, Amazon Bedrock, Azure OpenAI, or Hugging Face where applicable.
- Implement monitoring, observability, and cost optimization using Splunk, Prometheus, CloudWatch, or similar monitoring platforms.
- Collaborate with cross-functional teams, business stakeholders, architects, and customers to define technical roadmaps and deliver enterprise data solutions.
- Provide technical leadership, conduct architecture reviews, and mentor engineering teams on Databricks and cloud best practices.
Required Skills & Experience
- 8+ years of experience in Data Engineering, Big Data, Cloud Data Platforms, or Analytics.
- 4+ years of hands-on experience with the Databricks platform.
- Strong expertise in:
- Databricks Workspaces
- Delta Lake
- Delta Live Tables (DLT)
- Unity Catalog
- SQL Warehouses
- Notebooks
- Job orchestration
- Strong programming skills in Python and/or Scala.
- Extensive experience with Apache Spark and Spark SQL.
- Strong knowledge of Lakehouse architecture, data modeling, ETL/ELT design, and streaming data processing.
- Experience implementing MLOps practices using MLflow, model lifecycle management, and automated deployment pipelines.
- Hands-on experience with Terraform, Git, Azure DevOps, GitHub Actions, or Jenkins.
- Strong understanding of cloud platforms:
- AWS
- Microsoft Azure
- Google Cloud Platform (GCP)
- Experience implementing security, governance, and data access controls using Unity Catalog.
- Experience with monitoring, logging, and observability tools.
- Strong problem-solving, communication, and stakeholder management skills.
