Building a Medallion Architecture on Databricks (AWS)
The Medallion Architecture has become the de facto standard for organizing data in a lakehouse. In this post, we expl...
Data Architecture • Cloud Platforms
Designing Data Ecosystems
Designing Data Ecosystems
at Enterprise Scale
Assistant Director - Data Engineer at Moody's Analytics. Previously at FactSet, Franklin Templeton & S&P Global. Specializing in Microsoft Fabric, Azure Databricks on AWS, and modern lakehouse architectures.
Architectural Gallery
Enterprise-grade data platform designs and reference architectures
Medallion Lakehouse Architecture
Bronze-Silver-Gold layered data platform on Databricks with Delta Lake and Unity Catalog governance.
Microsoft Fabric End-to-End Pipeline
Unified analytics platform leveraging OneLake, Data Factory, and Power BI for enterprise reporting.
Real-Time Streaming Data Platform
Event-driven architecture with Kafka, Spark Structured Streaming, and Delta Live Tables.
Data Mesh on Cloud
Domain-oriented decentralized data ownership with federated governance and self-serve infrastructure.
Professional Path
A journey through data engineering leadership across global financial enterprises
Current
Assistant Director - Data Engineer
Moody's Analytics
Leading data platform modernization using Databricks on AWS — building lakehouse architectures, Delta Live Tables pipelines, and Unity Catalog governance frameworks.
Previous
Product Specialist II / Senior Data Engineer
FactSet
Architected high-throughput data pipelines processing financial market data at scale with Spark and cloud-native services.
Previous
Research Analyst / Data Engineer
Franklin Templeton
Built ETL frameworks and data warehousing solutions for investment analytics and portfolio management systems.
Previous
Data Researcher II/ Data Engineer
S&P Global
Developed data integration solutions and analytics dashboards for credit risk and market intelligence platforms.
Project Journey
Evolving with the data stack — from Python & SQL pipelines to Databricks and Microsoft Fabric — building the platforms that serve analysts, data scientists and ML engineers
2014
Structured Finance Pricing Pipeline
Python 2.7 · SQL — Data Engineer
Aggregated and normalized real-time vendor pricing for US structured finance securities into one consistent store for the pricing desk.
View repository →
2016
Market Data Pipeline & Feature Platform
Python · pandas — Data Engineer
Conformed multi-vendor market data into analysis-ready, point-in-time feature tables serving analysts and data scientists.
View repository →
2018
Error Telemetry Pipeline (DE for ML)
Python · NLP — Data Engineer → Senior Data Engineer
Built the ingestion, feature-engineering and train/serve-parity pipeline feeding downstream error-prediction and root-cause models.
View repository →
2019
Yield Curve Outlier Detection
AWS · Streamlit · Terraform — Senior Data Engineer
First cloud build: a deployed data-quality tool for yield-curve validation, provisioned on AWS (EC2/S3/Redshift) with Infrastructure as Code.
View repository →
2021
Platform Usage Analytics
Azure (ADF · Synapse · ADLS Gen2) · Power BI — Senior Data Engineer
Orchestrated an Azure data platform with a zoned lake and star schema, serving always-current self-service BI to stakeholders.
View repository →
2022
Customs & Trade Analytics Lakehouse
Databricks · PySpark · Delta · Unity Catalog — Senior DE → Data Architect
Moved to the lakehouse: a medallion architecture enriching the Orbis data product with large-scale international-trade analytics.
View repository →
2023
Grant Data Integration Pipeline
Databricks · Delta · Great Expectations — Data Architect
End-to-end automated integration with data quality as a first-class concern — validation gates, quarantine and a reusable ingestion template.
View repository →
2025
Financial Research RAG
Databricks GenAI · Mosaic AI Vector Search · MLflow — Data & AI Platform Architect
A governed RAG platform over financial research — grounded, cited answers with evaluation and access control on the Databricks AI platform.
View repository →
2026 · Present
Enterprise Lakehouse on Microsoft Fabric
Microsoft Fabric · OneLake · Direct Lake — Data & AI Platform Architect
A unified Fabric lakehouse serving BI (Direct Lake), analysts (SQL) and data science from one governed Gold layer — mirroring the Databricks build to show both platforms.
View repository →Technical Blog
Deep dives into data architecture, Databricks, and Microsoft Fabric
Data Architect Telugu
Empowering the Telugu tech community — enterprise data engineering, demystified.
తెలుగులో డేటా ఇంజనీరింగ్ — మన భాషలో, మన కోసం.
Series
తెలుగు · Telugu
Databricks Lakehouse — Zero to Production
డేటాబ్రిక్స్ లేక్హౌస్ — మొదటి నుండి ప్రొడక్షన్ వరకు
Your complete roadmap to mastering Databricks — clusters, notebooks, Delta Lake, Unity Catalog, and production-grade pipelines. Let's build this together!
Series
తెలుగు · Telugu
Microsoft Fabric — The Unified Analytics Revolution
మైక్రోసాఫ్ట్ ఫ్యాబ్రిక్ — యూనిఫైడ్ అనలిటిక్స్ విప్లవం
OneLake, Lakehouses, Data Factory, and Direct Lake mode — everything you need to architect modern analytics in Fabric. This changes the game.
Deep Dive
తెలుగు · Telugu
Medallion Architecture — Bronze, Silver & Gold Explained
మెడాలియన్ ఆర్కిటెక్చర్ — బ్రాంజ్, సిల్వర్ & గోల్డ్ వివరణ
The architecture pattern powering modern lakehouses. I'll walk you through real-world implementations with Delta Live Tables on Databricks.
Masterclass
తెలుగు · Telugu
PySpark for Data Engineers — Interview & Beyond
డేటా ఇంజనీర్ల కోసం పైస్పార్క్ — ఇంటర్వ్యూ & అంతకు మించి
Not just interview prep — real production patterns. Transformations, window functions, performance tuning, and the questions top companies actually ask.
Tutorial
తెలుగు · Telugu
Unity Catalog — Enterprise Data Governance
యూనిటీ క్యాటలాగ్ — ఎంటర్ప్రైజ్ డేటా గవర్నెన్స్
Access control, data lineage, and quality enforcement at scale. I'll show you how to set up governance that actually works across multi-cloud Databricks.
Hands-On
తెలుగు · Telugu
Delta Live Tables — Declarative ETL Pipelines
డెల్టా లైవ్ టేబుల్స్ — డిక్లరేటివ్ ETL పైప్లైన్స్
Stop writing boilerplate. DLT lets you declare your pipeline logic and handles orchestration, quality, and recovery. Let me show you how the pros do it.
Tech Stack
Tools and technologies I architect with daily
Cloud Platforms
Microsoft Azure
AWS
Microsoft Fabric
Databricks
Data Engineering
Apache Spark
PySpark
Delta Lake
Delta Live Tables
Apache Kafka
Airflow
Data Architecture
Medallion Architecture
Data Mesh
Lakehouse
Data Vault 2.0
Star Schema
Governance & Security
Unity Catalog
Purview
RBAC
Data Lineage
Data Quality
Languages & Tools
Python
SQL
Scala
Terraform
Git
Docker
Analytics & BI
Power BI
Databricks SQL
Azure Synapse
dbt
About Me
Building the future of enterprise data, one architecture at a time
Kamalakar Peta
I'm a Data Architect with deep expertise in designing and implementing enterprise-scale data platforms. My career spans leadership roles at some of the world's most respected financial services firms — Moody's Analytics, FactSet, Franklin Templeton, and S&P Global.
Currently at Moody's Analytics as Assistant Director - Data Engineer, I leverage Databricks on AWS to build lakehouse architectures that unify data engineering, data science, and analytics. I'm passionate about Microsoft Fabric, Azure Databricks, medallion architectures, and helping organizations unlock the full potential of their data estate.
Beyond work, I create Telugu-language tutorials on YouTube, making data engineering concepts accessible to the Telugu-speaking tech community worldwide.
11+
Years Experience
4
Tier-1 Firms
50+
Architectures Delivered