Nitin Keshav

Lets Simplify Cloud Service Across Cloud AWS vs AZURE vs GCP ✨ Data Streaming - AWS + Kinesis = Real-Time Data Streaming   - Azure + Event Hubs = Real-Time Data Streaming   - GCP + Pub/Sub = Real-Time Data Streaming ✨ Data Warehousing - AWS + Redshift = Data Warehousing   - Azure + Synapse Analytics = Data Warehousing   - GCP + BigQuery = Data Warehousing ✨ Object Storage - AWS + S3 = Object Storage   - Azure + Blob Storage = Object Storage   - GCP + GCS = Object Storage ✨ Data Integration/Orchestration - AWS + Glue = Data Integration   - Azure + Data Factory = Data Integration and Orchestration   - GCP + Dataflow = Data Integration and Stream Processing ✨ Serverless Computing - AWS + Lambda = Serverless Computing   - Azure + Functions = Serverless Computing   - GCP + Cloud Functions = Serverless Computing ✨ Relational Databases - AWS + RDS = Managed Relational Databases   - Azure + SQL Database = Managed Relational Databases   - GCP + Cloud SQL = Managed Relational Databases ✨ NoSQL Databases - AWS + DynamoDB = NoSQL Databases   - Azure + Cosmos DB = NoSQL Databases   - GCP + Firestore = NoSQL Databases ✨ Big Data Processing - AWS + EMR (Elastic MapReduce) = Big Data Processing   - Azure + HDInsight = Big Data Processing   - GCP + Dataproc = Big Data Processing ✨ Machine Learning/AI - AWS + SageMaker = Machine Learning Platform   - Azure + Machine Learning = Machine Learning Platform   - GCP + AI Platform = Machine Learning Platform ✨ Content Delivery Network (CDN) - AWS + CloudFront = Content Delivery Network   - Azure + Front Door/CDN = Content Delivery Network   - GCP + Cloud CDN = Content Delivery Network ✨ Identity and Access Management - AWS + IAM = Identity and Access Management   - Azure + Active Directory = Identity and Access Management   - GCP + IAM = Identity and Access Management Image credits : Internet ! ♻️ Repost to help others grow 🔔 Follow Abhisek Sahu for more ♻️ I share cloud , data analysis/data engineering tips, real world project breakdowns, and interview insights through my free newsletter. 🤝 Subscribe for free here → https://lnkd.in/e8CaGD3p #engineering #cloud #aws #gcp #azure

416

46 Comments

🚀 Deep Dive into HDFS: Building Distributed Storage from Scratch I've been on a fascinating journey exploring Hadoop HDFS fundamentals by building a local cluster (1 NameNode + 2 DataNodes) from the ground up. This hands-on experience has given me profound respect for the complexity of distributed systems. Why learn this in 2025? Today's cloud platforms (AWS EMR, Azure HDInsight, GCP Dataproc) provide managed solutions that abstract away infrastructure complexity. We can spin up Hadoop clusters with a few clicks—no manual configuration needed. But understanding what happens under the hood is invaluable. I have immense respect for Data Engineers and DevOps professionals who deployed these architectures in the pre-LLM era, armed only with documentation, StackOverflow threads, and YouTube tutorials. No ChatGPT to explain configs, no Copilot to autocomplete XML files—just pure grit, patience, and deep technical knowledge. What I've learned: ✅ How HDFS splits files into blocks and replicates them across DataNodes ✅ Fault tolerance mechanisms when nodes fail ✅ NameNode's role in metadata management vs DataNode's block storage ✅ Block size impact on storage efficiency and parallelism ✅ Administrative operations: quotas, snapshots, ACLs, replication management ✅ Integrating Python ETL workflows with HDFS as storage backend Key insight: HDFS alone provides distributed storage, NOT distributed compute. All processing in my setup happens client-side. This is just the foundation—YARN, MapReduce, Spark, and Hive build on top of this storage layer. Hands-on learning: I've documented everything in a GitHub repo—complete configs, step-by-step setup, experiments, and troubleshooting. Clone it, run it locally, and experiment! https://lnkd.in/gritdrRZ The repo uses small 10-byte blocks (vs 128MB production default) so you can visualise block splitting without needing gigabytes of test data. What's next: This is Part 1 of a series exploring the Hadoop ecosystem: HDFS (1NN + 2DN) ✅ Will explore YARN, Hive, Spark, Iceberg etc Want to go deeper? If you have access to multiple VMs, I highly encourage setting up each component on separate machines. Running NameNode and DataNodes on different physical/virtual hosts gives you authentic distributed system behavior—network latency, node failures, and all. Let's discuss! I'm eager to learn from the community. Drop your thoughts in the comments—happy to discuss, share, and learn together! #DataEngineering #Hadoop #HDFS #DistributedSystems #BigData #LearningInPublic #CloudComputing

14

1 Comment

𝗪𝗔𝗡𝗧 𝗔 ₹𝟯𝟱–𝟰𝟬+ 𝗟𝗣𝗔 𝗗𝗔𝗧𝗔 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥 𝗥𝗢𝗟𝗘? Then stop preparing like the interview is a Spark + SQL + Kafka trivia contest. Because at senior levels, the difficult part isn't knowing: repartition() ROW_NUMBER() consumer groups broadcast joins The difficult part is explaining WHEN, WHY, and WHAT HAPPENS WHEN IT FAILS. 🔵 𝗪𝗛𝗔𝗧 𝗦𝗘𝗡𝗜𝗢𝗥-𝗟𝗘𝗩𝗘𝗟 𝗜𝗡𝗧𝗘𝗥𝗩𝗜𝗘𝗪𝗦 𝗔𝗥𝗘 𝗥𝗘𝗔𝗟𝗟𝗬 𝗧𝗘𝗦𝗧𝗜𝗡𝗚 Imagine you're asked: → Your dashboard scans 3 TB for 24 hours. What do you change first? → A JOIN turns 100M rows → 4B rows. Do you tune Spark — or question the data model? → One Spark task is 12× slower than the median. Do you add more executors? → A broadcast join causes executor OOM. Was broadcasting actually the optimization? → Kafka processing succeeds, but the DB commit succeeds while offset commit fails. Can your pipeline safely recover? → Kafka lag keeps increasing. Why might adding consumers make zero difference? These aren't tool-knowledge questions. They're engineering judgment questions. 🟣 𝗧𝗛𝗘 𝗗𝗜𝗙𝗙𝗘𝗥𝗘𝗡𝗖𝗘 𝗜𝗦 𝗜𝗡 𝗧𝗛𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗠𝗶𝗱-𝗹𝗲𝘃𝗲𝗹: “Use Spark AQE.” 𝗦𝗲𝗻𝗶𝗼𝗿: “First I’ll inspect partition sizes, shuffle read/write, spill and key distribution. If the evidence shows skew, I’ll determine whether AQE can mitigate it or whether the data model/join strategy needs to change.” That difference is what separates knowing a technology from owning a system. 🟢 𝗜 𝗣𝗨𝗧 𝗧𝗛𝗜𝗦 𝗜𝗡𝗧𝗢 𝗔 𝗙𝗥𝗘𝗘 𝗙𝗜𝗘𝗟𝗗 𝗚𝗨𝗜𝗗𝗘 Inside: ▸ 16 architecture-grade interview problems ▸ SQL + Python + Spark + Kafka ▸ Production debugging framework ▸ System-design questions ▸ Failure & recovery patterns ▸ Cost vs latency trade-offs ▸ Replay + idempotency ▸ Engineering resources worth bookmarking ▸ 4-week architecture practice plan 𝗧𝗵𝗲 𝗴𝗼𝗮𝗹 𝗶𝘀𝗻’𝘁 𝘁𝗼 𝗺𝗲𝗺𝗼𝗿𝗶𝘇𝗲 𝟭𝟲 𝗮𝗻𝘀𝘄𝗲𝗿𝘀. It's to train yourself to think: Assumptions → Failure → Trade-off → Observability → Recovery 📘 𝗜’𝗩𝗘 𝗔𝗧𝗧𝗔𝗖𝗛𝗘𝗗 𝗧𝗛𝗘 𝗙𝗜𝗘𝗟𝗗 𝗚𝗨𝗜𝗗𝗘 I’ve attached a Senior Data Engineer Architecture & Interview Field Guide to help you prepare for senior-level interviews with real-world architecture and production scenarios. 📌 Save it. Practice it. Use it as your architecture drill. #DataEngineering #SeniorDataEngineer #SystemDesign #ApacheSpark #ApacheKafka

1

2 Comments

🚀 Snowflake Interview Questions for Senior Data Engineers – Part 2 In Part 1, we covered Architecture, Performance Optimization, and Data Modeling. Now, let’s dive into Security, Advanced Features, and Cost Optimization — areas where senior engineers are expected to think strategically. 🔹 4. Data Ingestion & Integration What are the different ways to load data into Snowflake (e.g., COPY, Snowpipe, external tables)? Can you explain how Snowpipe works and when it’s the right choice? How do external stages (S3, Azure Blob, GCS) integrate with Snowflake? What’s the role of file formats (Parquet, ORC, JSON, CSV) in Snowflake ingestion? How do you design a robust ELT/ETL pipeline into Snowflake? 🔹 5. Security & Governance How do Snowflake roles and grants work? What is the purpose of network policies in Snowflake? How do you enable row-level security (RLS) and column-level security (CLS)? Can you explain masking policies and their use cases? How do you implement data sharing securely across accounts or organizations? 🔹 6. Advanced Features How do streams and tasks enable near real-time pipelines in Snowflake? Can you explain the difference between replication and database failover in Snowflake? What is Time Travel in Snowflake, and how is it useful in recovery scenarios? How does zero-copy cloning work, and when would you use it? How would you use Snowflake’s external functions with AWS Lambda or Azure Functions? 🔹 7. Monitoring & Cost Management How do you monitor warehouse usage and query performance in Snowflake? What techniques do you use to optimize Snowflake cost without impacting performance? How do you analyze query history and warehouse utilization to detect inefficiencies? What are the best practices for managing multi-cluster warehouses? How do you track storage costs, including Time Travel and Fail-safe data? 👉 Thinking of making Part 3 all about real-world scenarios in Snowflake — like, “Your costs just doubled, how do you figure out why?” Should I go ahead and share those? Follow Avinash S. for more. #Snowflake #DataEngineering #CloudData #InterviewPrep #SeniorDataEngineer Would you like me to also create a Part 3 post that includes Scenario-Based Questions (e.g., “Your Snowflake costs doubled last month, how would you investigate?”) for a more practical angle?

65

3 Comments

How Databricks Stopped a Streaming App From “Freezing” During a New Episode Drop: A popular streaming platform dropped a new episode at 8 PM. Millions of users jumped in at the same time and suddenly: ❗ Some users couldn’t load the episode ❗ Others saw the wrong video quality ❗ A few got logged out randomly On the backend, nothing looked broken, but something felt off. The culprit? A slow user-session pipeline that updates: -device type -region -subscription status -streaming quality settings One micro-batch got delayed, the platform started serving old user-session data. Here’s how the team fixed it using Databricks: ✔ Used Databricks' job run view to detect the delayed batch. ✔ Used Delta Lake time travel to compare fresh vs stale session data. ✔ Enabled Auto Optimize and file compaction to speed up ingestion. ✔ Restarted the pipeline with proper schema enforcement. ✔ Added quality expectations to flag this next time. Within minutes: 🎥 Episodes loaded correctly ⚡ Streaming quality adjusted in real time 📈 User drop-offs went back to normal 😌 No major outage, just quick data engineering magic #DataEngineer #Databricks #Pyspark #DeltaLake #BigData #DataEngineering #RealTimeData #DataPipelines #ETL #DataQuality #Analytics #CloudComputing #Azure #AWS #GCP #SQL #AI #DataOps #Observability #C2C #USJobs #Hiring #ITTechJobs

4

1 Comment

I built a full Snowpipe setup runbook — here's everything you need to automate real-time data ingestion on Snowflake. Zero manual ingestion. 10 GB/sec. Sub-10s latency. The Snowpipe 2026 runbook is here. ❄️ Snowpipe is Snowflake's serverless auto-ingestion engine — it watches an S3 bucket, detects new files the moment they land, and loads them into Snowflake tables automatically, with zero manual intervention. I've packaged an end-to-end implementation runbook covering all 8 setup steps, annotated screenshots from a live environment, and the 10 most impactful Snowpipe Streaming enhancements released between 2025 and 2026. 🔹 4-Phase Architecture: The setup follows four sequential phases — Create File Format → Create Stage → Create Pipe (AUTO_INGEST=TRUE) → Configure S3 Event Notification with the Snowflake SQS ARN. Each phase must complete before the next, and the runbook walks through each with exact SQL and AWS console steps. 🔹S3 → SQS → Snowpipe Bridge: The critical link is the notification_channel ARN retrieved via DESC PIPE. This unique SQS ARN is pasted into the AWS S3 Event Notification, creating the live trigger. Recreating a pipe changes the ARN — a common production gotcha covered in the runbook. 🔹 10 GB/sec High-Performance Streaming (GA 2026): The new Snowpipe Streaming architecture delivers sub-10-second ingest-to-query latency at 10 GB/sec per table — a leap from the minutes-latency file-based classic model. Priced at a flat 0.0037 credits/GB, it eliminates complex per-core and per-file charges. 🔹 Auto Schema Evolution: Pipelines now automatically adapt when upstream data sources add new fields — no manual DDL, no downtime. This is a game-changer for fast-moving domains like 5G network data, IoT telemetry, and event streaming. 🔹Multi-Cloud GA (AWS + Azure + GCP): High-performance Snowpipe Streaming is now generally available across all cloud, removing vendor lock-in for hybrid and multi-cloud enterprises. 🔹Event Table Observability: Snowpipe now publishes detailed ingestion events — pipe state changes, per-file progress, and periodic digests — directly into Snowflake's Event Table, giving DevOps and SRE teams production-grade pipeline visibility. 🔹 Real-World Use Cases Covered: The runbook maps each concept to industry applications — Telecom streaming billions of network events, banks catching fraud before transactions complete, factories predicting equipment failures, ICU patient vitals feeding AI early-warning systems, and financial platforms querying pre-clustered time-series data at speed. Whether you're a data engineer building your first Snowflake pipeline or an architect designing petabyte-scale real-time infrastructure. Every step includes the exact command, the AWS console path, a screenshot from a live implementation, and a pro tip drawn from production experience.  If this was helpful kindly repost and follow Kapil Rangwani for more insights!!

15

1 Comment

Spark DAG Stage Splitting — A Performance Tuning Perspective Spark splits a DAG into stages only at shuffle boundaries—this is not an implementation detail, it’s the core execution model. A new stage is created whenever data must move across partitions, which typically happens due to wide transformations such as: • Aggregations (groupBy, reduceByKey) • Non-broadcast joins • distinct operations • repartition and global sorting Why this matters: • Shuffles introduce network IO and disk spill • More stages increase scheduling overhead • Poor join strategy can explode stage count • Stage size directly impacts failure recovery time Key tuning levers highlighted in the visuals: • Prefer broadcast joins where possible • Use reduceByKey instead of groupByKey • Avoid unnecessary repartition • Push filters before joins • Cache only reused intermediates Understanding where and why stages are created is often the difference between a slow Spark job and an efficient one. 👉 Curious to hear from others: Which Spark transformation do you optimize first when performance degrades? #ApacheSpark #SparkPerformance #DataEngineering #BigData #PerformanceTuning

1

Is this holding you back from securing a Principal, Lead or Staff level Analytics Engineering role? I had an interesting conversation with a top Senior Analytics Engineer recently. . On paper, they look like a perfect candidate for Lead / Principal roles: ✅ 7+ years in data ✅ Strong SQL & data modelling fundamentals ✅ Modern data stack experience ✅ Solid software engineering practices ✅ Currently at a tier-one tech company You’d expect interviews to be straightforward. But they’re struggling to land roles at the level they’re aiming for. When we dug into it, the reason became clear. All of their best examples of impact came from previous companies. When we tried to use examples from their current role, it was much harder, because of the type of work they’re doing now. ▫️ They’re in an embedded analytics setup: ▫️ Working closely with stakeholders ▫️ Building & refactoring models, dashboards, reports. ▫️ Responding to business data requests All valuable work. But they’re not: ▫️ Designing core data models used across teams ▫️ Shaping the analytics platform ▫️ Driving standards, governance or platform improvements When companies hire Lead or Principal Analytics Engineers, they’re usually looking for people who have: Built or evolved the analytics platform Owned core data models Created reusable systems used across teams Taken analytics from 0 → 1 or 1 → 3 If your work is mostly business-as-usual analytics, you rarely get the chance to build those stories. So you end up with an impressive logo on your CV But if you are not doing impactful work – you may not be adding to your employability. If you're aiming for Lead / Principal roles, it’s worth asking yourself: Am I building systems or just delivering outputs? Am I improving the analytics platform or just operating within it? Would my recent work impress a principal-level hiring manager? Does this resonate with anyone?

27

8 Comments