Rahul Kumar

Attended Data for Breakfast by Snowflake in Mumbai - and one idea stayed with me: 👉 AI is no longer the hard part. Making it useful is. A lot of conversations focused on moving from experiments to real production use cases. The sessions around agentic AI and how Amazon Web Services + Snowflake are enabling that shift were especially interesting. My biggest takeaway: Before building AI, you need to get your data, metadata, and governance right. Without context, even the best models will only give generic outputs. This is something I’ve personally seen while working on metadata-driven data systems - strong foundations make all the difference. Really insightful sessions from speakers like Madhan Arumugam Ramakrishnan, Farhan Choudhary, Vijayant Rai, and Sagar Pawar - especially around how enterprise AI is evolving in India. What I’m diving deeper into next: • Data + AI integration • Agentic workflows • Making data systems more accessible for business users Great mix of learning, networking, and real-world use cases. If you attended as well, would love to hear your key takeaways 👇 #DataEngineering #AI #Snowflake #AWS #Data #Analytics #AgenticAI #Mumbai #Learning Snowflake

61

1 Yorum

Most streaming architectures still pay for a hop nobody asked for. Databricks Zerobus Ingest — part of Lakeflow Connect — is now GA, and it removes that hop. The familiar pattern: your app, vehicle fleet or IoT device publishes to Kafka or Event Hubs, a connector or streaming job reads from it, and only then does the data land in Delta. Three systems to size, secure and pay for before a single row is queryable. Zerobus lets the producer write directly into a Unity Catalog governed streaming table. One call, no message bus in between. Databricks quotes sub-5-second latency across thousands of concurrent connections, with a production gRPC API (REST still Beta) and SDKs for Python, Java, Go, Rust and TypeScript. Available on AWS and Azure today. When I would reach for it: high-volume telemetry or event data whose only destination is the lakehouse. When I would keep Kafka: when several consumers need the same stream, or you need replay. Zerobus is deliberately single-sink. Good reminder that the cheapest component in an architecture is usually the one you delete. Where do you land — fewer moving parts, or keep the bus for optionality? #Databricks #DataEngineering #Lakehouse #StreamingData #Lakeflow

4

2 Yorum

💡Do you want to implement AI in your organization? Download this guide to learn what's worked for Oracle — and common pitfalls to avoid: https://lnkd.in/gEbQMcgd

19

Snowflake: Did I forget to add salt to the biryani? curl --location 'https://lnkd.in/gkGpnU4G' \  --header "Authorization: Bearer abcd" The /api/v2/databases endpoint also returns 404. This confirms that the Snowflake REST API v2 endpoints are not accessible with PAT authentication in your account configuration. Summary: ❌ REST API /api/v2/* endpoints return 404 with PAT Bearer token authentication ✅ Python SDK with PAT works perfectly

6

Is partitioning data overrated when scaling ML inference on Databricks? A recent case study compared two approaches – Liquid tables versus traditional partitioned tables – and here is what stood out: Key Takeaways • Liquid tables delivered up to 4x higher concurrency with lower latency • Partitioned tables benefited from data skipping but suffered from small-file overhead • Adding a salt key mitigated data skew but introduced complexity and planning overhead • Autoscaling clusters ensured cost efficiency without sacrificing performance • Optimal file size (100-200 MB) mattered more than deep partition hierarchies Controversial Insight Avoid over-partitioning. Too many partitions can throttle the Spark driver and increase job latency. Sometimes simpler Liquid tables win. Actionable Recommendations 1 Start with Liquid tables for ease of management and predictable performance 2 Use autoscaling clusters to balance cost and throughput 3 Aim for medium-sized files instead of exhaustive partition levels 4 Introduce salting only when you hit severe data skew, not by default By challenging the assumption that more partitions always equals better performance, you can streamline your MLOps pipeline and reduce costs. #MachineLearning #MLOps #Databricks #DataEngineering #BigData #Analytics

1