Understanding how to find the latest value in a date partition column in Spark — This is a very interesting piece as it is here to bust a myth about the partitioned columns, I mean partitioning is so amazing that at certain points we start taking it for granted. …

Understanding systematic sampling and its implementation — Hey Folks! So, if you are here you must understand what sampling is! If not, no worries, we will do a quick walkthrough of sampling followed by what is systematic sampling and how to do systematic sampling with Spark. So, let’s get started! What is sampling? In this section, we try to understand…

Sharing a fresh perspective on real-time data pipelines — How often have you heard these terminologies — real-time data pipelines or real-time data processing or real-time analytics or just real-time data? These are often discussed to solve some very interesting and critical use cases such as fault detection, anomaly detection, and many more. In this article, we will take…

Step by Step Guide to running your first project with DBT — DBT is all the rage now! When I first read about it I was like okay it’s a config-driven ETL, what else is new? Then I read more about it I was like umm.. …

Understanding spark LIMIT and its performance with large datasets — If you come from the SQL world, you must be familiar with the LIMIT clause. It is pretty commonly used to see a small chunk of data. But ever wondered how it works? Spark also provides the functionality to sub-select a chunk of data with LIMIT either via Dataframe or…

Understanding singleton objects in Scala — Hey Folks, happy weekend! If you are familiar with object-oriented programming you must be familiar with the concept of a class, widely defined as A class is a blueprint that defines the variables and the methods common to all objects of a certain kind.

Guide on building text to speech converter in Python — This morning I noticed the “Listen” feature of articles on Medium and instantly loved it. I believe this is a great addition for learning on the go, and it got me thinking, how can I build one on my own? It’s a very simple implementation(Guess how many lines it takes…

Understanding one of the founding principles of Data Mesh — Now, there are multiple understandings of viewing and approaching data as a product. This article is my attempt to share my understanding based on experience and learning of the concept. Before digging into the understanding of data as a product let’s first understand What exactly is a product? Very basic question though. We are surrounded…

A hilarious encounter of my sister’s first visit to Starbucks — How will you define the typical surroundings of Starbucks? Mellow music, coffee beans fragrance and people working on their laptops, catching up with friends, or just posting pictures of the Starbucks mug on Instagram and providing them with free advertising(P.S. …