Blog

Ever since I have started working with Kafka, I have came across these terms very frequently, At-least once, At-most once and Exactly Once. As an engineer, It is very important to understand these concepts. At-most once Configuration As the name suggests, At-most-once means the message will be delivered at-most once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is...

I got curious about how tinyUrl does redirection for its Url. Is it standard browser redirect with 301/302 status code or something else? With that created this tinyUrl tinyurl.com/madhur25 Figured out that it was indeed a browser redirect using 301 status code. $ curl https://tinyurl.com/madhur25 -vvv * STATE: INIT => CONNECT handle 0x600057310; line 1407 (connection #-5000) * Added connection 0. The cache now contains 1 members * Trying 104.20.138.65... * TCP_NODELAY set * STATE:...

MD5 MD5 is hashing algorithm. It produces 128 bit value Base64 Encoding Base64 character encodes 6 bits of the hash value. So if we use base64 encoding of above value, it will contain 21 characters

Capacity estimation is one of the most important exercises in design. Twitter Example 1B Total Users 200 M DAU 100 M new tweets everyday Each user follows 200 people avg How many favorites per day? Each user 5 favorite per day 200M *5 = 1B favorite Total tweet views A User visits their timeline 2 times a day and visits 5 other people pages Each page has 20 tweets 200M (2+5)20 = 28B / day...

Few months back, I had to make some changes to design of Communication Service. It was quite a complex architecture overall and I thought of documenting this in blogpost to outline my thought process. A communication service is responsible for reliably delivering the customer notification to the end user via any channel. In our case, the channels were email, SMS and push notification. The communication service was divided into multiple microservices: Communication Services This was...

In a complex multiple microservices architecture, especially where in one microservice ends up calling a chain of other microservices, the API latency can become a concern. This is because there is network hop at each microservice layer, which adds to the latency in addition to the actual logic executed by the microservice itself. For example, in an e-commerce application, an Add to Cart API request might end up calling multiple downstream microservices such as Pricing...

Recently, we migrated a large MySQL database containing GB’s of data, millions of rows in few tables to CosmosDB. This was done without downtime. I want to lay down the approach we took in this post. Our intention was to setup both the database active - active and incrementally shift the traffic point to Service v1 (which was reading/writing to MySQL) towards Service v2 (which was reading/writing to CosmosDB) The first challenge was doing the...

Many organizations aggregate their logs on Splunk, which is a popular product for data ingestion and building intelligence on top of it. I too have been playing with data hosted on Splunk in last few weeks. Recently, I encountered a problem, where-in the data needs to be extracted based on some specified condition within a JSON array. For example, the splunk record was as follows: { "name": "Test Name", "id": "ac646022-fa55-4e04-9856-a25a41b30014", "phases": [ { "name":...

Here are a few useful kubernetes commands: Login into the one of the pod’s shell kubectl exec --stdin --tty <<pod>> -- /bin/bash View services from all namespaces kubectl get service --all-namespaces Change the current namespace context kubectl config set-context --current --namespace = <<namespace>> Describe a pod kubectl describe <<pod>>

Distributed transactions rollback are a tough beast. There are many patterns to solve this such as: Two-phase commit Saga pattern In this post, we will look at one of the example of a problem and how to solve it using the Saga pattern. Let’s say, we are an e-commerce platform and building a backend service for taking and submitting orders. A typical flow would like this: Now, Just assume, what would happen if an intermittent...