Change Data Capture (CDC) is a very important topic for any backend developer to understand. Lot of systems backbone is based on Change data capture. When I started my career as a backend developer, I did not have any understand of CDC. It is only after talking to some of the industry experts, I realized how important this topic is. In this post, we will try to understand what is CDC and its practical usages:...
When working with Kubernetes, Liveness and Readiness Probes are very important concepts to understand. Readiness Probe Kubernetes fires readiness probe to the pod to determine if the pod is ready to serve traffic or not. It can simply be defined as readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 We can also provide readiness probe as HTTP or TCP commands Liveness Probe Kubernetes fires readiness probe to the pod to determine if...
Ever since I have started working with Kafka, I have came across these terms very frequently, At-least once, At-most once and Exactly Once. As an engineer, It is very important to understand these concepts. At-most once Configuration As the name suggests, At-most-once means the message will be delivered at-most once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is...
I got curious about how tinyUrl does redirection for its Url. Is it standard browser redirect with 301/302 status code or something else? With that created this tinyUrl tinyurl.com/madhur25 Figured out that it was indeed a browser redirect using 301 status code. $ curl https://tinyurl.com/madhur25 -vvv * STATE: INIT => CONNECT handle 0x600057310; line 1407 (connection #-5000) * Added connection 0. The cache now contains 1 members * Trying 104.20.138.65... * TCP_NODELAY set * STATE:...
MD5 MD5 is hashing algorithm. It produces 128 bit value Base64 Encoding Base64 character encodes 6 bits of the hash value. So if we use base64 encoding of above value, it will contain 21 characters
Capacity estimation is one of the most important exercises in design. Twitter Example 1B Total Users 200 M DAU 100 M new tweets everyday Each user follows 200 people avg How many favorites per day? Each user 5 favorite per day 200M *5 = 1B favorite Total tweet views A User visits their timeline 2 times a day and visits 5 other people pages Each page has 20 tweets 200M (2+5)20 = 28B / day...
Few months back, I had to make some changes to design of Communication Service. It was quite a complex architecture overall and I thought of documenting this in blogpost to outline my thought process. A communication service is responsible for reliably delivering the customer notification to the end user via any channel. In our case, the channels were email, SMS and push notification. The communication service was divided into multiple microservices: Communication Services This was...
In a complex multiple microservices architecture, especially where in one microservice ends up calling a chain of other microservices, the API latency can become a concern. This is because there is network hop at each microservice layer, which adds to the latency in addition to the actual logic executed by the microservice itself. For example, in an e-commerce application, an Add to Cart API request might end up calling multiple downstream microservices such as Pricing...
Recently, we migrated a large MySQL database containing GB’s of data, millions of rows in few tables to CosmosDB. This was done without downtime. I want to lay down the approach we took in this post. Our intention was to setup both the database active - active and incrementally shift the traffic point to Service v1 (which was reading/writing to MySQL) towards Service v2 (which was reading/writing to CosmosDB) The first challenge was doing the...
Many organizations aggregate their logs on Splunk, which is a popular product for data ingestion and building intelligence on top of it. I too have been playing with data hosted on Splunk in last few weeks. Recently, I encountered a problem, where-in the data needs to be extracted based on some specified condition within a JSON array. For example, the splunk record was as follows: { "name": "Test Name", "id": "ac646022-fa55-4e04-9856-a25a41b30014", "phases": [ { "name":...