Blog

Assuming you have spark downloaded on machine, you would run ./spark-class org.apache.spark.deploy.master.Master to run the Spark Master controller PS D:\spark\spark\bin> ./spark-class org.apache.spark.deploy.master.Master Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/10/16 09:53:02 INFO Master: Started daemon with process name: [email protected] 21/10/16 09:53:02 WARN Shell: Did not find winutils.exe: {} 21/10/16 09:53:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/10/16 09:53:02 INFO SecurityManager: Changing view acls to: user 21/10/16 09:53:02...

zsh shell by default does not come with auto suggestions facility unlike fish shell. If you are looking for auto suggestions similar to fish, have a look at https://github.com/zsh-users/zsh-autosuggestions For MAC OS, following steps are required to install git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions Add into ~/.zshrc plugins=( # other plugins... zsh-autosuggestions )

Change Data Capture (CDC) is a very important topic for any backend developer to understand. Lot of systems backbone is based on Change data capture. When I started my career as a backend developer, I did not have any understand of CDC. It is only after talking to some of the industry experts, I realized how important this topic is. In this post, we will try to understand what is CDC and its practical usages:...

When working with Kubernetes, Liveness and Readiness Probes are very important concepts to understand. Readiness Probe Kubernetes fires readiness probe to the pod to determine if the pod is ready to serve traffic or not. It can simply be defined as readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 We can also provide readiness probe as HTTP or TCP commands Liveness Probe Kubernetes fires readiness probe to the pod to determine if...

Ever since I have started working with Kafka, I have came across these terms very frequently, At-least once, At-most once and Exactly Once. As an engineer, It is very important to understand these concepts. At-most once Configuration As the name suggests, At-most-once means the message will be delivered at-most once. Once delivered, there is no chance of delivering again. If the consumer is unable to handle the message due to some exception, the message is...

I got curious about how tinyUrl does redirection for its Url. Is it standard browser redirect with 301/302 status code or something else? With that created this tinyUrl tinyurl.com/madhur25 Figured out that it was indeed a browser redirect using 301 status code. $ curl https://tinyurl.com/madhur25 -vvv * STATE: INIT => CONNECT handle 0x600057310; line 1407 (connection #-5000) * Added connection 0. The cache now contains 1 members * Trying 104.20.138.65... * TCP_NODELAY set * STATE:...

MD5 MD5 is hashing algorithm. It produces 128 bit value Base64 Encoding Base64 character encodes 6 bits of the hash value. So if we use base64 encoding of above value, it will contain 21 characters

Capacity estimation is one of the most important exercises in design. Twitter Example 1B Total Users 200 M DAU 100 M new tweets everyday Each user follows 200 people avg How many favorites per day? Each user 5 favorite per day 200M *5 = 1B favorite Total tweet views A User visits their timeline 2 times a day and visits 5 other people pages Each page has 20 tweets 200M (2+5)20 = 28B / day...

Few months back, I had to make some changes to design of Communication Service. It was quite a complex architecture overall and I thought of documenting this in blogpost to outline my thought process. A communication service is responsible for reliably delivering the customer notification to the end user via any channel. In our case, the channels were email, SMS and push notification. The communication service was divided into multiple microservices: Communication Services This was...

In a complex multiple microservices architecture, especially where in one microservice ends up calling a chain of other microservices, the API latency can become a concern. This is because there is network hop at each microservice layer, which adds to the latency in addition to the actual logic executed by the microservice itself. For example, in an e-commerce application, an Add to Cart API request might end up calling multiple downstream microservices such as Pricing...