Blog

In a well designed backend application, which makes calls to other HTTP services, it is very important to configure appropriate timeout values. In event of a service failure, a large timeout value can cause threads on the application to keep waiting for service to respond and may ultimately run out of threads which could have been used for doing other work. Otherwise, low timeout value can cause genuine requests to be aborted which would have...

The sample code below is available in my github repo Let’s face it, HTTP network calls can face intermittently. Even if you best configure the Linux Operating System, the TCP/IP networking stack and the best possible private / public cloud networks, failures are inevitable. Sometimes, the reason can be dropped packets at router / NIC or could be any issue in networking stack. For any well designed backend application, it is imperative to design for...

In this post, we will look at tools to test remote connectivity in Linux. With increasing usage of firewalls, server hardening due to cyber attacks, it becomes important to determine appropriate connectivity between two hots. Note that, we are looking for connectivity tests w.r.t. server systems such as databases, application servers etc. Organizations having multiple datacenters, systems running in private / public clouds, accessing vendor partner systems, its important to have a know how to...

Classloaders are an important concept for every engineer to be aware of. Classloaders are basic building blocks of Containers like Tomcat. Suppose, we have written a java application and have a singleton instance, we might say its singleton across JVM. Wrong. Its not singleton across JVM, its a singleton across classloader. The same application can be deployed in a container like Tomcat, as two different application. In that case, there will be 2 copies of...

Recently, I was trying to do a POC on Kubernetes and required a kuberentes Cluster. Its easy to spin up a single machine cluster using Minkube on Windows, Linux and MacOS But the behaviour of Minkube is not as close to real Kubernetes cluster primarily because: You can’t simulate multiple nodes You cannot have a public DNS for your services unless you use minikube tunnel The way minikube runs either in Docker, VirtualBox or over...

I was recently trying to add a package level suppression in both checkstyle / spotbugs and found it to be having wierd syntax. For checkstyle, First add a module in your checkstyle configuration file <module name="SuppressionFilter"> <property name="file" value="./suppressions.xml"/> </module> And then in the suppressions.xml , specify the package as follows: <?xml version="1.0"?> <!DOCTYPE suppressions PUBLIC "-//Puppy Crawl//DTD Suppressions 1.1//EN" "http://www.puppycrawl.com/dtds/suppressions_1_1.dtd"> <suppressions> <suppress files="[\\/]com[\\/]domain[\\/]package" checks="."/> </suppressions> For, Spotbugs, you specify the package level suppression in...

Recently, I was facing an issue in production where our backend service used to get stuck while making a HTTP client request through Apache HTTP After some debugging, I got this stack trace from the logs org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) The pecuilar thing was that it was happening for only...

Recently, our ELK infrastructure went down which aggregates log from our multiple servers. I wanted to investigate a production issue which involved looking up logs of multiple servers simultaneously. I tried different methods such as tmux, multitail and ssh command such as ssh HOST_1 "tail -f /path/to/file" | tee -a /path/to/merged/contents ssh HOST_2 "tail -f /path/to/file" | tee -a /path/to/merged/contents tail -f /path/to/merged/contents All these are nice solutions, but none of them is elegant. It...

If any host is using self-signed certificate, then a typical HTTPS request to that host will fail in the Java code with the following exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target A very NOT recommended solution to this problem is to bypass the certificate check import java.security.cert.CertificateException; import java.security.cert.X509Certificate; import javax.net.ssl.X509TrustManager; public class TrustAnyTrustManager implements X509TrustManager { public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException...

Recently, I faced a production issue where some of our requests were failing with the following exception org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:583) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:703) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Interestingly, the occurence of these errors in the logs were very few...