If you are looking to copy huge amount of data to/from AWS S3 Look no further than s5cmd Download the entire s3 bucket to local s5cmd cp s3://bucket/ . I was able to download almost 10 GB folder in couple of minutes, which is around 83MBps ~ 10 MB/s Whereas aws s3 cp --recursive s3://bucket/ . was giving me around 500 KB/s
Spring data MongoDB adds a _class attribute to every document written to MongoDB If we want to prvent adding Spring Data MongoDB from adding _class attribute, we can configure this behaviour through: @Configuration public class MongoConfig extends AbstractMongoClientConfiguration { @Override protected boolean autoIndexCreation() { return true; } @Bean public MappingMongoConverter mappingMongoConverter(MongoDatabaseFactory databaseFactory, MongoMappingContext mappingContext) { DbRefResolver dbRefResolver = new DefaultDbRefResolver(databaseFactory); MappingMongoConverter converter = new MappingMongoConverter(dbRefResolver, mappingContext); converter.setTypeMapper(new DefaultMongoTypeMapper(null)); return converter; } }
If you want to quickly spin up a local development environment of kubernets cluster, Kubernetes in docker aka Kind is the way to go Kind runs the kubernetes inside a docker container as opposed to Minikube which used to run kubernetes inside a VM Running Kind on linux requires to create a cluster configuration file cluster.yaml Here is the sample configuration which would work for most of the cases: kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: -...
The entire code of this article is present on this github repository Load testing WebSocket servers is crucial for ensuring their performance and scalability. This article will guide you through using Locust, a popular open-source load testing tool, to test your Java WebSocket server. We will use Jetty WebSocket in Java to run our websocket server. Jetty WebSocket is a Java library and part of the larger Jetty project, which is a popular open-source web...
Monitoring Redis Streams lag in Prometheus and Grafana In this post, we will look at how we can measure redis streams lag from the application perspective. We would want to measure the lag and plot this in visual format using Prometheus in Grafana Assumptions We are going to assume that we have set of streams and set of consumer groups which are consuming from the stream. We are using consumer groups because we want high...
Recently, we were doing automation testing for one of our backend services and encountered an unique scenario wherein we wanted to forward the time by 24 hrs because we had entities which were created on daily basis at certain time. To automate these scenarios, we had to wait for 24 unless we did either of these things: Somehow create those entities every few minutes instead of every 24 hours - This would require making the...
Recently I observed, Rtkit daemon has been flooding the Journalctl logs I frequently found logs such as below in the journalctl logs Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes of 1 users. Jun 13 22:51:50 madhur-b550mds3h rtkit-daemon[1922]: Supervising 2 threads of 1 processes...
Continuing my Homelab journey using Proxmox, I have setup a full fledged home network with multiple services running. Here is the quick diagram below for reference: Some of the services I am running on my homenetwork as of now are Jellyfin Commmafeed Prometheus Grafana Nginx Proxy Manager Snapdrop Uptime Kuma Change Detection qBittorrent Olivetin Vaultwarden ntfy Some of the things I am trying to solve: Easily able to VPN from outside to access my password...
When interacting with lot of websockets server, you need a nifty client to test out the behaviour of websocket server. Till now, I have been using wscat as the command line utility to interact with websockets server. However, this command line utility has several disadvantages: It doesn’t allow you to send custom ping messages It doesn’t support authentication Due to these limitations, I wrote a simple python script below which can send a custom ping...
RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data. There is a very good RediSearch getting started tutorial on github. There are several steps in bulding RediSearch module. The following steps worked on Amazon Linux 2 machine for us: yum install git perl perl-DateTime perl-JSON perl-Capture-Tiny...