Using curator to delete older elasticsearch indices
Tech Tip : Monitor virtual servers, carry on your scripting work by accessing your essential emulators and tools all at a centralized platform with hosted windows virtual desktop from <a title=Microsoft Virtual Desktop href=https://www.clouddesktoponline.com/>CloudDesktopOnline</a>. Visit <a title=Apps4Rent - Hosting Provider href=http://www.apps4rent.com/>Apps4Rent.com</a> to know more about cloud products suitable for you.
We use ELK stack heavily in our production systems for log aggregation and monitoring.
Our daily log size generated is aroudn 100GB. Since, we do not intend to keep the log files in ELK for more than a month, it becomes important that we delete those indices to free up disk space.
Fortunately, Logstash creates a new index every day by default.
Thus, we can ask Curator to simply delete the indices x days old and which follow a particular naming pattern.
pip install elasticsearch-curator to install Curator on your machine. I prefer to install it on the Elasticsearch machine itself.
Create a file
curator.yml with following contents.
Now, we need to define an action. i.e. What will curator do. There are many actions to choose from. Check the documentation for more information
- Cluster Routing
- Create Index
- Delete Indices
- Delete Snapshots
For this dicussion, we will use
Delete Indices as the action, since this is what we want to do.
Below is the sample action file
delete_indices.yml , which will delete the logstash indices which are older than 10 days.
To run this action, simple use the command
--dry-run mode will not actually delete the index. It can be used to test the output of the action.
If you want to schedule it in a cron, you can do so using
The above configuration will cleanup the indices older than 10 days everyday at 8 AM.