Jenkins: your Jenkins data directory is almost full and automated disk cleanup

By | 03/27/2019
 

We have a Jenkins instance with a bunch of jobs.

After a while – a new warning appeared:

Jenkins: your Jenkins data directory /var/lib/jenkins (aka JENKINS_HOME) is almost full

Checked disk usage with the ncdu and found few jobs which used really lot of space:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/
..
25G     jobs/APITests
...

And mainly space is used by a job’s logs:

root@jenkins-production:/data/jenkins# ls -lh /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log
-rw-r--r-- 1 root root 77M Mar  1 12:49 /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log

There is a lot of old builds and each keeps its log.

The simplest way to solve this issue is to enable the Discard Old Builds option in a job’s config:

Apparently, it was added in Jenkins v2.х, as previously I remember had to manually describe a workspace cleanup in a Jenkinsfiles.

Check disk used now:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/
23G     jobs/APITests/jobs/Projectname/jobs
23G     jobs/APITests/jobs/Projectname/

Save new settings, run a new build and check disk after it:

root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/
7.0G    jobs/APITests/jobs/Projectname/jobs
7.0G    jobs/APITests/jobs/Projectname/

Also, there is the Discard Old Build plugin which allows to configure the cleanup process more accurately.