We have a Jenkins instance with a bunch of jobs.
After a while – a new warning appeared:
Jenkins: your Jenkins data directory /var/lib/jenkins (aka JENKINS_HOME) is almost full
Checked disk usage with the ncdu and found few jobs which used really lot of space:
[simterm]
root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/ .. 25G jobs/APITests ...
[/simterm]
And mainly space is used by a job’s logs:
[simterm]
root@jenkins-production:/data/jenkins# ls -lh /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log -rw-r--r-- 1 root root 77M Mar 1 12:49 /data/jenkins/jobs/APITests/jobs/Projectname/jobs/Stage/jobs/ProjectnameStageAPItests/builds/551/log
[/simterm]
There is a lot of old builds and each keeps its log.
The simplest way to solve this issue is to enable the Discard Old Builds option in a job’s config:
Apparently, it was added in Jenkins v2.х, as previously I remember had to manually describe a workspace cleanup in a Jenkinsfiles.
Check disk used now:
[simterm]
root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/ 23G jobs/APITests/jobs/Projectname/jobs 23G jobs/APITests/jobs/Projectname/
[/simterm]
Save new settings, run a new build and check disk after it:
[simterm]
root@jenkins-production:/data/jenkins# du -h --max-depth 1 jobs/APITests/jobs/Projectname/ 7.0G jobs/APITests/jobs/Projectname/jobs 7.0G jobs/APITests/jobs/Projectname/
[/simterm]
Also, there is the Discard Old Build plugin which allows to configure the cleanup process more accurately.
