We are using SonarQube started from a Jenkins jobs.
See more at SonarQube: running tests from Jenkins Pipeline in Docker.
So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors:
java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122)
…
And host logs for SonarQube:
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.es.BulkIndexer] index [components], type [auth], id [AWzcfdpYaLFDDA6l6km3], message [ClusterBlockException[blocked by: [FORBIDD
EN/12/index read-only / allow delete (api)];]]
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – too many failures [167/167 documents], waiting for next run
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 INFO web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – 167 documents processed [167 failures] | time=51ms
Aug 29 12:30:48 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:48 INFO es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [mjJvGbfOSZyVqbKWKCGwxw][sonarqube][/opt/sonarqube/data/es6/nodes
/0] free: 11.6gb[11.8%], replicas will not be assigned to this node
Check the documentation about Disk-based shard allocation – need to set cluster.routing.allocation.disk.watermark.low
or disable those checks at all by setting the "cluster.routing.allocation.disk.threshold_enabled" : false
.
Although we have 12 GB free from the 100 GB total on the Jenkins host – but ElasticSearch by default count in percents so it’s not enough for him.
Because Elasticsearch is running inside (!) of the SonarQube’s container – let’s find where is the ElasticSearch config is stored there:
[simterm]
root@jenkins-production:/opt/jenkins# docker exec -ti jenkins_sonarqube_1 find / -name elasticsearch.yml find: ‘/proc/tty/driver’: Permission denied /opt/sonarqube/temp/conf/es/elasticsearch.yml /opt/sonarqube/elasticsearch/config/elasticsearch.yml
[/simterm]
Here we are – /opt/sonarqube/elasticsearch/config/elasticsearch.yml
.
Sonar is started from a Docker Compose file with some files/dirs mapping:
version: '3.5' networks: sonar: external: name: jenkins services: sonarqube: user: 1004:1004 image: sonarqube:7.9.1-community ports: - "9000:9000" networks: - sonar environment: - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar volumes: - /data/sonarqube/conf:/opt/sonarqube/conf - /data/sonarqube/logs:/opt/sonarqube/logs - /data/sonarqube/temp:/opt/sonarqube/temp - /data/sonarqube/data:/opt/sonarqube/data - /data/sonarqube/extensions:/opt/sonarqube/extensions - /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins logging: driver: "journald" ...
Available options for the disk management are:
cluster.routing.allocation.disk.threshold_enabled
: будет ли Elasticsearch вообще проверять состояние дискаcluster.routing.allocation.disk.watermark.low
: default 85%, Elasticsearch не будет размещать шарды на этой нодеcluster.routing.allocation.disk.watermark.high
: default 90%, Elasticsearch попробует перенести шарды с этой ноды на другиеcluster.routing.allocation.disk.watermark.flood_stage
: default 95%, запрещает обновление индексов, шарды которых на этой ноде
Create a new config on the host – /data/sonarqube/conf/elasticsearch.yml
:
cluster.routing.allocation.disk.watermark.flood_stage: 95% cluster.routing.allocation.disk.watermark.high: 90%
Map it to the SonarQube’s container SonarQube:
... volumes: - /data/sonarqube/conf:/opt/sonarqube/conf - /data/sonarqube/conf/elasticsearch.yml:/opt/sonarqube/elasticsearch/config/elasticsearch.yml ...
To access the Elasticsearch – need to set sonar.search.httpPort
for the SonarQube, see the How to Monitor ElasticSearch.
Enable this connector 9100:
... environment: - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar - sonar.search.httpPort=9100 ...
Restart the service:
[simterm]
root@jenkins-production:/home/admin# systemctl restart sonarqube
[/simterm]
And a bit logs says this option will be deprecated soon:
2019.08.29 10:56:15 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.08.29 10:56:15 WARN app[][o.s.a.es.EsSettings] Elasticsearch HTTP connector is enabled on port 9100. MUST NOT BE USED FOR PRODUCTION
2019.08.29 10:55:10 WARN es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
Check parameters after the restart to make sure Elasticsearch sees them.
As Elasticsearch listen on the localhost only inside (again!) the SonarQube’s container – use curl
to check from the SonarQube itself:
[simterm]
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl localhost:9100/_cluster/settings?include_defaults=true | jq '.[].cluster.routing.allocation.disk' null null { "threshold_enabled": "true", "watermark": { "low": "85%", "flood_stage": "95%", "high": "90%" }, "include_relocations": "true", "reroute_interval": "60s" }
[/simterm]
Okay – new settings applied, can proceed with builds now.
Used disc space per node can be checked by calling the next API:
[simterm]
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl 'http://localhost:9100/_cat/allocation?v&pretty' shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 24 613.4kb 86.3gb 11.5gb 97.9gb 88 127.0.0.1 127.0.0.1 sonarqube
[/simterm]
All available parameters for the Elasticsearch – https://nickcanzoneri.com/elasticsearch-settings.
Done.