SonarQube: solving “Unrecoverable indexation failures” and Elasticsearch “Disk watermark exceeded” errors

By | 08/29/2019
0 Comment
 

We are using SonarQube started from a Jenkins jobs.

See more at SonarQube: running tests from Jenkins Pipeline in Docker.

So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors:

java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122)
…

And host logs for SonarQube:

Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 ERROR web[][o.s.s.es.BulkIndexer] index [components], type [auth], id [AWzcfdpYaLFDDA6l6km3], message [ClusterBlockException[blocked by: [FORBIDD
EN/12/index read-only / allow delete (api)];]]
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 ERROR web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – too many failures [167/167 documents], waiting for next run
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:26 INFO  web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – 167 documents processed [167 failures] | time=51ms
Aug 29 12:30:48 jenkins-production docker-compose[12591]: sonarqube_1  | 2019.08.29 09:30:48 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [mjJvGbfOSZyVqbKWKCGwxw][sonarqube][/opt/sonarqube/data/es6/nodes
/0] free: 11.6gb[11.8%], replicas will not be assigned to this node

Check the documentation about Disk-based shard allocation – need to set cluster.routing.allocation.disk.watermark.low or disable those checks at all by setting the "cluster.routing.allocation.disk.threshold_enabled" : false.

Although we have 12 GB free from the 100 GB total on the Jenkins host – but ElasticSearch by default count in percents so it’s not enough for him.

Because Elasticsearch is running inside (!) of the SonarQube’s container – let’s find where is the ElasticSearch config is stored there:

[simterm]

root@jenkins-production:/opt/jenkins# docker exec -ti jenkins_sonarqube_1 find / -name elasticsearch.yml
find: ‘/proc/tty/driver’: Permission denied
/opt/sonarqube/temp/conf/es/elasticsearch.yml
/opt/sonarqube/elasticsearch/config/elasticsearch.yml

[/simterm]

Here we are – /opt/sonarqube/elasticsearch/config/elasticsearch.yml.

Sonar is started from a Docker Compose file with some files/dirs mapping:

version: '3.5'

networks:
  sonar:
    external:
      name: jenkins

services:

  sonarqube:
    user: 1004:1004
    image: sonarqube:7.9.1-community
    ports:
      - "9000:9000"
    networks:
      - sonar
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/logs:/opt/sonarqube/logs
      - /data/sonarqube/temp:/opt/sonarqube/temp
      - /data/sonarqube/data:/opt/sonarqube/data
      - /data/sonarqube/extensions:/opt/sonarqube/extensions
      - /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins
    logging:
      driver: "journald"
...

Available options for the disk management are:

  • cluster.routing.allocation.disk.threshold_enabled: будет ли Elasticsearch вообще проверять состояние диска
  • cluster.routing.allocation.disk.watermark.low: default 85%, Elasticsearch не будет размещать шарды на этой ноде
  • cluster.routing.allocation.disk.watermark.high: default 90%, Elasticsearch попробует перенести шарды с этой ноды на другие
  • cluster.routing.allocation.disk.watermark.flood_stage: default 95%, запрещает обновление индексов, шарды которых на этой ноде

Create a new config on the host – /data/sonarqube/conf/elasticsearch.yml:

cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.routing.allocation.disk.watermark.high: 90%

Map it to the SonarQube’s container SonarQube:

...
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/conf/elasticsearch.yml:/opt/sonarqube/elasticsearch/config/elasticsearch.yml
...

To access the Elasticsearch – need to set sonar.search.httpPort for the SonarQube, see the How to Monitor ElasticSearch.

Enable this connector 9100:

...
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
      - sonar.search.httpPort=9100
...

Restart the service:

[simterm]

root@jenkins-production:/home/admin# systemctl restart sonarqube

[/simterm]

And a bit logs says this option will be deprecated soon:

2019.08.29 10:56:15 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.08.29 10:56:15 WARN  app[][o.s.a.es.EsSettings] Elasticsearch HTTP connector is enabled on port 9100. MUST NOT BE USED FOR PRODUCTION
2019.08.29 10:55:10 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

Check parameters after the restart to make sure Elasticsearch sees them.

As Elasticsearch listen on the localhost only inside (again!) the SonarQube’s container – use curl to check from the SonarQube itself:

[simterm]

root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl localhost:9100/_cluster/settings?include_defaults=true | jq '.[].cluster.routing.allocation.disk'
null
null
{
  "threshold_enabled": "true",
  "watermark": {
    "low": "85%",
    "flood_stage": "95%",
    "high": "90%"
  },
  "include_relocations": "true",
  "reroute_interval": "60s"
}

[/simterm]

Okay – new settings applied, can proceed with builds now.

Used disc space per node can be checked by calling the next API:

[simterm]

root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl 'http://localhost:9100/_cat/allocation?v&pretty'
shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node
    24      613.4kb    86.3gb     11.5gb     97.9gb           88 127.0.0.1 127.0.0.1 sonarqube

[/simterm]

All available parameters for the Elasticsearch – https://nickcanzoneri.com/elasticsearch-settings.

Done.