Rancher pre v1.6.22 "Hosts stuck Reconnecting" Rancher server logs show 'Cursor returned more than one result'

Follow
Table of Contents

Issue

Hosts getting stuck in either state Reconnecting or Finishing-Reconnect and Rancher server logs include errors like the following: 2019-02-26 12:05:55,265 ERROR [51e1303d-21b2-409f-ba2e-7542e8de4941:9663402] [healthcheckInstanceHostMap:445975] [healthcheckinstancehostmap.remove] [] [ecutorService-3] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.InvalidResultException: Cursor returned more than one result

Pre-requisites

Rancher version lower than 1.6.22

Workaround

  1. In the Rancher MySQL database, find all the duplicates by checking column 3 for entries with more than a count of 1 in the return from the following query:

    select host_id,healthcheck_instance_id,count(*) from healthcheck_instance_host_map where removed is null group by host_id,healthcheck_instance_id order by 3;
  2. For each healthcheck_instance_id in any row with more than 1 in column 3, run the following command:

    update healthcheck_instance_host_map set state='removed', removed=now(), remove_time=now() where healthcheck_instance_id='<INSERT_HEALTHCHECK_ID>';
  3. Wait and watch the hosts view. The hosts should all finish reconnecting and instances should update.

Resolution

Upgrade to 1.6.22+ or 2.x

Further reading

https://github.com/rancher/rancher/issues/15284

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.