Bug with Voyager Ingress using AKS

Good morning guys. We had an incident earlier this morning with Voyager in our AKS cluster. We fixed it by creating a new ingress and we redirected the traffic to it. One of our ingresses started to send updates to Azure API repetitively. The problem only occured with 1 of our ingresses. The problematic ingress could not get traffic while doing that. When I go to Azure Portal, I can see an anormal amount of activity logs coming from our service principal (the one voyager uses).

When doing a kubectl describe ingresses.voyager.appscode.com/lucid:

Events:
  Type    Reason                         Age    From              Message
  ----    ------                         ----   ----              -------
  Normal  DeploymentReconcileSuccessful  89m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   89m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  86m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   86m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  76m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   76m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  66m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   66m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  56m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   56m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  46m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   46m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  36m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   36m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  26m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   26m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  16m    voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  StatsServiceDeleteSuccessful   16m    voyager-operator  Successfully deleted stats Service voyager-lucid-stats
  Normal  DeploymentReconcileSuccessful  6m37s  voyager-operator  Successfully patched HAProxy Deployment voyager-lucid
  Normal  ServiceReconcileSuccessful     6m37s  voyager-operator  Successfully patched LoadBalancer Service voyager-lucid
  Normal  StatsServiceDeleteSuccessful   6m36s  voyager-operator  Successfully deleted stats Service voyager-lucid-stats

Even now, Voyager continues to try to patch this ingress. I looked at the ingress logs and found nothing except that each 10 minutes, I can see a Sync/Add/Update cycle. I don’t know why Voyager behaves this way because this ingress has been stable for more than 5 months. We didn’t change the config and nobody touched the cluster at 3:20(EDT) in the morning.

Please note that the time here is in UTC (13:58 UTC = 9:58 EDT)

...
I0719 13:58:30.667523      25 reload.go:44] haproxy daemon running (pid 24)
Sync/Add/Update for Ingress lucid
Sync/Add/Update for Ingress lucid
Sync/Add/Update for Ingress lucid
I0719 13:59:00.667507      25 reload.go:44] haproxy daemon running (pid 24)
I0719 13:59:30.667509      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:00:00.667513      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:00:30.667548      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:01:00.667514      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:01:30.667662      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:02:00.667559      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:02:30.667508      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:03:00.667493      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:03:30.667543      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:04:00.667496      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:04:30.667483      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:05:00.667489      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:05:30.667494      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:06:00.667454      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:06:30.667490      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:07:00.667490      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:07:30.667485      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:08:00.667471      25 reload.go:44] haproxy daemon running (pid 24)
I0719 14:08:30.668599      25 reload.go:44] haproxy daemon running (pid 24)
Sync/Add/Update for Ingress lucid
Sync/Add/Update for Ingress lucid
Sync/Add/Update for Ingress lucid
I0719 14:09:00.667982      25 reload.go:44] haproxy daemon running (pid 24) 
...

We kept the old ingress alive in hope to find the root of the issue. The problem on this specific ingress is still happening now.

It seems that I have this issue too: https://github.com/appscode/voyager/issues/1375

I will update my post as I find more information.

EDIT: I found more information in Azure Change History

Thank you for your support,
Richer