2751-2800 of 10000 results (112ms)
2024-06-13 ยง
11:35 <jelto@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
11:29 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
11:28 <jelto@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
11:27 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster2001.codfw.wmnet [production]
11:22 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
11:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T367261)', diff saved to https://phabricator.wikimedia.org/P64830 and previous config saved to /var/cache/conftool/dbconfig/20240613-112122-marostegui.json [production]
11:20 <cgoubert@cumin1002> START - Cookbook sre.hosts.reboot-single for host kubemaster2001.codfw.wmnet [production]
11:19 <cgoubert@cumin1002> conftool action : set/pooled=inactive; selector: name=wikikube-ctrl2003.codfw.wmnet [production]
11:17 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2151 (T367261)', diff saved to https://phabricator.wikimedia.org/P64829 and previous config saved to /var/cache/conftool/dbconfig/20240613-111706-marostegui.json [production]
11:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
11:16 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1222 (T352010)', diff saved to https://phabricator.wikimedia.org/P64828 and previous config saved to /var/cache/conftool/dbconfig/20240613-111655-ladsgroup.json [production]
11:16 <moritzm> installing pillow security updates [production]
11:16 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
11:16 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
11:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T367261)', diff saved to https://phabricator.wikimedia.org/P64827 and previous config saved to /var/cache/conftool/dbconfig/20240613-111642-marostegui.json [production]
11:16 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
11:16 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T352010)', diff saved to https://phabricator.wikimedia.org/P64826 and previous config saved to /var/cache/conftool/dbconfig/20240613-111633-ladsgroup.json [production]
11:14 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster2002.codfw.wmnet [production]
11:09 <jiji@cumin1002> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad [production]
11:08 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:08 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
11:07 <cgoubert@cumin1002> START - Cookbook sre.hosts.reboot-single for host kubemaster2002.codfw.wmnet [production]
11:01 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P64825 and previous config saved to /var/cache/conftool/dbconfig/20240613-110135-marostegui.json [production]
11:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P64824 and previous config saved to /var/cache/conftool/dbconfig/20240613-110126-ladsgroup.json [production]
10:59 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:58 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:56 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster1001.eqiad.wmnet [production]
10:55 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
10:52 <fabfur@cumin1002> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
10:49 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
10:49 <cgoubert@cumin1002> START - Cookbook sre.hosts.reboot-single for host kubemaster1001.eqiad.wmnet [production]
10:48 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:48 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:48 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production [production]
10:48 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:47 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:47 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster1002.eqiad.wmnet [production]
10:47 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:46 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:46 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P64823 and previous config saved to /var/cache/conftool/dbconfig/20240613-104628-marostegui.json [production]
10:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P64822 and previous config saved to /var/cache/conftool/dbconfig/20240613-104619-ladsgroup.json [production]
10:43 <fabfur@cumin1002> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
10:42 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production [production]
10:41 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging [production]
10:41 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:41 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main2010.codfw.wmnet [production]
10:41 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:40 <cgoubert@cumin1002> START - Cookbook sre.hosts.reboot-single for host kubemaster1002.eqiad.wmnet [production]