3601-3650 of 10000 results (87ms)
2023-01-05 ยง
13:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1100 (T326156)', diff saved to https://phabricator.wikimedia.org/P42857 and previous config saved to /var/cache/conftool/dbconfig/20230105-133448-ladsgroup.json [production]
13:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1100 (T326156)', diff saved to https://phabricator.wikimedia.org/P42856 and previous config saved to /var/cache/conftool/dbconfig/20230105-133234-ladsgroup.json [production]
13:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1100.eqiad.wmnet with reason: Maintenance [production]
13:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1100.eqiad.wmnet with reason: Maintenance [production]
13:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42855 and previous config saved to /var/cache/conftool/dbconfig/20230105-133211-ladsgroup.json [production]
13:30 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
13:29 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
13:21 <effie> enable puppet on all mw servers [production]
13:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P42854 and previous config saved to /var/cache/conftool/dbconfig/20230105-131705-ladsgroup.json [production]
13:03 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
13:03 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
13:03 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
13:03 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
13:03 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
13:02 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
13:02 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
13:02 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
13:02 <oblivian@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
13:02 <oblivian@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
13:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P42853 and previous config saved to /var/cache/conftool/dbconfig/20230105-130158-ladsgroup.json [production]
13:01 <oblivian@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
13:01 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
13:01 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
13:01 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
13:00 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
13:00 <hashar> Restarted Gerrit for a plugin update [production]
12:58 <hashar@deploy1002> Finished deploy [gerrit/gerrit@b1ae5b4]: wm-checks-api: fix PCC handling of empty messages (duration: 00m 08s) [production]
12:58 <hashar@deploy1002> Started deploy [gerrit/gerrit@b1ae5b4]: wm-checks-api: fix PCC handling of empty messages [production]
12:52 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
12:49 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
12:49 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
12:48 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42852 and previous config saved to /var/cache/conftool/dbconfig/20230105-124651-ladsgroup.json [production]
12:45 <hashar@deploy1002> Finished deploy [gerrit/gerrit@b1ae5b4]: wm-checks-api: fix PCC handling of empty messages (duration: 00m 10s) [production]
12:45 <hashar@deploy1002> Started deploy [gerrit/gerrit@b1ae5b4]: wm-checks-api: fix PCC handling of empty messages [production]
12:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1096:3315 (T326156)', diff saved to https://phabricator.wikimedia.org/P42851 and previous config saved to /var/cache/conftool/dbconfig/20230105-124437-ladsgroup.json [production]
12:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
12:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
12:44 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:42 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
12:42 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
12:31 <ladsgroup:> Deployed security patch for T233004 T326293 [production]
12:02 <hashar> gerrit: running `copy-approvals` script to prepare for Gerrit 3.6 upgrade (T309870): `ssh -p 29418 gerrit.wikimedia.org gerrit copy-approvals --verbose` [production]
11:59 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:58 <hashar> Restarting Gerrit [production]