5251-5300 of 10000 results (65ms)
2022-06-10 §
06:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T298560)', diff saved to https://phabricator.wikimedia.org/P29611 and previous config saved to /var/cache/conftool/dbconfig/20220610-063119-ladsgroup.json [production]
06:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P29610 and previous config saved to /var/cache/conftool/dbconfig/20220610-061613-ladsgroup.json [production]
06:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P29609 and previous config saved to /var/cache/conftool/dbconfig/20220610-060108-ladsgroup.json [production]
05:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T298560)', diff saved to https://phabricator.wikimedia.org/P29608 and previous config saved to /var/cache/conftool/dbconfig/20220610-054603-ladsgroup.json [production]
00:33 <ejegg> rolled back payments-wiki from 05139a0c to 8c6208c2 [production]
00:23 <ejegg> updated payments-wiki from 8c6208c2 to 05139a0c [production]
2022-06-09 §
21:38 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1041.eqiad.wmnet [production]
21:34 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1041.eqiad.wmnet [production]
21:13 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
21:12 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1142.eqiad.wmnet with OS buster [production]
21:09 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
20:59 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [production]
20:56 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [production]
20:55 <ejegg> updated fundraising CiviCRM from b0b400ae to 3cb5e6dd [production]
20:52 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1143.eqiad.wmnet with OS buster [production]
20:49 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
20:47 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1145.eqiad.wmnet with OS buster [production]
20:46 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1146.eqiad.wmnet with OS buster [production]
20:46 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1144.eqiad.wmnet with OS buster [production]
20:45 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS buster [production]
20:44 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
20:43 <thcipriani> end utc late backport window [production]
20:40 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1142.eqiad.wmnet with OS buster [production]
20:39 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1016.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:39 <thcipriani@deploy1002> Synchronized php-1.39.0-wmf.15/extensions/GrowthExperiments/modules: Backport: [[gerrit:803969|Suggested edits: Fix loading states when fetching additional tasks (T309926)]] (duration: 03m 37s) [production]
20:38 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1146.eqiad.wmnet with OS buster [production]
20:37 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1145.eqiad.wmnet with OS buster [production]
20:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1144.eqiad.wmnet with OS buster [production]
20:35 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster [production]
20:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:26 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS buster [production]
20:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1016.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:23 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:18 <thcipriani> mwmaint1002:mwscript namespaceDupes.php kywiki --fix [production]
20:16 <thcipriani@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:803916|kywiki: Add $wgSitename, $wgMetaNamespace & $wgMetaNamespaceTalk (T309866)]] (duration: 03m 36s) [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:03 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host elastic2053.codfw.wmnet [production]
20:03 <bking@cumin1001> START - Cookbook sre.hosts.reboot-single for host elastic2053.codfw.wmnet [production]
20:01 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:59 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:56 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:54 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:51 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]