4301-4350 of 10000 results (124ms)
2024-10-16 ยง
22:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2159 (T376905)', diff saved to https://phabricator.wikimedia.org/P70196 and previous config saved to /var/cache/conftool/dbconfig/20241016-225716-ladsgroup.json [production]
22:57 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
22:57 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
22:57 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance [production]
22:56 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2159.codfw.wmnet with reason: Maintenance [production]
22:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2150 (T376905)', diff saved to https://phabricator.wikimedia.org/P70195 and previous config saved to /var/cache/conftool/dbconfig/20241016-225646-ladsgroup.json [production]
22:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P70194 and previous config saved to /var/cache/conftool/dbconfig/20241016-224139-ladsgroup.json [production]
22:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P70193 and previous config saved to /var/cache/conftool/dbconfig/20241016-222632-ladsgroup.json [production]
22:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2150 (T376905)', diff saved to https://phabricator.wikimedia.org/P70192 and previous config saved to /var/cache/conftool/dbconfig/20241016-221125-ladsgroup.json [production]
22:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2150 (T376905)', diff saved to https://phabricator.wikimedia.org/P70191 and previous config saved to /var/cache/conftool/dbconfig/20241016-220053-ladsgroup.json [production]
22:00 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance [production]
22:00 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2150.codfw.wmnet with reason: Maintenance [production]
21:17 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1245.eqiad.wmnet with reason: Maintenance [production]
21:17 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1245.eqiad.wmnet with reason: Maintenance [production]
21:12 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
21:12 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
20:44 <gmodena@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply [production]
20:44 <gmodena@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply [production]
20:43 <gmodena@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply [production]
20:43 <gmodena@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply [production]
20:39 <gmodena@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:39 <gmodena@deploy2002> helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:37 <gmodena@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:37 <gmodena@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:31 <brennen@deploy2002> Finished deploy [phabricator/deployment@40a63c9]: deploy phab2002 for T377374 (duration: 00m 08s) [production]
20:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2237 (T376905)', diff saved to https://phabricator.wikimedia.org/P70189 and previous config saved to /var/cache/conftool/dbconfig/20241016-203034-ladsgroup.json [production]
20:30 <brennen@deploy2002> Started deploy [phabricator/deployment@40a63c9]: deploy phab2002 for T377374 [production]
20:29 <gmodena@deploy2002> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:29 <gmodena@deploy2002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:26 <jhuneidi@deploy2002> Finished scap sync-world: Backport for [[gerrit:1080780|Make wikitech a target for CentralNotice banners (T377030)]] (duration: 10m 02s) [production]
20:21 <jhuneidi@deploy2002> ejegg, jhuneidi: Continuing with sync [production]
20:18 <jhuneidi@deploy2002> ejegg, jhuneidi: Backport for [[gerrit:1080780|Make wikitech a target for CentralNotice banners (T377030)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:18 <mutante> phab2002 - ln -s /var/lib/scap/scap/bin/scap /usr/bin/scap [production]
20:17 <mutante> phab2002 - after manually running bootstrap-scap-target.sh and "Scap from local bullseye wheels successfully installed at /var/lib/scap/scap" still "cannot open `/usr/bin/scap' (No such file or directory)" though. T303559 T310740 T377374 [production]
20:17 <jhuneidi@deploy2002> Started scap sync-world: Backport for [[gerrit:1080780|Make wikitech a target for CentralNotice banners (T377030)]] [production]
20:16 <mutante> phab2002 - manually bootstrapping scap since puppet did not do it due to dependency cycles: sudo -u scap /usr/local/bin/bootstrap-scap-target.sh deploy2002.codfw.wmnet /var/lib/scap T303559 T310740 T377374 [production]
20:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P70188 and previous config saved to /var/cache/conftool/dbconfig/20241016-201527-ladsgroup.json [production]
20:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P70187 and previous config saved to /var/cache/conftool/dbconfig/20241016-200020-ladsgroup.json [production]
19:54 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx-out1001.wikimedia.org [production]
19:50 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-out1001.wikimedia.org [production]
19:49 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx-out2001.wikimedia.org [production]
19:47 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-out2001.wikimedia.org [production]
19:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2237 (T376905)', diff saved to https://phabricator.wikimedia.org/P70186 and previous config saved to /var/cache/conftool/dbconfig/20241016-194513-ladsgroup.json [production]
19:47 <jhathaway@cumin1002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host mx-out2001.wikimedia.org [production]
19:47 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-out2001.wikimedia.org [production]
19:46 <jhathaway@cumin1002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host mx-out2001.wikimedia.org [production]
19:45 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-out2001.wikimedia.org [production]
19:45 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mx1001.wikimedia.org [production]
19:44 <jhathaway@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:44 <jhathaway@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mx1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jhathaway@cumin1002" [production]