7351-7400 of 10000 results (91ms)
2023-06-21 §
06:40 <hashar@deploy1002> Finished deploy [integration/docroot@51d2552]: Add TimedMediaHandler to docroot - T338458 (duration: 00m 11s) [production]
06:40 <hashar@deploy1002> Started deploy [integration/docroot@51d2552]: Add TimedMediaHandler to docroot - T338458 [production]
06:07 <ariel@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1003.eqiad.wmnet with reason: host reimage [production]
06:04 <ariel@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1003.eqiad.wmnet with reason: host reimage [production]
06:03 <ariel@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1003.eqiad.wmnet with OS bullseye [production]
00:20 <tzatziki> removing one file for legal compliance [production]
00:13 <tzatziki> removing 2files for legal compliance [production]
00:11 <tzatziki> removing one file for legal compliancee [production]
2023-06-20 §
23:33 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
22:47 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS buster [production]
22:37 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
22:23 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
22:19 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
22:16 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [production]
22:13 <tgr_> UTC late backports done [production]
22:11 <tgr@deploy1002> Finished scap: Backport for [[gerrit:931084|Section images: Fix ve.scrollIntoView override (T339900 T335209)]], [[gerrit:931685|Backport translations from master (T339225)]] (duration: 22m 30s) [production]
22:01 <tgr@deploy1002> tgr and kharlan: Backport for [[gerrit:931084|Section images: Fix ve.scrollIntoView override (T339900 T335209)]], [[gerrit:931685|Backport translations from master (T339225)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
21:59 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_esams [production]
21:59 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS buster [production]
21:49 <tgr@deploy1002> Started scap: Backport for [[gerrit:931084|Section images: Fix ve.scrollIntoView override (T339900 T335209)]], [[gerrit:931685|Backport translations from master (T339225)]] [production]
21:36 <otto@deploy1002> Synchronized wmf-config/ext-EventStreamConfig.php: wgEventStreams - page_content_change should use eventgate-analytics-external for canary events - T336817 (duration: 07m 22s) [production]
21:26 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
21:26 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
21:25 <sbassett> Deployed updated mitigation for T336027 [production]
21:18 <tgr@deploy1002> Finished scap: Backport for [[gerrit:931085|Remove unused data attribs on a/v sources (T199129)]] (duration: 18m 45s) [production]
21:01 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3053.esams.wmnet,cp3055.esams.wmnet,cp3057.esams.wmnet,cp3059.esams.wmnet,cp3061.esams.wmnet,cp3063.esams.wmnet,cp3065.esams.wmnet} and A:cp [production]
21:01 <tgr@deploy1002> jforrester and tgr: Backport for [[gerrit:931085|Remove unused data attribs on a/v sources (T199129)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
21:00 <tgr@deploy1002> Started scap: Backport for [[gerrit:931085|Remove unused data attribs on a/v sources (T199129)]] [production]
20:48 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1026.eqiad.wmnet with OS bullseye [production]
20:47 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
20:46 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
20:46 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1027.eqiad.wmnet with OS bullseye [production]
20:46 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
20:44 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
20:42 <gmodena@deploy1002> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:42 <gmodena@deploy1002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
20:31 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1026.eqiad.wmnet with reason: host reimage [production]
20:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
20:30 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
20:29 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: host reimage [production]
20:26 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1026.eqiad.wmnet with reason: host reimage [production]
20:25 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: host reimage [production]
20:16 <samtar@deploy1002> Finished scap: Backport for [[gerrit:931655|Turn off Zebra test for multiple wikis (T337956)]] (duration: 13m 32s) [production]
20:13 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1026.eqiad.wmnet with OS bullseye [production]
20:13 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1027.eqiad.wmnet with OS bullseye [production]
20:09 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS buster [production]
20:03 <samtar@deploy1002> ksarabia and samtar: Backport for [[gerrit:931655|Turn off Zebra test for multiple wikis (T337956)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
20:02 <samtar@deploy1002> Started scap: Backport for [[gerrit:931655|Turn off Zebra test for multiple wikis (T337956)]] [production]
19:33 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host dbproxy1023.eqiad.wmnet with OS bullseye [production]
19:33 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts parse1002.eqiad.wmnet [production]