901-950 of 10000 results (56ms)
2022-03-09 ยง
17:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:41 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
17:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:36 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300775)', diff saved to https://phabricator.wikimedia.org/P22217 and previous config saved to /var/cache/conftool/dbconfig/20220309-173630-marostegui.json [production]
17:36 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:33 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:31 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:29 <reedy@deploy1002> Synchronized php-1.38.0-wmf.25/extensions/WebAuthn/: T303404 (duration: 00m 53s) [production]
17:29 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:28 <reedy@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/WebAuthn/: T303404 (duration: 00m 51s) [production]
17:17 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2008.codfw.wmnet with OS bullseye [production]
17:04 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2008.codfw.wmnet with reason: host reimage [production]
17:01 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2008.codfw.wmnet with reason: host reimage [production]
16:56 <akosiaris> reboot rdb[2008,2010].codfw.wmnet,rdb[1010,1012].eqiad.wmnet for upgrades [production]
16:49 <akosiaris> reboot rdb2008 for upgrades [production]
16:45 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2008.codfw.wmnet with OS bullseye [production]
16:22 <moritzm> installing 5.10.103 kernels on bullseye hosts [production]
16:10 <btullis@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host karapace1001.eqiad.wmnet [production]
16:00 <btullis@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:57 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.25/includes/parser/Sanitizer.php: 31189c6aa4dc880a9eebe6824dbc031e9109384f: Ensure that the recognizedTagData static cache is properly initialized (T303360) (duration: 00m 51s) [production]
15:56 <btullis@cumin1001> START - Cookbook sre.dns.netbox [production]
15:56 <btullis@cumin1001> START - Cookbook sre.ganeti.makevm for new host karapace1001.eqiad.wmnet [production]
15:33 <jbond> deploy gerrit:740818 to add more genral rate limits for crawling cached and upload pages [production]
15:31 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2007.codfw.wmnet with OS bullseye [production]
15:28 <volans> uploaded spicerack_2.3.0 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
15:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2007.codfw.wmnet with reason: host reimage [production]
15:16 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2007.codfw.wmnet with reason: host reimage [production]
15:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
15:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
15:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
15:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
15:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
15:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
15:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
15:06 <taavi> UTC afternoon deploys done [production]
15:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
15:06 <awight@deploy1002> Synchronized php-1.38.0-wmf.25/extensions/VisualEditor/modules/ve-mw/ui/styles/pages/ve.ui.MWParameterPage.css: Backport: [[gerrit:769297|Fix missing padding on inline descriptions (T303386)]] (duration: 00m 49s) [production]
15:05 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on 6 hosts with reason: Maintenance [production]
15:05 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 16:00:00 on 6 hosts with reason: Maintenance [production]
15:05 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
15:05 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
15:05 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298294)', diff saved to https://phabricator.wikimedia.org/P22215 and previous config saved to /var/cache/conftool/dbconfig/20220309-150523-marostegui.json [production]
15:03 <awight@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/VisualEditor/modules/ve-mw/ui/styles/pages/ve.ui.MWParameterPage.css: Backport: [[gerrit:769296|Fix missing padding on inline descriptions (T303386)]] (duration: 00m 49s) [production]
15:01 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2007.codfw.wmnet with OS bullseye [production]
15:01 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
15:00 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
15:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]