601-650 of 10000 results (43ms)
2022-03-09 ยง
19:30 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:29 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:23 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:22 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:21 <dancy@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.24 refs T300201 (duration: 00m 50s) [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:20 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.24 refs T300201 [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:07 <dancy@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.25 refs T300201 (duration: 00m 49s) [production]
19:06 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.25 refs T300201 [production]
18:23 <marostegui@cumin2002> dbctl commit (dc=all): 'Depooling db1127 (T300775)', diff saved to https://phabricator.wikimedia.org/P22222 and previous config saved to /var/cache/conftool/dbconfig/20220309-182355-marostegui.json [production]
18:23 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300775)', diff saved to https://phabricator.wikimedia.org/P22221 and previous config saved to /var/cache/conftool/dbconfig/20220309-182316-marostegui.json [production]
18:07 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22220 and previous config saved to /var/cache/conftool/dbconfig/20220309-180741-marostegui.json [production]
17:52 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22219 and previous config saved to /var/cache/conftool/dbconfig/20220309-175205-marostegui.json [production]
17:42 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:42 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:41 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
17:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:36 <marostegui@cumin2002> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300775)', diff saved to https://phabricator.wikimedia.org/P22217 and previous config saved to /var/cache/conftool/dbconfig/20220309-173630-marostegui.json [production]
17:36 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:33 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:31 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
17:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:29 <reedy@deploy1002> Synchronized php-1.38.0-wmf.25/extensions/WebAuthn/: T303404 (duration: 00m 53s) [production]
17:29 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:28 <reedy@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/WebAuthn/: T303404 (duration: 00m 51s) [production]
17:17 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2008.codfw.wmnet with OS bullseye [production]
17:04 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2008.codfw.wmnet with reason: host reimage [production]
17:01 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2008.codfw.wmnet with reason: host reimage [production]
16:56 <akosiaris> reboot rdb[2008,2010].codfw.wmnet,rdb[1010,1012].eqiad.wmnet for upgrades [production]
16:49 <akosiaris> reboot rdb2008 for upgrades [production]
16:45 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2008.codfw.wmnet with OS bullseye [production]
16:22 <moritzm> installing 5.10.103 kernels on bullseye hosts [production]
16:10 <btullis@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host karapace1001.eqiad.wmnet [production]
16:00 <btullis@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:57 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.25/includes/parser/Sanitizer.php: 31189c6aa4dc880a9eebe6824dbc031e9109384f: Ensure that the recognizedTagData static cache is properly initialized (T303360) (duration: 00m 51s) [production]
15:56 <btullis@cumin1001> START - Cookbook sre.dns.netbox [production]
15:56 <btullis@cumin1001> START - Cookbook sre.ganeti.makevm for new host karapace1001.eqiad.wmnet [production]
15:33 <jbond> deploy gerrit:740818 to add more genral rate limits for crawling cached and upload pages [production]