2201-2250 of 10000 results (81ms)
2022-09-27 ยง
10:03 <moritzm> rebalance ganeti/codfw row D after completed Bullseye update T311686 [production]
09:14 <volans@cumin2002> START - Cookbook sre.hosts.provision for host logstash2037.mgmt.codfw.wmnet with reboot policy FORCED [production]
09:13 <volans@cumin2002> END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host logstash2037.mgmt.codfw.wmnet with reboot policy FORCED [production]
09:12 <volans@cumin2002> START - Cookbook sre.hosts.provision for host logstash2037.mgmt.codfw.wmnet with reboot policy FORCED [production]
08:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2130 (T314041)', diff saved to https://phabricator.wikimedia.org/P34942 and previous config saved to /var/cache/conftool/dbconfig/20220927-082023-ladsgroup.json [production]
08:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
08:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
08:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T314041)', diff saved to https://phabricator.wikimedia.org/P34941 and previous config saved to /var/cache/conftool/dbconfig/20220927-082001-ladsgroup.json [production]
08:15 <moritzm> restarting apache/FPM on mw canaries to pick up Expat security updates [production]
08:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P34938 and previous config saved to /var/cache/conftool/dbconfig/20220927-080454-ladsgroup.json [production]
08:00 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.thumbor (exit_code=0) rolling restart_daemons on A:thumbor-eqiad [production]
07:58 <jmm@cumin2002> START - Cookbook sre.misc-clusters.thumbor rolling restart_daemons on A:thumbor-eqiad [production]
07:57 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.thumbor (exit_code=0) rolling restart_daemons on A:thumbor-codfw [production]
07:54 <jmm@cumin2002> START - Cookbook sre.misc-clusters.thumbor rolling restart_daemons on A:thumbor-codfw [production]
07:52 <XioNoX> upgrade python3-pynetbox to 6.6.0 on cumin1001 - T310745 [production]
07:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P34937 and previous config saved to /var/cache/conftool/dbconfig/20220927-074948-ladsgroup.json [production]
07:49 <XioNoX> upgrade python3-pynetbox to 6.6.0 on cumin2002 - T310745 [production]
07:48 <moritzm> installing expat security updates on stretch/buster/bullseye [production]
07:39 <moritzm> uploaded expat 2.2.0-2+deb9u5+wmf1 to apt.wikimedia.org/stretch-wikimedia [production]
07:36 <jayme> published image docker-registry.discovery.wmnet/golang1.18:1.18-1 [production]
07:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1107 (T314041)', diff saved to https://phabricator.wikimedia.org/P34936 and previous config saved to /var/cache/conftool/dbconfig/20220927-073523-ladsgroup.json [production]
07:35 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance [production]
07:34 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance [production]
07:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T314041)', diff saved to https://phabricator.wikimedia.org/P34935 and previous config saved to /var/cache/conftool/dbconfig/20220927-073451-ladsgroup.json [production]
07:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T314041)', diff saved to https://phabricator.wikimedia.org/P34934 and previous config saved to /var/cache/conftool/dbconfig/20220927-073441-ladsgroup.json [production]
07:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34933 and previous config saved to /var/cache/conftool/dbconfig/20220927-071938-ladsgroup.json [production]
07:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34932 and previous config saved to /var/cache/conftool/dbconfig/20220927-070431-ladsgroup.json [production]
06:59 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'show' for AS: 8220 [production]
06:58 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'show' for AS: 8220 [production]
06:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T314041)', diff saved to https://phabricator.wikimedia.org/P34930 and previous config saved to /var/cache/conftool/dbconfig/20220927-064925-ladsgroup.json [production]
05:28 <marostegui> Install 10.6.10 on db1124, db1125, pc1014, pc2014 T318128 [production]
03:57 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
03:51 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
03:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
03:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
03:40 <mwpresync@deploy1002> Pruned MediaWiki: 1.40.0-wmf.1 (duration: 02m 03s) [production]
03:38 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.3 refs T314192 (duration: 36m 01s) [production]
03:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
03:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
03:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
03:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.3 refs T314192 [production]
02:35 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:34 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]