1601-1650 of 10000 results (61ms)
2022-06-30 ยง
14:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2160.codfw.wmnet with reason: host reimage [production]
14:38 <_joe_> updating python-poolcounter to 0.0.2 across the fleet [production]
14:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1128 (T309311)', diff saved to https://phabricator.wikimedia.org/P30666 and previous config saved to /var/cache/conftool/dbconfig/20220630-143436-ladsgroup.json [production]
14:34 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2160.codfw.wmnet with reason: host reimage [production]
14:29 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P30665 and previous config saved to /var/cache/conftool/dbconfig/20220630-142519-ladsgroup.json [production]
14:25 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
14:23 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:22 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:20 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:14 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host db2160.codfw.wmnet with OS bullseye [production]
14:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P30664 and previous config saved to /var/cache/conftool/dbconfig/20220630-141014-ladsgroup.json [production]
14:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2154.codfw.wmnet with OS bullseye [production]
13:55 <moritzm> installing firejail security updates on stretch [production]
13:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P30663 and previous config saved to /var/cache/conftool/dbconfig/20220630-135509-ladsgroup.json [production]
13:52 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: host reimage [production]
13:48 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host stat1009.eqiad.wmnet with OS bullseye [production]
13:48 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2154.codfw.wmnet with reason: host reimage [production]
13:47 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host db2154.codfw.wmnet with OS bullseye [production]
13:40 <Amir1> killed refreshLinkRecommendations.php on arzwiki (T299021) [production]
13:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2003.codfw.wmnet [production]
13:39 <urbanecm> [urbanecm@mwmaint1002 /srv/mediawiki/php]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=metawiki --userlist /home/urbanecm/users.txt # T311012, users.txt has `New user message` only [production]
13:39 <urbanecm> run `CentralAuthUser::importLocalNames` for `New user message` (T311012) [production]
13:38 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
13:38 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
13:38 <bmansurov@deploy1002> deploy aborted: (no justification provided) (duration: 00m 03s) [production]
13:38 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
13:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P30662 and previous config saved to /var/cache/conftool/dbconfig/20220630-133743-ladsgroup.json [production]
13:37 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
13:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
13:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1128 (T309311)', diff saved to https://phabricator.wikimedia.org/P30661 and previous config saved to /var/cache/conftool/dbconfig/20220630-133619-ladsgroup.json [production]
13:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
13:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1128.eqiad.wmnet with reason: Maintenance [production]
13:36 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host poolcounter2003.codfw.wmnet [production]
13:35 <urbanecm> [urbanecm@mwmaint1002 /srv/mediawiki/php]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=rowiki --userlist /home/urbanecm/users.txt # T275935, users.txt has `MediaWiki message delivery` only [production]
13:35 <urbanecm> run `CentralAuthUser::importLocalNames` for `MediaWiki message delivery` (T275935) [production]
13:35 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2004.codfw.wmnet [production]
13:34 <slyngshede@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet [production]
13:34 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host stat1009.eqiad.wmnet with OS bullseye [production]
13:34 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2155.codfw.wmnet with OS bullseye [production]
13:33 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host stat1009.eqiad.wmnet with OS buster [production]
13:32 <urbanecm> [urbanecm@mwmaint1002 /srv/mediawiki/php]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=rowiki --userlist /home/urbanecm/users.txt # T311012, users.txt has `New user message` only [production]
13:31 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host poolcounter2004.codfw.wmnet [production]
13:29 <slyngshede@cumin1001> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
13:28 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host stat1009.eqiad.wmnet with OS buster [production]
13:26 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
13:26 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
13:25 <urbanecm@deploy1002> Synchronized php-1.39.0-wmf.18/skins/Vector/resources/skins.vector.styles/layouts/screen.less: a927e6fbf56f031c42737cd9710eb0531bab43e1: Fixes Content sub unreadable in Vector 22 (T311564) (duration: 03m 18s) [production]
13:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:25 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]