301-350 of 10000 results (69ms)
2023-02-14 ยง
21:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2128 (T328255)', diff saved to https://phabricator.wikimedia.org/P44639 and previous config saved to /var/cache/conftool/dbconfig/20230214-210102-ladsgroup.json [production]
20:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2128 (T328255)', diff saved to https://phabricator.wikimedia.org/P44638 and previous config saved to /var/cache/conftool/dbconfig/20230214-205709-ladsgroup.json [production]
20:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328255)', diff saved to https://phabricator.wikimedia.org/P44637 and previous config saved to /var/cache/conftool/dbconfig/20230214-205633-ladsgroup.json [production]
20:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P44636 and previous config saved to /var/cache/conftool/dbconfig/20230214-204126-ladsgroup.json [production]
20:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P44635 and previous config saved to /var/cache/conftool/dbconfig/20230214-202620-ladsgroup.json [production]
20:24 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1005.eqiad.wmnet with reason: host reimage [production]
20:21 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1005.eqiad.wmnet with reason: host reimage [production]
20:21 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dns4004.wikimedia.org with reason: failure during reimaging [production]
20:21 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dns4004.wikimedia.org with reason: failure during reimaging [production]
20:20 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns4004.wikimedia.org with OS buster [production]
20:12 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.23 refs T325586 [production]
20:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328255)', diff saved to https://phabricator.wikimedia.org/P44634 and previous config saved to /var/cache/conftool/dbconfig/20230214-201114-ladsgroup.json [production]
20:09 <bking@cumin1001> START - Cookbook sre.ganeti.reimage for host an-airflow1005.eqiad.wmnet with OS bullseye [production]
20:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2123 (T328255)', diff saved to https://phabricator.wikimedia.org/P44633 and previous config saved to /var/cache/conftool/dbconfig/20230214-200822-ladsgroup.json [production]
20:08 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
20:08 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
20:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328255)', diff saved to https://phabricator.wikimedia.org/P44632 and previous config saved to /var/cache/conftool/dbconfig/20230214-200801-ladsgroup.json [production]
19:55 <AndyRussG> update SmashPig 683df497 -> c6775c60 [production]
19:53 <dduvall@deploy1002> Pruned MediaWiki: 1.40.0-wmf.21 (duration: 02m 10s) [production]
19:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P44631 and previous config saved to /var/cache/conftool/dbconfig/20230214-195255-ladsgroup.json [production]
19:50 <dduvall@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.23 refs T325586 (duration: 09m 14s) [production]
19:46 <gehel@puppetmaster1001> conftool action : set/pooled=yes; selector: name=elastic2069.codfw.wmnet,service=elasticsearch-psi-ssl [production]
19:46 <cdanis@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host aux-k8s-ctrl1002.eqiad.wmnet with OS bullseye [production]
19:46 <gehel@puppetmaster1001> conftool action : set/weight=10; selector: name=elastic2069.codfw.wmnet,service=elasticsearch-psi-ssl [production]
19:45 <gehel@puppetmaster1001> conftool action : set/weight=10; selector: name=elastic2069.cofdw.wmnet,service=elasticsearch-psi-ssl [production]
19:43 <gehel@puppetmaster1001> conftool action : set/weight=10; selector: name=elastic2069.cofdw.wmnet [production]
19:43 <gehel@puppetmaster1001> conftool action : set/pooled=active; selector: name=elastic2069.cofdw.wmnet [production]
19:41 <dduvall@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.23 refs T325586 [production]
19:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P44630 and previous config saved to /var/cache/conftool/dbconfig/20230214-193748-ladsgroup.json [production]
19:37 <dduvall> did not run `docker system prune` due to objections [production]
19:36 <mutante> root@deploy1002:/srv# rm -rf deployment.T307349/ [production]
19:34 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mc-gp1003'] [production]
19:34 <cdanis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-ctrl1002.eqiad.wmnet with reason: host reimage [production]
19:33 <dduvall> running `docker system prune` on deploy1002 to free up disk space on /srv [production]
19:31 <dduvall> scap sync-world failed due to lack of disk space on deploy1002 /srv (cc T325586) [production]
19:31 <cdanis@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-ctrl1002.eqiad.wmnet with reason: host reimage [production]
19:27 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mc-gp1003'] [production]
19:25 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mc-gp1003'] [production]
19:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328255)', diff saved to https://phabricator.wikimedia.org/P44629 and previous config saved to /var/cache/conftool/dbconfig/20230214-192242-ladsgroup.json [production]
19:22 <cdanis@cumin1001> START - Cookbook sre.ganeti.reimage for host aux-k8s-ctrl1002.eqiad.wmnet with OS bullseye [production]
19:21 <cdanis@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host aux-k8s-ctrl1001.eqiad.wmnet with OS bullseye [production]
19:17 <papaul> upgrading firmware on mc-gp1003 [production]
19:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2111 (T328255)', diff saved to https://phabricator.wikimedia.org/P44628 and previous config saved to /var/cache/conftool/dbconfig/20230214-191550-ladsgroup.json [production]
19:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
19:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
19:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]