3101-3150 of 10000 results (86ms)
2023-05-04 §
17:26 <eoghan@cumin1001> START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet [production]
17:25 <cmooney@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye [production]
17:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1147 (T335838)', diff saved to https://phabricator.wikimedia.org/P47630 and previous config saved to /var/cache/conftool/dbconfig/20230504-172546-ladsgroup.json [production]
17:25 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance [production]
17:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P47629 and previous config saved to /var/cache/conftool/dbconfig/20230504-172534-ladsgroup.json [production]
17:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1147.eqiad.wmnet with reason: Maintenance [production]
17:25 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2047.codfw.wmnet [production]
17:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T335838)', diff saved to https://phabricator.wikimedia.org/P47628 and previous config saved to /var/cache/conftool/dbconfig/20230504-172523-ladsgroup.json [production]
17:24 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1047.eqiad.wmnet [production]
17:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1200 (T335845)', diff saved to https://phabricator.wikimedia.org/P47627 and previous config saved to /var/cache/conftool/dbconfig/20230504-172228-ladsgroup.json [production]
17:22 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs10[11-21].eqiad.wmnet: Upgrade Cassandra — T335383 - eevans@cumin1001 [production]
17:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
17:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
17:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T335845)', diff saved to https://phabricator.wikimedia.org/P47626 and previous config saved to /var/cache/conftool/dbconfig/20230504-172204-ladsgroup.json [production]
17:16 <mutante> aphlict2001 - not active, rebooting [production]
17:15 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2046.codfw.wmnet [production]
17:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P47625 and previous config saved to /var/cache/conftool/dbconfig/20230504-171300-ladsgroup.json [production]
17:11 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1046.eqiad.wmnet [production]
17:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P47624 and previous config saved to /var/cache/conftool/dbconfig/20230504-171028-ladsgroup.json [production]
17:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P47623 and previous config saved to /var/cache/conftool/dbconfig/20230504-171017-ladsgroup.json [production]
17:09 <brennen> phab1004 deployed and restarted, phab up, MR widget still seems to work [production]
17:08 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2046.codfw.wmnet [production]
17:08 <brennen@deploy1002> Finished deploy [phabricator/deployment@0529926]: deploy latest state to phab1004 (duration: 00m 34s) [production]
17:07 <brennen@deploy1002> Started deploy [phabricator/deployment@0529926]: deploy latest state to phab1004 [production]
17:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P47622 and previous config saved to /var/cache/conftool/dbconfig/20230504-170658-ladsgroup.json [production]
17:05 <brennen@deploy1002> Finished deploy [phabricator/deployment@0529926]: deploy latest state to phab2002 (duration: 00m 37s) [production]
17:05 <brennen@deploy1002> Started deploy [phabricator/deployment@0529926]: deploy latest state to phab2002 [production]
17:04 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1046.eqiad.wmnet [production]
17:03 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: maintenance upgrade [production]
17:03 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: maintenance upgrade [production]
17:02 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: maintenance upgrade [production]
17:02 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: maintenance upgrade [production]
17:01 <mutante> Phabricator upgrade - maintenance incoming [production]
17:00 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs2011.codfw.wmnet [production]
16:58 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2045.codfw.wmnet [production]
16:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P47621 and previous config saved to /var/cache/conftool/dbconfig/20230504-165753-ladsgroup.json [production]
16:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T335838)', diff saved to https://phabricator.wikimedia.org/P47620 and previous config saved to /var/cache/conftool/dbconfig/20230504-165521-ladsgroup.json [production]
16:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P47619 and previous config saved to /var/cache/conftool/dbconfig/20230504-165511-ladsgroup.json [production]
16:52 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc2045.codfw.wmnet [production]
16:52 <sbassett@deploy1002> Finished scap: Backport for [[gerrit:914823|Re-enable the Graph extension on test2wiki (T334940)]] (duration: 07m 04s) [production]
16:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P47618 and previous config saved to /var/cache/conftool/dbconfig/20230504-165152-ladsgroup.json [production]
16:51 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1045.eqiad.wmnet [production]
16:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2179 (T335838)', diff saved to https://phabricator.wikimedia.org/P47617 and previous config saved to /var/cache/conftool/dbconfig/20230504-164850-ladsgroup.json [production]
16:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
16:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
16:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T335838)', diff saved to https://phabricator.wikimedia.org/P47616 and previous config saved to /var/cache/conftool/dbconfig/20230504-164826-ladsgroup.json [production]
16:46 <sbassett@deploy1002> sbassett: Backport for [[gerrit:914823|Re-enable the Graph extension on test2wiki (T334940)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
16:46 <jiji@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc1045.eqiad.wmnet [production]
16:45 <hnowlan@puppetmaster1001> conftool action : set/weight=5; selector: service=thumbor,name=thumbor100[1256].eqiad.wmnet [production]
16:44 <sbassett@deploy1002> Started scap: Backport for [[gerrit:914823|Re-enable the Graph extension on test2wiki (T334940)]] [production]