|
2022-11-30
ยง
|
| 22:09 |
<pt1979@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2001'] |
[production] |
| 22:09 |
<mutante> |
registry2004 - apt-get update; apt-install jwt-authorizer (after apt1001: sudo -i reprepro --ignore=wrongdistribution -C main include buster-wikimedia /home/jelto/jwt-authorizer_1.1.0-1_amd64.changes) - T309646 |
[production] |
| 22:09 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 22:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P41942 and previous config saved to /var/cache/conftool/dbconfig/20221130-220832-ladsgroup.json |
[production] |
| 22:05 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 22:02 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 22:00 |
<cjming@deploy1002> |
Finished scap: Backport for [[gerrit:862340|CommonSettings: make Ext:Phonos use $wgCopyUploadProxy (T321084)]] (duration: 06m 08s) |
[production] |
| 21:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P41941 and previous config saved to /var/cache/conftool/dbconfig/20221130-215937-ladsgroup.json |
[production] |
| 21:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P41940 and previous config saved to /var/cache/conftool/dbconfig/20221130-215653-ladsgroup.json |
[production] |
| 21:55 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 21:55 |
<cjming@deploy1002> |
cjming and musikanimal: Backport for [[gerrit:862340|CommonSettings: make Ext:Phonos use $wgCopyUploadProxy (T321084)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 21:54 |
<cjming@deploy1002> |
Started scap: Backport for [[gerrit:862340|CommonSettings: make Ext:Phonos use $wgCopyUploadProxy (T321084)]] |
[production] |
| 21:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130 (T322618)', diff saved to https://phabricator.wikimedia.org/P41939 and previous config saved to /var/cache/conftool/dbconfig/20221130-215326-ladsgroup.json |
[production] |
| 21:51 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 21:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2130 (T322618)', diff saved to https://phabricator.wikimedia.org/P41938 and previous config saved to /var/cache/conftool/dbconfig/20221130-215115-ladsgroup.json |
[production] |
| 21:51 |
<robh@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs5004'] |
[production] |
| 21:51 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 21:51 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 21:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T322618)', diff saved to https://phabricator.wikimedia.org/P41937 and previous config saved to /var/cache/conftool/dbconfig/20221130-215054-ladsgroup.json |
[production] |
| 21:50 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs5004.mgmt.eqsin.wmnet with reboot policy FORCED |
[production] |
| 21:47 |
<brett@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS buster |
[production] |
| 21:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P41936 and previous config saved to /var/cache/conftool/dbconfig/20221130-214431-ladsgroup.json |
[production] |
| 21:41 |
<mutante> |
phabricator - deactivating diffusion repo wikisp-mw-config T191182 |
[production] |
| 21:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2175 (T323907)', diff saved to https://phabricator.wikimedia.org/P41935 and previous config saved to /var/cache/conftool/dbconfig/20221130-214147-ladsgroup.json |
[production] |
| 21:40 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 21:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P41934 and previous config saved to /var/cache/conftool/dbconfig/20221130-213547-ladsgroup.json |
[production] |
| 21:35 |
<dancy@deploy1002> |
Finished scap: Update PrivateSettings.php for T315491 (duration: 05m 40s) |
[production] |
| 21:34 |
<robh@cumin2002> |
START - Cookbook sre.hosts.provision for host lvs5004.mgmt.eqsin.wmnet with reboot policy FORCED |
[production] |
| 21:33 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:32 |
<robh@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:30 |
<dancy@deploy1002> |
Started scap: Update PrivateSettings.php for T315491 |
[production] |
| 21:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106 (T322618)', diff saved to https://phabricator.wikimedia.org/P41933 and previous config saved to /var/cache/conftool/dbconfig/20221130-212924-ladsgroup.json |
[production] |
| 21:27 |
<cjming@deploy1002> |
Finished scap: Backport for [[gerrit:861926|Enable Phonos on testwiki (T321084)]] (duration: 23m 45s) |
[production] |
| 21:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P41932 and previous config saved to /var/cache/conftool/dbconfig/20221130-212041-ladsgroup.json |
[production] |
| 21:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1106 (T322618)', diff saved to https://phabricator.wikimedia.org/P41931 and previous config saved to /var/cache/conftool/dbconfig/20221130-211615-ladsgroup.json |
[production] |
| 21:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:15 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:15 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T322618)', diff saved to https://phabricator.wikimedia.org/P41930 and previous config saved to /var/cache/conftool/dbconfig/20221130-211536-ladsgroup.json |
[production] |
| 21:15 |
<brett@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage |
[production] |
| 21:11 |
<brett@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage |
[production] |
| 21:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T322618)', diff saved to https://phabricator.wikimedia.org/P41929 and previous config saved to /var/cache/conftool/dbconfig/20221130-210534-ladsgroup.json |
[production] |
| 21:04 |
<cjming@deploy1002> |
cjming and musikanimal: Backport for [[gerrit:861926|Enable Phonos on testwiki (T321084)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 21:03 |
<cjming@deploy1002> |
Started scap: Backport for [[gerrit:861926|Enable Phonos on testwiki (T321084)]] |
[production] |
| 21:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2116 (T322618)', diff saved to https://phabricator.wikimedia.org/P41928 and previous config saved to /var/cache/conftool/dbconfig/20221130-210324-ladsgroup.json |
[production] |
| 21:03 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 21:03 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 21:03 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2103 (T322618)', diff saved to https://phabricator.wikimedia.org/P41927 and previous config saved to /var/cache/conftool/dbconfig/20221130-210302-ladsgroup.json |
[production] |
| 21:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2175 (T323907)', diff saved to https://phabricator.wikimedia.org/P41926 and previous config saved to /var/cache/conftool/dbconfig/20221130-210234-ladsgroup.json |
[production] |