|
2025-04-30
§
|
| 15:29 |
<arnaudb@dns1004> |
END - running authdns-update |
[production] |
| 15:21 |
<mutante> |
gerrit failover in progress |
[production] |
| 15:20 |
<Krinkle> |
Removed @nrayio (former staff [[User:NRay (WMF)]]) from https://www.npmjs.com/settings/wikimedia/members |
[production] |
| 15:20 |
<moritzm> |
installing ucf security updates |
[production] |
| 15:20 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T392806)', diff saved to https://phabricator.wikimedia.org/P75719 and previous config saved to /var/cache/conftool/dbconfig/20250430-152007-fceratto.json |
[production] |
| 15:20 |
<Krinkle> |
Removed @joaquin (former staff) from https://www.npmjs.com/settings/wikimedia/members |
[production] |
| 15:16 |
<arnaudb@cumin1002> |
START - Cookbook sre.gerrit.failover from gerrit2002.wikimedia.org to gerrit1003.wikimedia.org |
[production] |
| 15:15 |
<arnaudb@dns1004> |
START - running authdns-update |
[production] |
| 15:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1200 (T392806)', diff saved to https://phabricator.wikimedia.org/P75718 and previous config saved to /var/cache/conftool/dbconfig/20250430-151222-fceratto.json |
[production] |
| 15:12 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75717 and previous config saved to /var/cache/conftool/dbconfig/20250430-151158-fceratto.json |
[production] |
| 15:01 |
<moritzm> |
installing postgresql-15 security updates |
[production] |
| 14:59 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet — T390514, T392989 |
[production] |
| 14:59 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2207.codfw.wmnet with reason: Maintenance |
[production] |
| 14:58 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P75715 and previous config saved to /var/cache/conftool/dbconfig/20250430-145651-fceratto.json |
[production] |
| 14:55 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 14:55 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 14:54 |
<moritzm> |
installing werkzeug security updates |
[production] |
| 14:50 |
<mszabo@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] (duration: 14m 22s) |
[production] |
| 14:45 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore2004 — T390514, T392989 |
[production] |
| 14:44 |
<sukhe@dns1004> |
END - running authdns-update |
[production] |
| 14:44 |
<mszabo@deploy1003> |
mszabo: Continuing with sync |
[production] |
| 14:42 |
<mszabo@deploy1003> |
mszabo: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:42 |
<sukhe@dns1004> |
START - running authdns-update |
[production] |
| 14:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P75713 and previous config saved to /var/cache/conftool/dbconfig/20250430-144144-fceratto.json |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2218.codfw.wmnet with reason: Maintenance |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<mszabo@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] |
[production] |
| 14:34 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet |
[production] |
| 14:29 |
<moritzm> |
installing ruby2.7 security updates |
[production] |
| 14:28 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet |
[production] |
| 14:27 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2002.codfw.wmnet |
[production] |
| 14:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75712 and previous config saved to /var/cache/conftool/dbconfig/20250430-142636-fceratto.json |
[production] |
| 14:21 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2002.codfw.wmnet |
[production] |
| 14:20 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2001.codfw.wmnet |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75711 and previous config saved to /var/cache/conftool/dbconfig/20250430-141845-fceratto.json |
[production] |
| 14:18 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183 (T392806)', diff saved to https://phabricator.wikimedia.org/P75710 and previous config saved to /var/cache/conftool/dbconfig/20250430-141819-fceratto.json |
[production] |
| 14:14 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2001.codfw.wmnet |
[production] |
| 14:14 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 14:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 14:11 |
<moritzm> |
failover Ganeti master in codfw to ganeti2021 |
[production] |
| 14:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet |
[production] |
| 14:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet |
[production] |
| 14:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2209.codfw.wmnet with reason: Maintenance |
[production] |
| 14:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P75709 and previous config saved to /var/cache/conftool/dbconfig/20250430-140312-fceratto.json |
[production] |
| 14:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet |
[production] |