|
2025-04-30
§
|
| 16:28 |
<krinkle@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1138922|missing.php: Redesign to match current error pages (T113114)]], [[gerrit:1138508|missing.php: Check for auth.wikimedia.org domain on missing.php (T391994)]] (duration: 16m 26s) |
[production] |
| 16:27 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P75727 and previous config saved to /var/cache/conftool/dbconfig/20250430-162741-fceratto.json |
[production] |
| 16:23 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 16:23 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 16:22 |
<krinkle@deploy1003> |
krinkle, pppery: Continuing with sync |
[production] |
| 16:18 |
<krinkle@deploy1003> |
krinkle, pppery: Backport for [[gerrit:1138922|missing.php: Redesign to match current error pages (T113114)]], [[gerrit:1138508|missing.php: Check for auth.wikimedia.org domain on missing.php (T391994)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 16:12 |
<arnaudb> |
Gerrit maintenance over |
[production] |
| 16:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210 (T392806)', diff saved to https://phabricator.wikimedia.org/P75726 and previous config saved to /var/cache/conftool/dbconfig/20250430-161234-fceratto.json |
[production] |
| 16:12 |
<krinkle@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1138922|missing.php: Redesign to match current error pages (T113114)]], [[gerrit:1138508|missing.php: Check for auth.wikimedia.org domain on missing.php (T391994)]] |
[production] |
| 16:05 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1210 (T392806)', diff saved to https://phabricator.wikimedia.org/P75725 and previous config saved to /var/cache/conftool/dbconfig/20250430-160556-fceratto.json |
[production] |
| 16:05 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:05 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T392806)', diff saved to https://phabricator.wikimedia.org/P75724 and previous config saved to /var/cache/conftool/dbconfig/20250430-160530-fceratto.json |
[production] |
| 16:02 |
<aikochou@deploy1003> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 15:53 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 15:53 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 15:50 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P75722 and previous config saved to /var/cache/conftool/dbconfig/20250430-155023-fceratto.json |
[production] |
| 15:38 |
<arnaudb@cumin1002> |
END (ERROR) - Cookbook sre.gerrit.failover (exit_code=97) from gerrit2002.wikimedia.org to gerrit1003.wikimedia.org |
[production] |
| 15:35 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P75720 and previous config saved to /var/cache/conftool/dbconfig/20250430-153516-fceratto.json |
[production] |
| 15:29 |
<arnaudb@dns1004> |
END - running authdns-update |
[production] |
| 15:21 |
<mutante> |
gerrit failover in progress |
[production] |
| 15:20 |
<Krinkle> |
Removed @nrayio (former staff [[User:NRay (WMF)]]) from https://www.npmjs.com/settings/wikimedia/members |
[production] |
| 15:20 |
<moritzm> |
installing ucf security updates |
[production] |
| 15:20 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T392806)', diff saved to https://phabricator.wikimedia.org/P75719 and previous config saved to /var/cache/conftool/dbconfig/20250430-152007-fceratto.json |
[production] |
| 15:20 |
<Krinkle> |
Removed @joaquin (former staff) from https://www.npmjs.com/settings/wikimedia/members |
[production] |
| 15:16 |
<arnaudb@cumin1002> |
START - Cookbook sre.gerrit.failover from gerrit2002.wikimedia.org to gerrit1003.wikimedia.org |
[production] |
| 15:15 |
<arnaudb@dns1004> |
START - running authdns-update |
[production] |
| 15:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1200 (T392806)', diff saved to https://phabricator.wikimedia.org/P75718 and previous config saved to /var/cache/conftool/dbconfig/20250430-151222-fceratto.json |
[production] |
| 15:12 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75717 and previous config saved to /var/cache/conftool/dbconfig/20250430-151158-fceratto.json |
[production] |
| 15:01 |
<moritzm> |
installing postgresql-15 security updates |
[production] |
| 14:59 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet — T390514, T392989 |
[production] |
| 14:59 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2207.codfw.wmnet with reason: Maintenance |
[production] |
| 14:58 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P75715 and previous config saved to /var/cache/conftool/dbconfig/20250430-145651-fceratto.json |
[production] |
| 14:55 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 14:55 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 14:54 |
<moritzm> |
installing werkzeug security updates |
[production] |
| 14:50 |
<mszabo@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] (duration: 14m 22s) |
[production] |
| 14:45 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore2004 — T390514, T392989 |
[production] |
| 14:44 |
<sukhe@dns1004> |
END - running authdns-update |
[production] |
| 14:44 |
<mszabo@deploy1003> |
mszabo: Continuing with sync |
[production] |
| 14:42 |
<mszabo@deploy1003> |
mszabo: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:42 |
<sukhe@dns1004> |
START - running authdns-update |
[production] |
| 14:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P75713 and previous config saved to /var/cache/conftool/dbconfig/20250430-144144-fceratto.json |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2218.codfw.wmnet with reason: Maintenance |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<mszabo@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] |
[production] |
| 14:34 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet |
[production] |
| 14:29 |
<moritzm> |
installing ruby2.7 security updates |
[production] |
| 14:28 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet |
[production] |