|
2025-02-03
ยง
|
| 23:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73111 and previous config saved to /var/cache/conftool/dbconfig/20250203-235947-marostegui.json |
[production] |
| 23:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P73110 and previous config saved to /var/cache/conftool/dbconfig/20250203-234440-marostegui.json |
[production] |
| 23:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P73109 and previous config saved to /var/cache/conftool/dbconfig/20250203-232933-marostegui.json |
[production] |
| 23:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73108 and previous config saved to /var/cache/conftool/dbconfig/20250203-231428-marostegui.json |
[production] |
| 23:10 |
<dwisehaupt@dns1004> |
END - running authdns-update |
[production] |
| 23:09 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:09 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt franio1002 - vriley@cumin1002" |
[production] |
| 23:09 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt franio1002 - vriley@cumin1002" |
[production] |
| 23:08 |
<dwisehaupt@dns1004> |
START - running authdns-update |
[production] |
| 23:01 |
<vriley@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:44 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:44 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt franio1001 - vriley@cumin1002" |
[production] |
| 22:43 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt franio1001 - vriley@cumin1002" |
[production] |
| 22:39 |
<vriley@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:05 |
<AntiComposite> |
Hard reboot cvn-apache10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ |
[cvn] |
| 21:58 |
<AntiComposite> |
Hard reboot cvn-app10 from Horizon per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/ZOCPVXX6BKLC76OHMIQW26YLBCKEBTGQ/ |
[cvn] |
| 21:15 |
<urbanecm@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1116853|[Growth] enwiki: Enable mentorship for 75% of new accounts (T384505)]] (duration: 10m 22s) |
[production] |
| 21:08 |
<urbanecm@deploy2002> |
urbanecm: Continuing with sync |
[production] |
| 21:08 |
<urbanecm@deploy2002> |
urbanecm: Backport for [[gerrit:1116853|[Growth] enwiki: Enable mentorship for 75% of new accounts (T384505)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:04 |
<urbanecm@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1116853|[Growth] enwiki: Enable mentorship for 75% of new accounts (T384505)]] |
[production] |
| 20:48 |
<mutante> |
hard rebooting instance wikistats-bookworm from horizon |
[wikistats] |
| 20:33 |
<rzl@deploy2002> |
Finished scap sync-world: T383952, T384137 (duration: 06m 10s) |
[production] |
| 20:32 |
<rzl@deploy2002> |
rzl: Continuing with sync |
[production] |
| 20:31 |
<rzl@deploy2002> |
rzl: T383952, T384137 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:29 |
<rzl@deploy2002> |
Started scap sync-world: T383952, T384137 |
[production] |
| 19:46 |
<rzl@deploy2002> |
Started scap sync-world: T383952, T384137 |
[production] |
| 19:40 |
<swfrench-wmf> |
ran reprepro include mercurius 1.1.0-1 - T385225 |
[production] |
| 19:36 |
<ejegg> |
fundraising civicrm upgraded from 3e566467 to abe0fc61 |
[production] |
| 19:00 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:00 |
<vriley@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt fransw1001 - vriley@cumin1002" |
[production] |
| 19:00 |
<vriley@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt fransw1001 - vriley@cumin1002" |
[production] |
| 18:55 |
<vriley@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:51 |
<swfrench@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 18:50 |
<swfrench@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-api-int: apply |
[production] |
| 18:50 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1169.eqiad.wmnet onto db1251.eqiad.wmnet |
[production] |
| 18:46 |
<swfrench@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 18:45 |
<swfrench@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-api-int: apply |
[production] |
| 18:41 |
<swfrench-wmf> |
mw-api-int to ~ 1% of traffic on PHP 8.1 in codfw - T383845 |
[production] |
| 18:39 |
<swfrench@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 18:38 |
<swfrench@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-api-int: apply |
[production] |
| 18:28 |
<swfrench-wmf> |
mw-api-int to ~ 1% of traffic on PHP 8.1 in eqiad - T383845 |
[production] |
| 18:27 |
<swfrench@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 18:25 |
<swfrench@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-api-int: apply |
[production] |
| 18:13 |
<swfrench@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1115966|Enroll 10% of client sessions in PHP 8.1 (T383845)]] (duration: 11m 13s) |
[production] |
| 18:07 |
<swfrench@deploy2002> |
swfrench: Continuing with sync |
[production] |
| 18:06 |
<swfrench@deploy2002> |
swfrench: Backport for [[gerrit:1115966|Enroll 10% of client sessions in PHP 8.1 (T383845)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 18:02 |
<swfrench@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1115966|Enroll 10% of client sessions in PHP 8.1 (T383845)]] |
[production] |
| 18:01 |
<urbanecm> |
[urbanecm@deploy2002 ~]$ mwscript-k8s -f -- extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=newiki --logwiki=metawiki 'Tarasssst' 'TR101' # T385503 |
[production] |
| 17:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73107 and previous config saved to /var/cache/conftool/dbconfig/20250203-175904-marostegui.json |
[production] |
| 17:58 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1238.eqiad.wmnet with reason: Maintenance |
[production] |