|
2024-03-19
§
|
| 14:42 |
<effie> |
Traffic+Services switchover complete, codfw is depooled - Τ357547 |
[production] |
| 14:40 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:36 |
<wmbot~h2o@tools-sgebastion-10> |
./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC |
[tools.stewardbots] |
| 14:22 |
<effie> |
depooling services from codfw - T357547 |
[production] |
| 14:16 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:07 |
<effie> |
Completely depool codfw from user traffic - T357547 |
[production] |
| 13:55 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-info.json --execute --throttle 50000000 T326419 |
[production] |
| 13:50 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file mediawiki.httpd.accesslog-sampled.json --execute --throttle 50000000 T326419 |
[production] |
| 13:45 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<hashar> |
Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1011303 | T357986 |
[releng] |
| 13:41 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:41 |
<claime> |
Deploying changeprop and changeprop-jobqueue - T353876 |
[production] |
| 13:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:30 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.ceph.osd.drain_node (T348643) |
[admin] |
| 13:19 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 13:08 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1024.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 13:08 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1024.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 13:08 |
<hashar> |
Deleted `language-screenshots-VisualEditor` Jenkins job # T360425 |
[releng] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:06 |
<claime> |
manually adding 20 replicas to mw-parsoid to help with big reparse |
[production] |
| 12:59 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox |
[production] |
| 12:59 |
<cmooney@cumin1002> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox |
[production] |
| 12:59 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary |
[production] |
| 12:49 |
<wmbot~taavi@tools-bastion-12> |
toolforge jobs restart irc |
[tools.wikibugs] |
| 12:49 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 12:49 |
<cmooney@cumin1002> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 12:48 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 12:48 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 12:48 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 12:11 |
<slyngs> |
Switch idp-test to upgraded Bookworm host |
[production] |
| 11:45 |
<cgoubert@deploy2002> |
Finished scap: mediawiki: Route /w/CREDITS and /w/COPYING to /w/static.php - gerrit:1012627 - T359643 (duration: 06m 59s) |
[production] |
| 11:40 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 11:39 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 11:22 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 |
[tools] |
| 11:21 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 11:21 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 |
[tools] |
| 11:21 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 11:19 |
<oblivian@deploy2002> |
dancy and oblivian: Continuing with sync |
[production] |
| 11:19 |
<taavi> |
point dev.toolforge.org to tools-bastion-12 T314665 |
[tools] |
| 11:18 |
<oblivian@deploy2002> |
dancy and oblivian: Backport for [[gerrit:1012427|static.php: Handle COPYING and CREDITS files (T359643)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |