|
2025-03-17
§
|
| 10:05 |
<marostegui@cumin1002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section es7 |
[production] |
| 10:02 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section es6 |
[production] |
| 10:01 |
<marostegui@cumin1002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section es6 |
[production] |
| 10:00 |
<marostegui@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] (duration: 23m 25s) |
[production] |
| 09:56 |
<vgutierrez@cumin1002> |
END (FAIL) - Cookbook sre.loadbalancer.upgrade (exit_code=1) upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica |
[production] |
| 09:55 |
<vgutierrez@cumin1002> |
START - Cookbook sre.loadbalancer.upgrade upgradeing P{lvs4010.ulsfo.wmnet} and A:liberica |
[production] |
| 09:50 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 09:50 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 09:45 |
<vgutierrez@cumin1002> |
START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS bookworm |
[production] |
| 09:37 |
<marostegui@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] |
[production] |
| 09:25 |
<moritzm> |
installing intel-microcode security updates |
[production] |
| 09:19 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
| 09:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet |
[production] |
| 08:46 |
<moritzm> |
freed 28G of disk space on maps1009 |
[production] |
| 08:30 |
<moritzm> |
updated bookworm installer image to Bookworm 12.10 T389034 |
[production] |
| 08:23 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync |
[production] |
| 08:23 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync |
[production] |
| 08:23 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop: sync |
[production] |
| 08:23 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/changeprop: sync |
[production] |
| 07:29 |
<marostegui@cumin2002> |
END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 |
[production] |
| 07:28 |
<marostegui@cumin2002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 |
[production] |
| 07:17 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 |
[production] |
| 07:17 |
<marostegui@cumin1002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 |
[production] |
| 07:16 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 |
[production] |
| 07:15 |
<marostegui@cumin1002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 |
[production] |
| 07:14 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 |
[production] |
| 07:13 |
<marostegui@cumin1002> |
START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 |
[production] |
|
2025-03-15
§
|
| 18:11 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 18:00 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 15:31 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 (T388965) |
[tools] |
| 15:14 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 (T388965) |
[tools] |
| 15:14 |
<wmbot~dcaro@urcuchillay> |
END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 (T388965) |
[tools] |
| 15:14 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 (T388965) |
[tools] |
| 12:55 |
<dcaro> |
there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit T388965 |
[tools] |
| 11:22 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed ab6621b22d (l10n updates: ar) |
[tools.lexeme-forms] |
| 00:51 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2046 |
[production] |
| 00:51 |
<jhancock@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host ganeti2046 |
[production] |