|
2024-03-19
§
|
| 15:17 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 15:17 |
<claime> |
Raising mw-web and mw-api-ext replicas for additional read-only traffic - T357547 |
[production] |
| 15:15 |
<fabfur@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 15:14 |
<fabfur> |
repooling cp4037 for brief time (T358109) |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-notice.json --execute --throttle 50000000 T326419 |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:05 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: mariadb::misc::db_inventory |
[production] |
| 15:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc::phabricator |
[production] |
| 15:00 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:00 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:00 |
<effie> |
restart kartotherian on eqiad |
[production] |
| 14:58 |
<effie> |
pooling kartotherian on codfw back |
[production] |
| 14:57 |
<jiji@cumin1002> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 14:57 |
<jiji@cumin1002> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad |
[production] |
| 14:55 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: mariadb::misc::phabricator |
[production] |
| 14:42 |
<effie> |
Traffic+Services switchover complete, codfw is depooled - Τ357547 |
[production] |
| 14:40 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:22 |
<effie> |
depooling services from codfw - T357547 |
[production] |
| 14:16 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:07 |
<effie> |
Completely depool codfw from user traffic - T357547 |
[production] |
| 13:55 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-info.json --execute --throttle 50000000 T326419 |
[production] |
| 13:50 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file mediawiki.httpd.accesslog-sampled.json --execute --throttle 50000000 T326419 |
[production] |
| 13:45 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:41 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:41 |
<claime> |
Deploying changeprop and changeprop-jobqueue - T353876 |
[production] |
| 13:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:19 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 13:08 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1024.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 13:08 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1024.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 13:06 |
<claime> |
manually adding 20 replicas to mw-parsoid to help with big reparse |
[production] |
| 12:59 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox |
[production] |
| 12:59 |
<cmooney@cumin1002> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox |
[production] |
| 12:59 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary |
[production] |
| 12:49 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 12:49 |
<cmooney@cumin1002> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 12:48 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |