|
2024-03-19
§
|
| 15:36 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-host for host db2098.codfw.wmnet |
[production] |
| 15:35 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-warning.json --execute --throttle 50000000 T326419 |
[production] |
| 15:32 |
<brennen@deploy2002> |
Finished deploy [phabricator/deployment@9617e09]: deploy to phab1004 for T358610 (duration: 01m 19s) |
[production] |
| 15:31 |
<brennen@deploy2002> |
Started deploy [phabricator/deployment@9617e09]: deploy to phab1004 for T358610 |
[production] |
| 15:31 |
<brennen@deploy2002> |
Finished deploy [phabricator/deployment@9617e09]: deploy to phab2002 for T358610 (duration: 00m 23s) |
[production] |
| 15:30 |
<brennen@deploy2002> |
Started deploy [phabricator/deployment@9617e09]: deploy to phab2002 for T358610 |
[production] |
| 15:30 |
<brennen> |
starting phabricator/phorge update (T358610) |
[production] |
| 15:30 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:29 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:29 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:29 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:26 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:25 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:23 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:22 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 15:20 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc::db_inventory |
[production] |
| 15:20 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 15:19 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 15:19 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-web: apply |
[production] |
| 15:18 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-web: apply |
[production] |
| 15:17 |
<fabfur@cumin1002> |
conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 15:17 |
<claime> |
Raising mw-web and mw-api-ext replicas for additional read-only traffic - T357547 |
[production] |
| 15:15 |
<fabfur@cumin1002> |
conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet |
[production] |
| 15:14 |
<fabfur> |
repooling cp4037 for brief time (T358109) |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-notice.json --execute --throttle 50000000 T326419 |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 15:05 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: mariadb::misc::db_inventory |
[production] |
| 15:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc::phabricator |
[production] |
| 15:00 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:00 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:00 |
<effie> |
restart kartotherian on eqiad |
[production] |
| 14:58 |
<effie> |
pooling kartotherian on codfw back |
[production] |
| 14:57 |
<jiji@cumin1002> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 14:57 |
<jiji@cumin1002> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad |
[production] |
| 14:55 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-role for role: mariadb::misc::phabricator |
[production] |
| 14:42 |
<effie> |
Traffic+Services switchover complete, codfw is depooled - Τ357547 |
[production] |
| 14:40 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:22 |
<effie> |
depooling services from codfw - T357547 |
[production] |
| 14:16 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter depool all services in codfw: Northward DC Switchover, March 2024 - T357547 |
[production] |
| 14:07 |
<effie> |
Completely depool codfw from user traffic - T357547 |
[production] |
| 13:55 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file rsyslog-info.json --execute --throttle 50000000 T326419 |
[production] |
| 13:50 |
<herron> |
kafka-logging1001:~# kafka reassign-partitions -reassignment-json-file mediawiki.httpd.accesslog-sampled.json --execute --throttle 50000000 T326419 |
[production] |
| 13:45 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 13:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 13:42 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |