|
2021-11-11
§
|
| 11:31 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests |
[production] |
| 11:28 |
<jynus@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1001.eqiad.wmnet with OS buster |
[production] |
| 11:04 |
<jynus@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbprov1001.eqiad.wmnet with OS buster |
[production] |
| 10:56 |
<jynus@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2001.codfw.wmnet with OS buster |
[production] |
| 10:50 |
<arturo> |
add user `srv-networktests` as project user (T294955) |
[cloudinfra] |
| 10:50 |
<arturo> |
add user `srv-networktests` as project user (T294955) |
[tools] |
| 10:47 |
<arturo> |
add user `srv-networktests` as project user (T294955) |
[bastion] |
| 10:37 |
<moritzm> |
updated routinator in thirdparty/routinator for bullseye-wikimedia to 0.10.12 T292503 |
[production] |
| 10:24 |
<jynus@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbprov2001.codfw.wmnet with OS buster |
[production] |
| 10:18 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS buster |
[production] |
| 10:15 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests |
[production] |
| 10:15 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1004.wikimedia.org with reason: working on network tests |
[production] |
| 10:15 |
<vgutierrez> |
pool cp3065 running haproxy - T290005 |
[production] |
| 09:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove contributions from s5 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17725 and previous config saved to /var/cache/conftool/dbconfig/20211111-092528-marostegui.json |
[production] |
| 09:13 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS buster |
[production] |
| 09:10 |
<vgutierrez> |
depool cp3065 to be reimaged as cache::upload_haproxy - T290005 |
[production] |
| 09:03 |
<arturo> |
pull all packages for buster-wikimedia/thirdparty/kubeadm-k8s-1-21 (T282942) |
[production] |
| 08:35 |
<majavah> |
disabling pod preset controller in preparation for T291913 |
[paws] |
| 08:17 |
<marostegui> |
Upgrade db2078 T288720 |
[production] |
| 08:13 |
<marostegui> |
Restart db1132 T288720 |
[production] |
| 06:56 |
<elukey> |
`systemctl start prometheus-mysqld-exporter@analytics_meta` on db1108 |
[production] |
| 06:56 |
<elukey> |
`systemctl start prometheus-mysqld-exporter@analytics_meta` on db1108 |
[analytics] |
| 06:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1104.eqiad.wmnet with OS buster |
[production] |
| 06:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1104.eqiad.wmnet with OS buster |
[production] |
| 06:06 |
<marostegui> |
Stop replication on db1104 (old master) T294321 |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1104 (old master) T294321', diff saved to https://phabricator.wikimedia.org/P17723 and previous config saved to /var/cache/conftool/dbconfig/20211111-060242-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1109 to s8 primary and set section read-write T294321', diff saved to https://phabricator.wikimedia.org/P17722 and previous config saved to /var/cache/conftool/dbconfig/20211111-060102-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T294321', diff saved to https://phabricator.wikimedia.org/P17721 and previous config saved to /var/cache/conftool/dbconfig/20211111-060031-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting s8 eqiad failover from db1104 to db1109 - T294321 |
[production] |
| 05:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 |
[production] |
| 05:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 |
[production] |
| 02:52 |
<eileen> |
civicrm revision 7e38867f -> 817e514a (latest) |
[production] |
| 00:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:18 |
<reedy@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Set wgForeignUploadTargets on officewiki T295510 (duration: 00m 56s) |
[production] |
|
2021-11-10
§
|
| 23:46 |
<ebernhardson> |
start test backup/restore of 1tb commonswiki from relforge to swift in eqiad |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=DoubleRedirects |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=BrokenRedirects |
[production] |
| 22:06 |
<bblack> |
dns2002 - restart ntp.servce to fix drmrs peering |
[production] |
| 22:01 |
<bblack> |
dns1002 - restart ntp.servce to fix drmrs peering |
[production] |
| 21:56 |
<bblack> |
dns2001 - restart ntp.service to fix drmrs peering |
[production] |
| 21:53 |
<bblack> |
dns1001 - restart ntp.service to see if drmrs associations cleared up after dns changes, etc |
[production] |
| 21:24 |
<bblack> |
asw1-b1[23]-drmrs: added ipv6 router-advertisement clauses, which work, but probably imperfectly :) |
[production] |
| 19:52 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS buster |
[production] |
| 19:51 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6002.wikimedia.org with OS buster |
[production] |
| 19:51 |
<ottomata> |
altering {eqiad,codfw}.maps.tiles_change to increase to 6 partitions in kafka main-eqiad, main-codfw and jumbo-eqiad: https://phabricator.wikimedia.org/T293366#7497076 |
[production] |
| 19:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:49 |
<mutante> |
- removing manually added things in Horizon Hiera that were already in the repo, please don't keep adding in web UI, we don't want to repeat the same thing we did in deployment-prep |
[devtools] |
| 19:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:43 |
<cjming> |
end of UTC evening backport & config window |
[production] |