|
2021-11-11
§
|
| 06:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1104.eqiad.wmnet with OS buster |
[production] |
| 06:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1104.eqiad.wmnet with OS buster |
[production] |
| 06:06 |
<marostegui> |
Stop replication on db1104 (old master) T294321 |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1104 (old master) T294321', diff saved to https://phabricator.wikimedia.org/P17723 and previous config saved to /var/cache/conftool/dbconfig/20211111-060242-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1109 to s8 primary and set section read-write T294321', diff saved to https://phabricator.wikimedia.org/P17722 and previous config saved to /var/cache/conftool/dbconfig/20211111-060102-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T294321', diff saved to https://phabricator.wikimedia.org/P17721 and previous config saved to /var/cache/conftool/dbconfig/20211111-060031-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting s8 eqiad failover from db1104 to db1109 - T294321 |
[production] |
| 05:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 |
[production] |
| 05:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 31 hosts with reason: Primary switchover s8 T294321 |
[production] |
| 02:52 |
<eileen> |
civicrm revision 7e38867f -> 817e514a (latest) |
[production] |
| 00:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:18 |
<reedy@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Set wgForeignUploadTargets on officewiki T295510 (duration: 00m 56s) |
[production] |
|
2021-11-10
§
|
| 23:46 |
<ebernhardson> |
start test backup/restore of 1tb commonswiki from relforge to swift in eqiad |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=DoubleRedirects |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=BrokenRedirects |
[production] |
| 22:06 |
<bblack> |
dns2002 - restart ntp.servce to fix drmrs peering |
[production] |
| 22:01 |
<bblack> |
dns1002 - restart ntp.servce to fix drmrs peering |
[production] |
| 21:56 |
<bblack> |
dns2001 - restart ntp.service to fix drmrs peering |
[production] |
| 21:53 |
<bblack> |
dns1001 - restart ntp.service to see if drmrs associations cleared up after dns changes, etc |
[production] |
| 21:24 |
<bblack> |
asw1-b1[23]-drmrs: added ipv6 router-advertisement clauses, which work, but probably imperfectly :) |
[production] |
| 19:52 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS buster |
[production] |
| 19:51 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6002.wikimedia.org with OS buster |
[production] |
| 19:51 |
<ottomata> |
altering {eqiad,codfw}.maps.tiles_change to increase to 6 partitions in kafka main-eqiad, main-codfw and jumbo-eqiad: https://phabricator.wikimedia.org/T293366#7497076 |
[production] |
| 19:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:49 |
<mutante> |
- removing manually added things in Horizon Hiera that were already in the repo, please don't keep adding in web UI, we don't want to repeat the same thing we did in deployment-prep |
[devtools] |
| 19:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:43 |
<cjming> |
end of UTC evening backport & config window |
[production] |
| 19:42 |
<cjming> |
end of UTC late backport & config window |
[production] |
| 19:41 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:737814|Lower mobile web click tracking rate (T295432)]] (duration: 00m 55s) |
[production] |
| 19:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:35 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:737814|Lower mobile web click tracking rate (T295432)]] (duration: 00m 57s) |
[production] |
| 19:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:23 |
<legoktm> |
uploaded php-pcov_1.0.6-4+wmf1~buster1_amd64.changes to apt.wm.o (T243847) |
[production] |
| 18:57 |
<mutante> |
removing mediawiki font packages from parsoid hosts - T294378 |
[production] |
| 18:37 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6002.wikimedia.org with OS buster |
[production] |
| 18:37 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS buster |
[production] |
| 18:20 |
<btullis> |
btullis@an-launcher1002:~$ sudo systemctl reset-failed monitor_refine_event_sanitized_analytics_delayed.service |
[analytics] |
| 18:19 |
<dancy@deploy1002> |
Finished scap: Config: [[gerrit:737976|Get rid of obsolete train-versions.json file]] (duration: 15m 57s) |
[production] |
| 18:09 |
<bblack> |
drmrs - rebooting a bunch of hosts to bios for further settings, please ignore any accidental alerts - they do *look* like they're alert-disabled) |
[production] |
| 18:08 |
<vgutierrez> |
restart haproxy on cp4026 and cp5006 to enable hitless reloads - T290005 |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:03 |
<dancy@deploy1002> |
Started scap: Config: [[gerrit:737976|Get rid of obsolete train-versions.json file]] |
[production] |
| 17:10 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6001.wikimedia.org with OS buster |
[production] |
| 17:08 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/737970 in deployment-prep. Should only affect deployment-webperf11. |
[releng] |
| 16:49 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6002.wikimedia.org with OS buster |
[production] |
| 16:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |