|
2021-11-10
§
|
| 23:46 |
<ebernhardson> |
start test backup/restore of 1tb commonswiki from relforge to swift in eqiad |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=DoubleRedirects |
[production] |
| 23:33 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript updateSpecialPages.php --wiki=foundationwiki --only=BrokenRedirects |
[production] |
| 22:06 |
<bblack> |
dns2002 - restart ntp.servce to fix drmrs peering |
[production] |
| 22:01 |
<bblack> |
dns1002 - restart ntp.servce to fix drmrs peering |
[production] |
| 21:56 |
<bblack> |
dns2001 - restart ntp.service to fix drmrs peering |
[production] |
| 21:53 |
<bblack> |
dns1001 - restart ntp.service to see if drmrs associations cleared up after dns changes, etc |
[production] |
| 21:24 |
<bblack> |
asw1-b1[23]-drmrs: added ipv6 router-advertisement clauses, which work, but probably imperfectly :) |
[production] |
| 19:52 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS buster |
[production] |
| 19:51 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6002.wikimedia.org with OS buster |
[production] |
| 19:51 |
<ottomata> |
altering {eqiad,codfw}.maps.tiles_change to increase to 6 partitions in kafka main-eqiad, main-codfw and jumbo-eqiad: https://phabricator.wikimedia.org/T293366#7497076 |
[production] |
| 19:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:49 |
<mutante> |
- removing manually added things in Horizon Hiera that were already in the repo, please don't keep adding in web UI, we don't want to repeat the same thing we did in deployment-prep |
[devtools] |
| 19:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:43 |
<cjming> |
end of UTC evening backport & config window |
[production] |
| 19:42 |
<cjming> |
end of UTC late backport & config window |
[production] |
| 19:41 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:737814|Lower mobile web click tracking rate (T295432)]] (duration: 00m 55s) |
[production] |
| 19:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:35 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:737814|Lower mobile web click tracking rate (T295432)]] (duration: 00m 57s) |
[production] |
| 19:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:23 |
<legoktm> |
uploaded php-pcov_1.0.6-4+wmf1~buster1_amd64.changes to apt.wm.o (T243847) |
[production] |
| 18:57 |
<mutante> |
removing mediawiki font packages from parsoid hosts - T294378 |
[production] |
| 18:37 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6002.wikimedia.org with OS buster |
[production] |
| 18:37 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS buster |
[production] |
| 18:20 |
<btullis> |
btullis@an-launcher1002:~$ sudo systemctl reset-failed monitor_refine_event_sanitized_analytics_delayed.service |
[analytics] |
| 18:19 |
<dancy@deploy1002> |
Finished scap: Config: [[gerrit:737976|Get rid of obsolete train-versions.json file]] (duration: 15m 57s) |
[production] |
| 18:09 |
<bblack> |
drmrs - rebooting a bunch of hosts to bios for further settings, please ignore any accidental alerts - they do *look* like they're alert-disabled) |
[production] |
| 18:08 |
<vgutierrez> |
restart haproxy on cp4026 and cp5006 to enable hitless reloads - T290005 |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:03 |
<dancy@deploy1002> |
Started scap: Config: [[gerrit:737976|Get rid of obsolete train-versions.json file]] |
[production] |
| 17:10 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6001.wikimedia.org with OS buster |
[production] |
| 17:08 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/737970 in deployment-prep. Should only affect deployment-webperf11. |
[releng] |
| 16:49 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns6002.wikimedia.org with OS buster |
[production] |
| 16:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:32 |
<ebernhardson@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 55s) |
[production] |
| 16:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:28 |
<elukey> |
move atskafka to the new CA bundle - T291905 |
[production] |
| 16:26 |
<elukey> |
move kafkatee instances (analytics-test,centralog) to the new CA bundle - T291905 |
[production] |
| 16:21 |
<_joe_> |
cherry-picking puppet patches for mediawiki on php 7.4 to deployment-prep |
[releng] |
| 16:14 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6002.wikimedia.org with OS buster |
[production] |
| 16:12 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS buster |
[production] |
| 16:07 |
<hashar> |
reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/737677 |
[releng] |
| 15:52 |
<ebernhardson@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T295480: Move all cirrussearch traffic to codfw (duration: 00m 56s) |
[production] |
| 15:35 |
<majavah> |
delete shut down deployment-logstash* and deployment-mwmaint01 instances |
[releng] |