|
2020-09-10
ยง
|
| 22:31 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:11 |
<ejegg> |
updated payments-wiki from be81063168 to e41ab173e0 |
[production] |
| 22:06 |
<mutante> |
added mcrouter cert for parse2020, ran mcrouter_generate_certs |
[production] |
| 21:51 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:49 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:09 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:07 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:52 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:52 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:25 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.8 |
[production] |
| 20:23 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:21 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:20 |
<longma> |
correction: T257976 - 1.36.0-wmf.8 to all wikis |
[production] |
| 20:20 |
<longma> |
deploying 1.36.0-wmf.8 to all wikis |
[production] |
| 20:02 |
<krinkle@deploy1001> |
Synchronized php-1.36.0-wmf.8/includes/resourceloader/ResourceLoaderSkinModule.php: Ibe2c9f8d024f6 (duration: 01m 05s) |
[production] |
| 19:44 |
<Urbanecm> |
End of [urbanecm@mwmaint2001 ~]$ mwscript updateCollation.php --wiki=trwiktionary --previous-collation=uppercase # T262163 |
[production] |
| 19:12 |
<mholloway-shell@deploy1001> |
Started restart [recommendation-api/deploy@db7fd80]: (no justification provided) |
[production] |
| 19:07 |
<Urbanecm> |
Start of [urbanecm@mwmaint2001 ~]$ mwscript updateCollation.php --wiki=trwiktionary --previous-collation=uppercase # T262163 |
[production] |
| 19:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 95d2b575c683a1c5c2972a9bf0cf3b87059fbd74: Set $wgCategoryCollation = uca-tr on trwiktionary (T262163) (duration: 01m 05s) |
[production] |
| 18:58 |
<Urbanecm> |
[urbanecm@mwmaint2001 ~]$ mwscript namespaceDupes.php --wiki=frwiktionary --fix # T262398 |
[production] |
| 18:58 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 09e487e76158026ba161acffad277928d2603891: Add a new namespace to frwiktionary (T262398) (duration: 01m 04s) |
[production] |
| 18:40 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.8/includes/EditPage.php: 824094428c5f41dc9eef7d65c8440dadda4d4dbd: EditPage: Fix member call on boolean when undo is impossible (T262463) (duration: 01m 03s) |
[production] |
| 18:37 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.6/includes/EditPage.php: 824094428c5f41dc9eef7d65c8440dadda4d4dbd: EditPage: Fix member call on boolean when undo is impossible (T262463) (duration: 01m 07s) |
[production] |
| 18:26 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: 0cde0b15fc1daca2cef904bc7add7e9a1c58e3c9: Add throttle rule for Czech senior citizens course (T262415) (duration: 01m 05s) |
[production] |
| 18:24 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:00 |
<mutante> |
helium (former backup host) is being removed from ferm rules on all hosts, it was replaced by backup1001 (T260717) |
[production] |
| 17:33 |
<bblack> |
dns servers: upgrading remainder of fleet to gdnsd-3.3.0-1~wmf1 |
[production] |
| 16:50 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:48 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<bblack> |
authdns1001 - upgrade gdnsd to 3.3.0-1~wmf1 |
[production] |
| 16:06 |
<bblack> |
dns4001 - upgrade gdnsd to 3.3.0-1~wmf1 |
[production] |
| 16:04 |
<bblack> |
reprepro: uploaded gdnsd-3.3.0-1~wmf1 - T261340 |
[production] |
| 15:33 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:30 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:04 |
<volans> |
uploaded cumin_4.0.0 to apt.wikimedia.org buster-wikimedia (no code changes) |
[production] |
| 13:58 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 13:52 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 13:48 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 13:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:42 |
<moritzm> |
rebooting etherpad1002 (etherpad.wikimedia.org) for kernel update |
[production] |
| 13:24 |
<moritzm> |
installing rake security updates on stretch |
[production] |
| 13:10 |
<ebernhardson> |
delete lldwiki_{content|general} indices from search.svc.{eqiad|codfw}.wmnet:9643 (psi), they should be on 9443 (omega) |
[production] |
| 12:57 |
<klausman> |
Ran puppet-merge to get my dotfiles from https://gerrit.wikimedia.org/r/c/operations/puppet/+/626367 out |
[production] |
| 12:34 |
<moritzm> |
installing firejail updates on maps/thumbor/restbase |
[production] |
| 12:01 |
<moritzm> |
upgrading deployment servers to git 2.20 T262244 |
[production] |
| 11:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1106', diff saved to https://phabricator.wikimedia.org/P12557 and previous config saved to /var/cache/conftool/dbconfig/20200910-113758-marostegui.json |
[production] |
| 11:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P12556 and previous config saved to /var/cache/conftool/dbconfig/20200910-113426-marostegui.json |
[production] |
| 11:13 |
<matthiasmullie> |
Euro B&C done |
[production] |