|
2020-02-27
ยง
|
| 15:35 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:32 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 04s) |
[production] |
| 15:31 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.21/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 05s) |
[production] |
| 15:29 |
<moritzm> |
restarting mw canaries to pick up curl update |
[production] |
| 15:23 |
<moritzm> |
installing curl security updates on stretch/buster |
[production] |
| 15:17 |
<vgutierrez> |
reimage lvs4006 with buster - T245984 |
[production] |
| 15:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 at 50% T245621', diff saved to https://phabricator.wikimedia.org/P10542 and previous config saved to /var/cache/conftool/dbconfig/20200227-150302-jynus.json |
[production] |
| 14:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:35 |
<vgutierrez> |
reimage lvs4007 with buster - T245984 |
[production] |
| 14:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: 7e3a57a: Increase arwiki WikiGap throttle lift to 400 accounts (T246092) (duration: 01m 05s) |
[production] |
| 13:28 |
<_joe_> |
installing envoy in eqiad too |
[production] |
| 13:13 |
<cdanis> |
s/camping/clamping/ |
[production] |
| 13:11 |
<XioNoX> |
esams/knams rollback tcp-mss camping and prepending |
[production] |
| 13:07 |
<_joe_> |
restarting envoy, after chowning the log files, on all codfw mw servers where it was installed |
[production] |
| 13:06 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q2M (was Q8M) again (T219123) ?cachebust (duration: 01m 03s) |
[production] |
| 13:05 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q2M (was Q8M) again (T219123) (duration: 01m 03s) |
[production] |
| 13:03 |
<_joe_> |
re-stopped puppet on codfw |
[production] |
| 12:56 |
<XioNoX> |
delete specific tcp-mss on cr2-eqiad:equinix (will cause an interface flap) - T244610 |
[production] |
| 12:41 |
<XioNoX> |
bump BGP prefix-limit on all routers - T246110 |
[production] |
| 12:37 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8M (was Q6M) again (T219123) ?cachebust (duration: 01m 03s) |
[production] |
| 12:36 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8M (was Q6M) again (T219123) (duration: 01m 04s) |
[production] |
| 12:27 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6M (was Q2M) again (T219123) cachebust? (duration: 01m 17s) |
[production] |
| 12:24 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6M (was Q2M) again (T219123) (duration: 01m 45s) |
[production] |
| 12:20 |
<vgutierrez@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 12:19 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 12:18 |
<vgutierrez@cumin2001> |
END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) |
[production] |
| 12:18 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 12:14 |
<vgutierrez> |
replace lvs2003 with lvs2009 - T196560 T245984 T246334 |
[production] |
| 12:11 |
<Urbanecm> |
EU SWAT done |
[production] |
| 12:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: daee105: Add ids.si.edu to the wgCopyUploadsDomains whitelist of Wikimedia Commons (T246330; take II) (duration: 01m 04s) |
[production] |
| 12:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: daee105: Add ids.si.edu to the wgCopyUploadsDomains whitelist of Wikimedia Commons (T246330) (duration: 01m 05s) |
[production] |
| 11:48 |
<vgutierrez> |
run decommision script against lvs2006.codfw.wmnet - T246329 |
[production] |
| 11:47 |
<vgutierrez@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:47 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:45 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1084 at 10% T245621', diff saved to https://phabricator.wikimedia.org/P10538 and previous config saved to /var/cache/conftool/dbconfig/20200227-114542-jynus.json |
[production] |
| 11:35 |
<addshore> |
pause item migration script at Q50 million T219123 |
[production] |
| 11:02 |
<vgutierrez> |
start pybal on lvs2003 - T196560 T245984 |
[production] |
| 10:58 |
<vgutierrez> |
stop pybal on lvs2003 to let lvs2010 take the traffic for a little bit - T196560 T245984 |
[production] |
| 10:54 |
<vgutierrez> |
replacing lvs2006 with lvs2010 - T196560 T245984 |
[production] |
| 09:35 |
<jynus> |
upgrade and restart db1084 T246323 |
[production] |
| 09:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1098 (s6 & s7)', diff saved to https://phabricator.wikimedia.org/P10536 and previous config saved to /var/cache/conftool/dbconfig/20200227-090344-jynus.json |
[production] |
| 08:26 |
<jynus> |
killed SpecialFewestRevisions::reallyDoQuery long running query on db1101:s8, causing lag |
[production] |
| 08:14 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1098 at 50%', diff saved to https://phabricator.wikimedia.org/P10535 and previous config saved to /var/cache/conftool/dbconfig/20200227-081449-jynus.json |
[production] |
| 03:52 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:50 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:31 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:28 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:27 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:26 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |