7201-7250 of 10000 results (77ms)
2020-02-27 ยง
16:02 <effie> disable puppet on thumbor* [production]
15:59 <moritzm> installing e2fsck security updates on buster [production]
15:56 <moritzm> installing python-django updates (packaged Debian version) [production]
15:52 <moritzm> installing python-pysaml security updates [production]
15:37 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:35 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:32 <reedy@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 04s) [production]
15:31 <reedy@deploy1001> Synchronized php-1.35.0-wmf.21/extensions/ConfirmEdit/includes/auth/CaptchaPreAuthenticationProvider.php: T245280 (duration: 01m 05s) [production]
15:29 <moritzm> restarting mw canaries to pick up curl update [production]
15:23 <moritzm> installing curl security updates on stretch/buster [production]
15:17 <vgutierrez> reimage lvs4006 with buster - T245984 [production]
15:03 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1084 at 50% T245621', diff saved to https://phabricator.wikimedia.org/P10542 and previous config saved to /var/cache/conftool/dbconfig/20200227-150302-jynus.json [production]
14:55 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:53 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:35 <vgutierrez> reimage lvs4007 with buster - T245984 [production]
14:08 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: 7e3a57a: Increase arwiki WikiGap throttle lift to 400 accounts (T246092) (duration: 01m 05s) [production]
13:28 <_joe_> installing envoy in eqiad too [production]
13:13 <cdanis> s/camping/clamping/ [production]
13:11 <XioNoX> esams/knams rollback tcp-mss camping and prepending [production]
13:07 <_joe_> restarting envoy, after chowning the log files, on all codfw mw servers where it was installed [production]
13:06 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q2M (was Q8M) again (T219123) ?cachebust (duration: 01m 03s) [production]
13:05 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q2M (was Q8M) again (T219123) (duration: 01m 03s) [production]
13:03 <_joe_> re-stopped puppet on codfw [production]
12:56 <XioNoX> delete specific tcp-mss on cr2-eqiad:equinix (will cause an interface flap) - T244610 [production]
12:41 <XioNoX> bump BGP prefix-limit on all routers - T246110 [production]
12:37 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8M (was Q6M) again (T219123) ?cachebust (duration: 01m 03s) [production]
12:36 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8M (was Q6M) again (T219123) (duration: 01m 04s) [production]
12:27 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6M (was Q2M) again (T219123) cachebust? (duration: 01m 17s) [production]
12:24 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6M (was Q2M) again (T219123) (duration: 01m 45s) [production]
12:20 <vgutierrez@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
12:19 <vgutierrez@cumin2001> START - Cookbook sre.hosts.decommission [production]
12:18 <vgutierrez@cumin2001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
12:18 <vgutierrez@cumin2001> START - Cookbook sre.hosts.decommission [production]
12:14 <vgutierrez> replace lvs2003 with lvs2009 - T196560 T245984 T246334 [production]
12:11 <Urbanecm> EU SWAT done [production]
12:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: daee105: Add ids.si.edu to the wgCopyUploadsDomains whitelist of Wikimedia Commons (T246330; take II) (duration: 01m 04s) [production]
12:05 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: daee105: Add ids.si.edu to the wgCopyUploadsDomains whitelist of Wikimedia Commons (T246330) (duration: 01m 05s) [production]
11:48 <vgutierrez> run decommision script against lvs2006.codfw.wmnet - T246329 [production]
11:47 <vgutierrez@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:47 <vgutierrez@cumin2001> START - Cookbook sre.hosts.decommission [production]
11:45 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1084 at 10% T245621', diff saved to https://phabricator.wikimedia.org/P10538 and previous config saved to /var/cache/conftool/dbconfig/20200227-114542-jynus.json [production]
11:35 <addshore> pause item migration script at Q50 million T219123 [production]
11:02 <vgutierrez> start pybal on lvs2003 - T196560 T245984 [production]
10:58 <vgutierrez> stop pybal on lvs2003 to let lvs2010 take the traffic for a little bit - T196560 T245984 [production]
10:54 <vgutierrez> replacing lvs2006 with lvs2010 - T196560 T245984 [production]
09:35 <jynus> upgrade and restart db1084 T246323 [production]
09:03 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1098 (s6 & s7)', diff saved to https://phabricator.wikimedia.org/P10536 and previous config saved to /var/cache/conftool/dbconfig/20200227-090344-jynus.json [production]
08:26 <jynus> killed SpecialFewestRevisions::reallyDoQuery long running query on db1101:s8, causing lag [production]
08:14 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1098 at 50%', diff saved to https://phabricator.wikimedia.org/P10535 and previous config saved to /var/cache/conftool/dbconfig/20200227-081449-jynus.json [production]
03:52 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]