3851-3900 of 10000 results (41ms)
2020-05-13 §
09:38 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:37 <marostegui> Upgrade db2102 to the new 10.4.13 - T250666 [production]
09:32 <_joe_> installing purged 0.11 on cp2027 T133821 [production]
09:21 <_joe_> installing purged 0.11 on cp2028 T133821 [production]
09:11 <moritzm> re-enabling puppet [production]
09:08 <mutante> rsyncing /home dirs from people.wikimedia.org to new backend people1002 [production]
09:00 <moritzm> disabling puppet temporarily [production]
08:53 <_joe_> uploaded purged 0.11 [production]
08:52 <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool pc1010 as pc1 master T252182 (duration: 01m 17s) [production]
07:42 <jayme> imported helm 2.16.7-1 to main for stretch-wikimedia [production]
07:41 <jayme> imported helm 2.16.7-1 to main for buster-wikimedia [production]
07:29 <godog> roll-restart logstash in codfw/eqiad for configuration change [production]
07:14 <elukey> upload spark2_2.4.4-bin-hadoop2.6-2 for buster/stretch on apt1001 [production]
05:33 <ryankemper> wdqs2004 was depooled ~3 hours ago and was re-pooled ~10 mins ago after verifying the wdqs service was healthy [production]
05:32 <ryankemper> wdqs1003 was depooled ~6 hours ago and was re-pooled ~10 mins ago after verifying the wdqs service was healthy [production]
05:27 <_joe_> restarting php-fpm on mw1374, children dying with SIGILL [production]
05:11 <root@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
05:11 <root@cumin1001> Updating IPMI password on 1 hosts - root@cumin1001 [production]
05:10 <root@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
05:10 <root@cumin1001> END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) [production]
05:10 <root@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
04:52 <kart_> Updated cxserver to 2020-05-11-082207-production (T250004) [production]
04:47 <kartik@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:44 <kartik@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:42 <kartik@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
02:27 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
02:10 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
00:43 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
00:33 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
2020-05-12 §
23:09 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
23:06 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
20:15 <hashar@deploy1001> Synchronized php-1.35.0-wmf.32/includes/revisionlist/RevisionItemBase.php: Fix RevisionItemBase::getId to actually return an int, as intended - T252076 (duration: 01m 06s) [production]
19:55 <dpifke@deploy1001> Finished deploy [performance/navtiming@48110b9]: Fixes swapped dc/host labels - T238086 (duration: 00m 05s) [production]
19:55 <dpifke@deploy1001> Started deploy [performance/navtiming@48110b9]: Fixes swapped dc/host labels - T238086 [production]
19:05 <hashar@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.32 [production]
18:41 <legoktm> started codereview-archiver script in screen on mwmaint1002 [production]
18:23 <otto@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
18:23 <otto@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
18:17 <otto@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
18:17 <otto@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
18:14 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
18:14 <otto@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
17:49 <bblack> 'gdnsdctl replace' on all authdns to load new maxmind data [production]
17:43 <bblack> updating maxmind database on puppetmasters (usually automated weekly; we're mid-cycle) [production]
17:10 <James_F> Running AbuseFilter updateVarDumps on testwikis on mwmaint1002 T246539 [production]
16:55 <James_F> Running AbuseFilter updateVarDumps on closed wikis on mwmaint1002 T246539 [production]
16:55 <mstyles@deploy1001> Finished deploy [wdqs/wdqs@f617307]: v0.3.31 (duration: 14m 53s) [production]
16:40 <mstyles@deploy1001> Started deploy [wdqs/wdqs@f617307]: v0.3.31 [production]
16:35 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:48 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]