151-200 of 10000 results (37ms)
2017-09-20 ยง
16:04 <gehel@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch [production]
16:03 <gehel> depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw) [production]
15:46 <ejegg> limited donations queue consumer to 100 per run [production]
15:36 <ejegg> updated payments-wiki from 896f16ac380aaa6022749b395f9187ecf4bc802b to 9da84823ff4ac3071d218ec211fb59c054de9e15 [production]
15:36 <gehel> upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231 [production]
15:33 <ejegg> limited donations queue consumer to 200 per run [production]
14:03 <akosiaris@tin> Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s) [production]
13:42 <otto@tin> Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s) [production]
13:24 <ema> begin rolling codfw varnish-be restarts, 20m spacing [production]
13:20 <dereckson@tin> Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 3/3) (duration: 00m 48s) [production]
13:18 <dereckson@tin> Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 2/3) (duration: 00m 49s) [production]
13:17 <dereckson@tin> Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw ([[Gerrit:378850]], 1/3, no-op in prod) (duration: 00m 49s) [production]
12:51 <ema> restarted varnish-be on cp4028 [production]
12:28 <gehel> upgrading nodejs to 6.11 on maps-test2003 for testing - T171707 [production]
12:19 <ema> restarted varnish-be on cp4027 [production]
12:18 <moritzm> uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top) [production]
12:08 <hoo@tin> Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s) [production]
11:58 <mobrovac@tin> Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s) [production]
11:56 <ema> restarted varnish-be on cp4018 [production]
11:51 <mobrovac@tin> Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 [production]
11:49 <jynus> stopping replication on db1101 for faster repartition work T176311 [production]
11:45 <jynus@tin> Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s) [production]
11:34 <ema> restarted varnish-be on cp4010 [production]
11:25 <mobrovac@tin> Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s) [production]
11:25 <mobrovac@tin> Started deploy [restbase/deploy@dea2b41]: (no justification provided) [production]
11:03 <moritzm> rolling out new debdeploy release across the fleet [production]
11:02 <mobrovac@tin> Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s) [production]
11:02 <mobrovac@tin> Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) [production]
11:02 <ema> restarted varnish-be on cp4008 [production]
11:02 <mobrovac@tin> Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s) [production]
11:01 <mobrovac@tin> Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) [production]
11:01 <mobrovac@tin> Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s) [production]
11:01 <mobrovac@tin> Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) [production]
11:00 <mobrovac@tin> Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s) [production]
11:00 <mobrovac@tin> Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) [production]
09:55 <bblack> restarted varnish-be on cp3042 [production]
09:29 <bblack> restarted varnish-be on cp1052 [production]
09:23 <ema> restarted varnish-be on cp3043 [production]
09:21 <ema> restarted varnish-be on cp3041 [production]
09:19 <ema> restarted varnish-be on cp3032 [production]
09:14 <ema> restarted varnish-be on cp3040 [production]
08:42 <bblack> restarted varnish-be on cp1053 [production]
08:41 <bblack> restarted varnish-be on cp1068 [production]
08:40 <bblack> restarted varnish-be on cp1054 [production]
08:40 <bblack> restarted varnish-be on cp1065 [production]
07:11 <ema> cp3030 varnish-be restart due to 503s [production]
03:16 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s) [production]
03:08 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s) [production]
02:31 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s) [production]
00:55 <bblack> cp1066 - backend restart, mailbox lag, text [production]