201-250 of 10000 results (23ms)
2015-06-05 §
19:51 <mutante> chown root:root / on terbium [production]
19:50 <godog> bounce redis on rdb1002/rdb1004 to pick up new slave limits [production]
19:29 <godog> bounce redis again on rdb1003 after increasing the slave limits more [production]
19:17 <godog> bounce redis on rdb1003 after bumping slave limits [production]
19:07 <godog> redis master logs shows periodic 'cmd=sync scheduled to be closed ASAP for overcoming of output buffer limits.' indicating the slave fails to sync [production]
18:40 <godog> spike in redis network starting at ~15.00 UTC, correlates with ocg failures [production]
18:01 <moritzm> restarted gerrit on ytterbium for java update [production]
14:43 <jynus> short lag period on db1049, traffic automatically redirected to other slave and back to normal [production]
14:07 <moritzm> added ubuntu-meta-1.325+wmf1 for trusty-wikimedia to apt.wikimedia.org (T100004) [production]
14:07 <moritzm> added ubuntu-meta-1.267.1+wmf1 for precise-wikimedia to apt.wikimedia.org (T100004) [production]
12:44 <jynus> Synchronized wmf-config/db-eqiad.php: Depool es1007 (duration: 01m 08s) [production]
12:08 <jynus> Synchronized wmf-config/db-eqiad.php: Repool es1009 (duration: 01m 08s) [production]
11:30 <_joe_> uploaded new HHVM package, installing on mw1025 for testing [production]
09:17 <moritzm> added redis_2.6.13-1+wmf1 to precise-wikimedia on apt.wikimedia.org [production]
06:24 <moritzm> added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org [production]
05:24 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Jun 5 05:22:50 UTC 2015 (duration 22m 49s) [production]
04:10 <springle> Synchronized wmf-config/db-eqiad.php: depool db1073 (duration: 01m 08s) [production]
02:26 <LocalisationUpdate> completed (1.26wmf8) at 2015-06-05 02:25:20+00:00 [production]
02:21 <l10nupdate> Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s) [production]
01:27 <tgr> deploying schema changes for Gather on enwiki, enwikivoyage, hewiki (T98490, T101460) [production]
00:08 <catrope> Synchronized php-1.26wmf8/vendor/oojs/oojs-ui/php/Tag.php: Fix OOUI fatals (T99210) (duration: 00m 13s) [production]
2015-06-04 §
23:40 <catrope> Synchronized php-1.26wmf8/extensions/MobileFrontend: SWAT (duration: 00m 13s) [production]
23:28 <catrope> Synchronized wmf-config/InitialiseSettings.php: Disable VE A/B test for new accounts on enwiki (duration: 00m 13s) [production]
22:39 <ejegg> updated payments from d22e44e3fab2b937707c2776384cb93a49b4cfd3 to 43c7952d2a31deaea97e8319f5612d644dce43c8 [production]
22:21 <ottomata> doing controlled restart of kafka brokers services to apply auto create topic config [production]
21:48 <jgage> analyics1013 crashed, rebooted [production]
21:42 <ori> Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: 1b20d62c26: Revert "awful hack: disable fss on zhwiki only, except on mw1017" (duration: 00m 13s) [production]
21:34 <ori> performing rolling restart of HHVMs for hhvm-fss upgrade [production]
21:27 <bd808> restarted logstash and elasticsearch on logstash100[1-3] to pick up latest jre updates [production]
18:48 <mutante> restarted apache on silver/wikitech [production]
18:20 <jynus> Synchronized wmf-config/db-eqiad.php: Depool es1009 and master-slave switchover (duration: 00m 13s) [production]
18:01 <awight> Enabling PayPal audit parser job [production]
17:57 <jynus> Synchronized wmf-config/db-eqiad.php: Repool es1008 (duration: 00m 15s) [production]
17:44 <jynus> Synchronized wmf-config/db-codfw.php: Repool es2008 and its slaves (duration: 00m 13s) [production]
17:21 <ori> Disabling Puppet and nutcracker on mw1017 to control for parser cache [production]
17:18 <jynus> Synchronized wmf-config/db-codfw.php: Depool es2008 and its slaves (duration: 00m 13s) [production]
17:17 <jynus> Synchronized wmf-config/db-eqiad.php: Depool es1008 (duration: 00m 12s) [production]
16:33 <kartik> Finished scap: Update ContentTranslation (duration: 09m 17s) [production]
16:24 <kartik> Started scap: Update ContentTranslation [production]
15:54 <moritzm> added redis_2.8.4-2+wmf1 to trusty-wikimedia on apt.wikimedia.org [production]
15:48 <anomie> Synchronized php-1.26wmf8/includes/jobqueue/: SWAT: jobqueue: Record stats on how long it takes before a job is run [[gerrit:215748]] (duration: 00m 14s) [production]
15:38 <anomie> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ApiFeatureUsage everywhere [[gerrit:215901]] (duration: 00m 19s) [production]
15:36 <anomie> Synchronized wmf-config/CommonSettings.php: SWAT: Remove obsolete 'ValidateExtendedMetadataCache' hook [[gerrit:215900]] (duration: 00m 12s) [production]
15:35 <anomie> Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Added staff-recommender campaign [[gerrit:215865]] (duration: 00m 12s) [production]
15:30 <anomie> Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Add wikis for deployment on 20150406 [[gerrit:215281]] (duration: 00m 12s) [production]
15:12 <ori> Synchronized php-1.26wmf8/includes/libs/ReplacementArray.php: Ia5f3dc84605: awful hack: disable fss on zhwiki only, except on mw1017 (duration: 00m 17s) [production]
15:09 <_joe_> puppet disabled, fss disabled on mw1017 [production]
14:42 <YuviPanda> running sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on all labs nodes [production]
14:36 <awight> Disable PayPal audit parsing job [production]
12:19 <springle> Synchronized wmf-config/db-eqiad.php: repool db1072, warm up (duration: 00m 13s) [production]