7401-7450 of 10000 results (55ms)
2015-06-29 §
15:51 <bblack> disabling puppet on caches temporarily ... [production]
15:49 <krenair> Synchronized php-1.26wmf11/extensions/OpenStackManager: https://gerrit.wikimedia.org/r/#/c/221648/ (duration: 00m 13s) [production]
15:29 <krenair> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221405/ (duration: 00m 15s) [production]
15:27 <krenair> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221612/ (duration: 00m 12s) [production]
15:24 <krenair> Synchronized w/static/images/project-logos/zhwiki-hans-2x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 14s) [production]
15:24 <krenair> Synchronized w/static/images/project-logos/zhwiki-hans-1.5x.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s) [production]
15:23 <krenair> Synchronized w/static/images/project-logos/zhwiki-hans.png: https://gerrit.wikimedia.org/r/#/c/221113/ (duration: 00m 12s) [production]
15:21 <krenair> Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/221009/ (duration: 00m 11s) [production]
15:18 <krenair> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/221047/ (duration: 00m 13s) [production]
15:12 <krenair> Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.link.js: https://gerrit.wikimedia.org/r/#/c/221605 (duration: 00m 13s) [production]
15:02 <krenair> Synchronized php-1.26wmf11/extensions/ContentTranslation/modules/tools/ext.cx.tools.formatter.js: https://gerrit.wikimedia.org/r/#/c/221604/ (duration: 00m 14s) [production]
14:34 <jynus> rebooting and reinstalling db1022 [production]
12:06 <YuviPanda> restarting rsync with new exclusions file on labstore1002 to codfw [production]
12:06 <YuviPanda> excluded maps, mwoffliner and video project from rsync of broken FS to speed it up [production]
11:59 <YuviPanda> interupt rsync on labstore1001 to prevent it from copying mwofflienr files [production]
11:00 <_joe_> shutting down etcd1003, cleaning exported resources [production]
10:32 <_joe_> effectively removing etcd1003 from the cluster [production]
10:17 <_joe_> starting removal of etcd1003 from the etcd cluster [production]
08:49 <_joe_> joined conf1003 to the etcd cluster [production]
08:21 <jynus> Synchronized wmf-config/db-eqiad.php: Depool db1022 for reinstall (duration: 00m 12s) [production]
08:12 <_joe_> adding conf1002 to the etcd cluster as a member [production]
07:46 <akosiaris> disabling ntp everywhere expect selected hosts in anticipation for the leap second [production]
04:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Jun 29 04:51:48 UTC 2015 (duration 51m 47s) [production]
03:08 <jgage> jmxtrans filled disks on all kafka brokers, 21GB log files. removed logs and restarted services. [production]
02:23 <LocalisationUpdate> completed (1.26wmf11) at 2015-06-29 02:23:47+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 53s) [production]
00:52 <springle> restart eventlogging auto-purge on m4 [production]
00:51 <springle> restart replication on dbstore2002 [production]
00:00 <springle> pausing replication on dbstore2002 [production]
2015-06-28 §
23:51 <ori> Synchronized php-1.26wmf11/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I6ffdc977e87: Parse older format of Geo cookies (duration: 00m 13s) [production]
04:30 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Jun 28 04:30:54 UTC 2015 (duration 30m 53s) [production]
02:20 <LocalisationUpdate> completed (1.26wmf11) at 2015-06-28 02:20:52+00:00 [production]
02:17 <l10nupdate> Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 56s) [production]
2015-06-27 §
23:30 <bd808> Deleted corrupt shards on logstash1004 and logstash1005. Recovery in process [production]
20:12 <ori> Delegated full access to Google Webmaster Tools for myself (olivneh@). [production]
04:58 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Jun 27 04:58:46 UTC 2015 (duration 58m 45s) [production]
02:23 <LocalisationUpdate> completed (1.26wmf11) at 2015-06-27 02:23:40+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 46s) [production]
2015-06-26 §
23:57 <bd808> Logstash log ingestion working again after forcing recovery of replicas for logstash-2015.06.26; new logs were being rejected with only a primary shard available [production]
23:54 <bd808> re-enabled allocation on logstash elasticsearch cluster [production]
23:05 <bblack> restarted gitblit on antimony, AGAIN [production]
22:57 <mutante> restarted gitblit [production]
22:44 <catrope> Synchronized php-1.26wmf11/extensions/Flow: Temporarily make subpages in Flow-occupied namespaces non-Flow again (duration: 00m 14s) [production]
22:36 <bd808> set indices.recovery.concurrent_streams to 4 on logstash ES cluster [production]
22:36 <godog> set indices.recovery.max_bytes_per_sec to 10mb on logstash ES cluster [production]
22:25 <godog> set indices.recovery.max_bytes_per_sec to 50mb on logstash ES cluster [production]
22:25 <jamesofur> Reset email address of User:Chwms identity verified in person at editathon [production]
22:09 <bd808> restarted logstash on logstash1001 [production]
21:10 <urandom> taking xenon down to be rebootstrapped [production]
20:10 <bd808> Deleted 4 corrupt indices (logstash-2015.05.30 logstash-2015.05.31 logstash-2015.06.03 logstash-2015.06.06) on logstash1004 [production]