3651-3700 of 10000 results (31ms)
2014-06-11 §
10:45 <godog> restarted uwsgi on tungsten, hung on fetching many metrics [production]
09:54 <_joe_> restarted apache on palladium - passenger crashed [production]
05:26 <paravoid> restarting all swift daemons across the cluster to fix runaway threads due to rsyslog restart [production]
05:04 <springle> beginning schema changes bug 49193 page_content_model [production]
03:29 <LocalisationUpdate> ResourceLoader cache refresh completed at Wed Jun 11 03:28:14 UTC 2014 (duration 28m 13s) [production]
02:29 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-11 02:28:18+00:00 [production]
02:15 <LocalisationUpdate> completed (1.24wmf7) at 2014-06-11 02:14:43+00:00 [production]
2014-06-10 §
23:34 <andrewbogott> updated labs Trusty image w/puppet3, made default [production]
23:19 <mutante> rebooting unresponsive ms-be1003 [production]
21:10 <RobH> montly sms credit check: 1,447.36 SMS credits. will check again in 30 days [production]
19:47 <hashar> Jenkins restarted apparently properly. Any breakage would probably be related to the version switch :-D [production]
19:45 <ottomata> power cycling analytics1012, attempting to reinstall as kafka broker with new kafka partman recipe [production]
19:42 <hashar> Jenkins upgraded from 1.532.2 to 1.554.2 (i.e. bumped to a new LTS version). [production]
19:37 <hashar> Broke Jenkins by silently upgrading it :-( [production]
19:09 <Krinkle> git-deploy: Deploying integration/slave-scripts I9521890b911714edf2 [production]
18:59 <reedy> Synchronized php-1.24wmf8/skins/vector/components/tabs.less: (no message) (duration: 00m 14s) [production]
18:58 <mutante> shutting down ekrem [production]
18:18 <reedy> Synchronized wmf-config/InitialiseSettings.php: Enable data transclusion for wikiquote (duration: 00m 14s) [production]
18:15 <reedy> Synchronized docroot and w: Update non Wikipedias to 1.24wmf8 (duration: 00m 16s) [production]
18:15 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Update non Wikipedias to 1.24wmf8 [production]
18:14 <reedy> Synchronized php-1.24wmf8/extensions/Wikidata/: (no message) (duration: 00m 16s) [production]
17:29 <_joe|away> restarted profiler-to-carbon, stuck waiting data from mwprof [production]
15:21 <mutante> ekrem - rm from stored configs/icinga [production]
15:12 <mutante> ekrem - revoke salt,puppet keys, stop agents/minion [production]
07:42 <springle> enabled pt-slave-delay for dbstore1001, 24h all shards [production]
06:12 <springle> xtrabackup clone db1043 to db1048 [production]
04:57 <springle> db1048 down for upgrade [production]
03:40 <springle> switched mchenry to use m2-master/m2-slave for OTRS address lookups [production]
03:25 <LocalisationUpdate> ResourceLoader cache refresh completed at Tue Jun 10 03:24:19 UTC 2014 (duration 24m 18s) [production]
02:29 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-10 02:28:14+00:00 [production]
02:27 <springle> switched traffic db1048 to db1020. broke gerrit briefly; see ops email [production]
02:15 <LocalisationUpdate> completed (1.24wmf7) at 2014-06-10 02:14:41+00:00 [production]
01:33 <chasemp> restarted gerrit on ytterbium [production]
01:01 <manybubbles> upgraded all elasticsearch servers in production to 1.2.1. They are just restoring the last few shards on the last node now and they'll spend a few hours tonight rebalancing after the upgrade but otherwise I'm done. [production]
00:41 <mwalker> updating donationinterface on payments from b4c5cf1bceb70d65eae28cdd0873036dc33c8992 to 6d74002f2634f41f7038daa7357ff6de55ee4880 for worldpay form error [production]
2014-06-09 §
23:58 <manybubbles> lied - upgrading elastic1014 [production]
23:57 <manybubbles> upgrading elastic1015 [production]
23:30 <Krinkle> Reloading Zuul to deploy 6727b8b [production]
23:12 <maxsem> Synchronized php-1.24wmf8/extensions/MobileApp: (no message) (duration: 00m 03s) [production]
23:11 <maxsem> Synchronized php-1.24wmf7/extensions/MobileApp: (no message) (duration: 00m 03s) [production]
23:03 <maxsem> Synchronized wmf-config/InitialiseSettings.php: https://bugzilla.wikimedia.org/66377 (duration: 00m 04s) [production]
20:42 <manybubbles> upgraded elastic1007-elastic1010 without issue - starting elastic1010 [production]
20:08 <subbu> deployed Parsoid 9b673587 (deploy sha 7d0097a1) [production]
19:23 <ottomata> disabling puppet on analytics1012 [production]
19:00 <ottomata> decomissioning analytics1012 in hadoop cluster, this will become a Kafka broker [production]
17:59 <manybubbles> elastic1004-1006 upgraded without trouble - cluster is working on filling elatic1006 before moving on to 1007, and the rest [production]
17:04 <andrewbogott> switching labs to puppet3 [production]
17:03 <awight> update crm from b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7 to b6815d29de97b80a0ab65db576213a604f0c7cb9 [production]
16:30 <manybubbles> upgrading elastic1003 - upgrade is going well so far so I'm going to stop watching it as closely and let it be more automated [production]
15:28 <manybubbles> elastic1001 went well, doing 1002 by hand again [production]