9551-9600 of 10000 results (36ms)
2014-06-07 §
21:31 <manybubbles> elastic1017 is sick - thrashing to death on io - restarting Elasticsearch to see if it recovers unthrashed [production]
17:56 <godog> restarted ES on elastic1017.eqiad.wmnet (at 17:22 UTC) [production]
03:24 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Jun 7 03:23:32 UTC 2014 (duration 23m 31s) [production]
02:44 <bd808> Restarted logstash on deployment-logstash1; last even logged at 2014-06-06T22:11:04 [releng]
02:31 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-07 02:29:57+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf7) at 2014-06-07 02:16:30+00:00 [production]
2014-06-06 §
23:51 <Krinkle> Restarted Jenkins, force stopped Zuul, started Zuul, configure Jenkins via web interface (disable Gearman, save, enable German); Seems to be back up now, finally. [production]
22:52 <mutante> same for rhenium, titanium, bast1001, calcium, carbon, ytterbium, stat1003 [production]
22:43 <RoanKattouw> Restarting Jenkins didn't help, jobs still aren't making it across from Zuul into Jenkins [production]
22:36 <RoanKattouw> Restarting stuck Jenkins [production]
22:35 <mutante> same for holmium, hafnium, silver, netmon1001, magnesium, neon, antimony [production]
22:17 <mutante> upgraded ssl packages on zirconium [production]
21:57 <Krinkle> Took Jenkins slave on gallium temporarily offline and back online to resolve possible stagnation [production]
20:56 <awight_> updated crm from ded541894a70922e098fb3ea48306c8ec0f0f6aa to b38497a9d0ef75fe2b20b03b649ac13a5e3f47a7 [production]
19:26 <bblack> - synced labs/private on deployment-salt again [releng]
18:25 <mwalker> updating payments from e823354822c7a35e6c2069d3e72180a45dbc89dc to b4c5cf1bceb70d65eae28cdd0873036dc33c8992 for globalcollect oid hack [production]
16:30 <bd808> Rebooted deployment-salt [releng]
16:27 <bd808> Made /var/log a symlink to /srv/var-log on deployment-salt [releng]
16:26 <bblack> Updated labs/private.git on puppetmaster. brings in updated zero+netmapper password for beta [releng]
16:18 <bd808> Changed from role::labs::lvm::biglogs to role::labs::lvm::srv on deployment-salt and made /var/lib a symlink to /srv/var-lib [releng]
15:45 <bd808> /var on deployment-salt still at 97% full after moving logs; /var/lib is our problem [releng]
15:43 <bd808> Archived deployment-salt:/var/log to /data/project/deployment-salt [releng]
15:40 <bd808> Disabled puppet on deployment-salt to work on disk space issues [releng]
14:04 <hashar> Gerrit back. chase rebooted it :) [production]
13:55 <hashar> Gerrit having some troubles: error: RPC failed; result=22, HTTP code = 503 (while cloning CirrusSearch ) [production]
12:58 <cmjohnson1> replacing raid controller db1020 [production]
12:44 <hashar> Updated labs/private.git on puppetmaster. Brings Brandon Black change "add labs copy of zerofetcher auth file" {{gerrit|137918}} [releng]
06:12 <Tim> on osmium installed nodejs for testing [production]
04:24 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Jun 6 04:23:08 UTC 2014 (duration 23m 7s) [production]
03:13 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-06 03:12:19+00:00 [production]
02:48 <mwalker> added role::labs::lvm::biglogs to deployment-salt because it is out of room on /var and I don't know what I can delete [releng]
02:43 <LocalisationUpdate> completed (1.24wmf7) at 2014-06-06 02:42:28+00:00 [production]
01:25 <bd808> Live hacked /etc/apache2/wmf/hhvm.conf on apaches to allow them to start [releng]
00:38 <bblack> nginx restarted on ssl* [production]
00:30 <bd808> `git stash`ed dirty dblist files found in /a/common on deployment-bastion [releng]
00:16 <mutante> fixed permissions on bugzilla's index.cgi, sry [production]
2014-06-05 §
23:18 <maxsem> Synchronized php-1.24wmf7/includes/ChangeTags.php: https://gerrit.wikimedia.org/r/#/c/137563/ (duration: 00m 03s) [production]
23:16 <maxsem> Synchronized php-1.24wmf8/includes/ChangeTags.php: https://gerrit.wikimedia.org/r/#/c/137563/ (duration: 00m 03s) [production]
23:06 <maxsem> Synchronized php-1.24wmf7/extensions/TemplateData: https://gerrit.wikimedia.org/r/#/c/137751/ (duration: 00m 04s) [production]
22:15 <ori> Synchronized wmf-config/CommonSettings.php: Ife5081549: Put $wgRCFeeds[rcs100x] config behind $wmfRealm check (duration: 00m 04s) [production]
22:12 <ori> updated /a/common to {{Gerrit|Ife5081549}}: Put $wgRCFeeds['rcs100x'] config behind $wmfRealm check [production]
21:48 <ori> updated eventlogging to a8602c1d879f [production]
21:34 <MaxSem> Renaming geo_killlist and geo_updates to *_old [production]
18:36 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 14s) [production]
18:35 <reedy> Synchronized database lists: (no message) (duration: 00m 13s) [production]
18:17 <Reedy> Created FlaggedRevs tables on ckbwiki [production]
18:11 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Update group0 to 1.24wmf8 [production]
18:06 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf7 [production]
17:00 <reedy> Synchronized wmf-config/: Wrap some long lines, add some docs (duration: 00m 26s) [production]
16:43 <bblack> rebooting lvs3002 [production]