251-300 of 10000 results (19ms)
2014-10-20 §
07:07 <_joe_> rolling restart of ocg services [production]
04:29 <springle> removed old /var/log/ocg* on ocg1001 and ocg1003 and forced logrotate, / space critical [production]
03:42 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Oct 20 03:42:24 UTC 2014 (duration 42m 23s) [production]
02:50 <springle> Synchronized wmf-config/db-eqiad.php: repool db1042 (duration: 00m 06s) [production]
02:34 <springle> Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s) [production]
02:28 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-20 02:28:35+00:00 [production]
02:16 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-20 02:16:31+00:00 [production]
2014-10-19 §
03:36 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Oct 19 03:36:03 UTC 2014 (duration 36m 2s) [production]
02:25 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-19 02:25:44+00:00 [production]
02:14 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-19 02:14:23+00:00 [production]
2014-10-18 §
15:09 <springle> Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s) [production]
13:49 <_joe_> restarted apache on the puppetmasters [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Oct 18 03:51:18 UTC 2014 (duration 51m 17s) [production]
02:33 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-18 02:33:01+00:00 [production]
02:20 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-18 02:20:20+00:00 [production]
2014-10-17 §
23:46 <bd808> Ran trebuchet to create initial tag for iegreview/iegreview [production]
23:35 <ori> pooled mw1114 (hhvm api server) to test whether new package resolves overload behavior [production]
23:29 <ori> updating hhvm app servers to 3.3.0-20140925+wmf3 [production]
22:18 <mutante> graceful Apache on antimony - svn fixed, gitblit behind varnish [production]
22:09 <mutante> graceful Apache on neon - icinga and tendril done, ishmael = misc-web [production]
21:47 <mutante> graceful Apache on stat1001 [production]
21:25 <springle> Synchronized wmf-config/db-eqiad.php: repool es1004, es1007, es1010 (duration: 00m 07s) [production]
21:19 <mutante> graceful Apache on netmon1001 [production]
21:13 <mutante> graceful Apache on puppetmaster [production]
20:34 <andrewbogott> removed some stray .zip files from /tmp on ocg1002 [production]
20:16 <ori> disabled puppet on osmium to debug hhvm [production]
15:36 <paravoid> killed tampa config remnants on all cr1/cr2s [production]
15:30 <bblack> restarted puppetmasters [production]
14:44 <_joe_> load test on hhvm done [production]
14:26 <_joe_> load test on the hhvm cluster [production]
10:41 <akosiaris> uploaded apertium 3.3 on apt.wikimedia.org (trusty-wikimedia) [production]
09:17 <_joe_> manually killed long-running stuck processes on ocg1001, moving to the rest of the cluster [production]
08:56 <_joe_> restarted the ocg cluster [production]
08:44 <akosiaris> uploaded cg3 and lttoolbox on apt.wikimedia.org [production]
05:37 <_joe_> depooling both hhvm api appservers [production]
04:52 <springle> xtrabackup clone es1010 to es2008 [production]
04:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Oct 17 04:51:02 UTC 2014 (duration 51m 1s) [production]
04:17 <springle> Synchronized wmf-config/db-eqiad.php: reduce load on db1059 (duration: 00m 21s) [production]
04:05 <springle> upgrade es1010 to trusty (clone failed, needs trusty) [production]
03:54 <springle> xtrabackup clone es1010 to es2008 [production]
03:53 <springle> xtrabackup clone es1007 to es2006 [production]
03:37 <springle> Synchronized wmf-config/db-eqiad.php: depool es1007 and es1010 (duration: 00m 09s) [production]
03:13 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-17 03:13:33+00:00 [production]
02:39 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-17 02:39:26+00:00 [production]
01:16 <mutante> powering up server formerly known as cp1001 [production]
00:47 <hoo> Synchronized php-1.25wmf4/extensions/Wikidata/: Fix ORMTable usage, IE 11 freeze bug and adopt to further core changes (duration: 00m 14s) [production]
2014-10-16 §
22:15 <K4-713> undo last payments settings change [production]
22:09 <K4-713> payments settings updates [production]
20:15 <reedy> Synchronized database lists: (no message) (duration: 00m 18s) [production]
20:15 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 17s) [production]