701-750 of 10000 results (30ms)
2014-10-20 §
13:26 <paravoid> cr2-ulsfo: "request chassis mic {off,on}line fpc-slot 1 mic-slot 1" to reboot broken card [production]
12:28 <apergos> upgraded salt master (plus minion) on palladium to 2014.1.11, all neww precise installs will get this version now, other minion upgrades to follow shortly [production]
11:51 <godog> temporarily stopped ircecho/icinga-wm on neon, shower of alarms [production]
11:42 <godog> killed stray/old copy of diamond that was filling up conntrack on virt1000 [production]
09:53 <akosiaris> restarted ocg on ocg1001, ocg1002, ocg1003 [production]
07:07 <_joe_> rolling restart of ocg services [production]
04:29 <springle> removed old /var/log/ocg* on ocg1001 and ocg1003 and forced logrotate, / space critical [production]
03:42 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Oct 20 03:42:24 UTC 2014 (duration 42m 23s) [production]
02:50 <springle> Synchronized wmf-config/db-eqiad.php: repool db1042 (duration: 00m 06s) [production]
02:34 <springle> Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s) [production]
02:28 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-20 02:28:35+00:00 [production]
02:16 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-20 02:16:31+00:00 [production]
2014-10-19 §
03:36 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Oct 19 03:36:03 UTC 2014 (duration 36m 2s) [production]
02:25 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-19 02:25:44+00:00 [production]
02:14 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-19 02:14:23+00:00 [production]
2014-10-18 §
15:09 <springle> Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s) [production]
13:49 <_joe_> restarted apache on the puppetmasters [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Oct 18 03:51:18 UTC 2014 (duration 51m 17s) [production]
02:33 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-18 02:33:01+00:00 [production]
02:20 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-18 02:20:20+00:00 [production]
2014-10-17 §
23:46 <bd808> Ran trebuchet to create initial tag for iegreview/iegreview [production]
23:35 <ori> pooled mw1114 (hhvm api server) to test whether new package resolves overload behavior [production]
23:29 <ori> updating hhvm app servers to 3.3.0-20140925+wmf3 [production]
22:18 <mutante> graceful Apache on antimony - svn fixed, gitblit behind varnish [production]
22:09 <mutante> graceful Apache on neon - icinga and tendril done, ishmael = misc-web [production]
21:47 <mutante> graceful Apache on stat1001 [production]
21:25 <springle> Synchronized wmf-config/db-eqiad.php: repool es1004, es1007, es1010 (duration: 00m 07s) [production]
21:19 <mutante> graceful Apache on netmon1001 [production]
21:13 <mutante> graceful Apache on puppetmaster [production]
20:34 <andrewbogott> removed some stray .zip files from /tmp on ocg1002 [production]
20:16 <ori> disabled puppet on osmium to debug hhvm [production]
15:36 <paravoid> killed tampa config remnants on all cr1/cr2s [production]
15:30 <bblack> restarted puppetmasters [production]
14:44 <_joe_> load test on hhvm done [production]
14:26 <_joe_> load test on the hhvm cluster [production]
10:41 <akosiaris> uploaded apertium 3.3 on apt.wikimedia.org (trusty-wikimedia) [production]
09:17 <_joe_> manually killed long-running stuck processes on ocg1001, moving to the rest of the cluster [production]
08:56 <_joe_> restarted the ocg cluster [production]
08:44 <akosiaris> uploaded cg3 and lttoolbox on apt.wikimedia.org [production]
05:37 <_joe_> depooling both hhvm api appservers [production]
04:52 <springle> xtrabackup clone es1010 to es2008 [production]
04:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Oct 17 04:51:02 UTC 2014 (duration 51m 1s) [production]
04:17 <springle> Synchronized wmf-config/db-eqiad.php: reduce load on db1059 (duration: 00m 21s) [production]
04:05 <springle> upgrade es1010 to trusty (clone failed, needs trusty) [production]
03:54 <springle> xtrabackup clone es1010 to es2008 [production]
03:53 <springle> xtrabackup clone es1007 to es2006 [production]
03:37 <springle> Synchronized wmf-config/db-eqiad.php: depool es1007 and es1010 (duration: 00m 09s) [production]
03:13 <LocalisationUpdate> completed (1.25wmf4) at 2014-10-17 03:13:33+00:00 [production]
02:39 <LocalisationUpdate> completed (1.25wmf3) at 2014-10-17 02:39:26+00:00 [production]
01:16 <mutante> powering up server formerly known as cp1001 [production]