1301-1350 of 10000 results (25ms)
2014-09-02 §
10:00 <_joe_> depooling mw1192, high CPU temperatures; we may need to check fan status [production]
07:20 <_joe_> powercycling mw1192, blank console, unresponsive [production]
07:02 <springle> removed all-but-latest large slow logs on elastic1004 and elastic1014 [production]
06:22 <springle> removed txt files filling up db1047 /tmp, looked like analytics SELECT INTO OUTFILE, dated mid-August [production]
05:58 <springle> dump s3 db1035 to db1069:3313 [production]
05:37 <springle> Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 08s) [production]
04:59 <springle> Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 07s) [production]
03:35 <springle> Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s) [production]
03:16 <LocalisationUpdate> ResourceLoader cache refresh completed at Tue Sep 2 03:15:22 UTC 2014 (duration 15m 21s) [production]
02:53 <springle> restarted dbstore1002 mysqld for upgrade [production]
02:26 <LocalisationUpdate> completed (1.24wmf19) at 2014-09-02 02:25:36+00:00 [production]
02:15 <LocalisationUpdate> completed (1.24wmf18) at 2014-09-02 02:14:30+00:00 [production]
2014-09-01 §
23:00 <Krinkle> Running extensions/GlobalCssJs/removeOldManualUserPages.php per m:GlobalCssJs [production]
21:50 <ori> disabled gerrit account Caothu9669; spam [production]
19:12 <Reedy> Deleted php-1.24wmf[6-8] from apaches via dsh [production]
19:01 <reedy> Purged l10n cache for 1.24wmf13 [production]
19:01 <reedy> Purged l10n cache for 1.24wmf14 [production]
19:00 <reedy> Purged l10n cache for 1.24wmf15 [production]
18:59 <reedy> Purged l10n cache for 1.24wmf16 [production]
18:58 <reedy> Purged l10n cache for 1.24wmf17 [production]
16:44 <ottomata> removed some large slow query logs from elastic* nodes, need to look into this... [production]
12:10 <springle> Synchronized wmf-config/db-eqiad.php: repool db1044, take 2 (duration: 00m 06s) [production]
12:04 <springle> Synchronized wmf-config/db-eqiad.php: depool db1044 (duration: 00m 06s) [production]
11:31 <springle> Synchronized wmf-config/db-eqiad.php: pool db1044, warm up (duration: 00m 06s) [production]
09:31 <springle> Synchronized wmf-config/db-eqiad.php: repool db1027 (duration: 00m 07s) [production]
07:27 <godog> deploy latest ring to swift eqiad-prod [production]
07:11 <godog> powercycle ms-be1010 "cpu soft lockup" on console [production]
05:28 <springle> xtrabackup clone db1027 to db1044 [production]
05:26 <springle> Synchronized wmf-config/db-eqiad.php: depool db1027 while cloning (duration: 00m 07s) [production]
03:15 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Sep 1 03:14:22 UTC 2014 (duration 14m 21s) [production]
02:28 <LocalisationUpdate> completed (1.24wmf19) at 2014-09-01 02:27:39+00:00 [production]
02:17 <LocalisationUpdate> completed (1.24wmf18) at 2014-09-01 02:16:18+00:00 [production]
2014-08-31 §
19:56 <hashar> Jenkins updated HHVM to (3.3-dev+20140728+wmf5) over (3.3-dev+20140728+wmf4) [production]
14:38 <bblack> restarted apache on strontium [production]
14:34 <bblack> restarted apache on tungsten, machine is overloaded [production]
03:15 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Aug 31 03:14:41 UTC 2014 (duration 14m 40s) [production]
02:29 <LocalisationUpdate> completed (1.24wmf19) at 2014-08-31 02:28:42+00:00 [production]
02:18 <LocalisationUpdate> completed (1.24wmf18) at 2014-08-31 02:17:24+00:00 [production]
02:00 <ori> Stopped HHVM jobrunner and disabled Puppet on mw1053 due to bug 70177. [production]
2014-08-30 §
08:18 <godog> restart mailman on sodium, pending https://gerrit.wikimedia.org/r/#/c/156766/ [production]
08:01 <springle> Synchronized wmf-config/db-eqiad.php: repool db1071, warm up (duration: 00m 07s) [production]
06:33 <jgage> analytics1021 back in service after election [production]
06:14 <jgage> upgraded & rebooted analytics1021 [production]
06:01 <springle> Synchronized wmf-config/db-eqiad.php: return db1037 to normal load (duration: 00m 06s) [production]
05:03 <springle> Synchronized wmf-config/db-eqiad.php: repool es1001 (duration: 00m 06s) [production]
04:06 <springle> upgrade es1001 to mariadb 10 [production]
03:56 <springle> xtrabackup clone db1037 to db1071 [production]
03:55 <springle> Synchronized wmf-config/db-eqiad.php: reduce db1037 load while cloning (duration: 00m 06s) [production]
03:31 <springle> Synchronized wmf-config/db-eqiad.php: pool db1073, depool db1071 (duration: 00m 07s) [production]
03:19 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Aug 30 03:17:57 UTC 2014 (duration 17m 56s) [production]