|
2014-09-02
§
|
| 10:00 |
<_joe_> |
depooling mw1192, high CPU temperatures; we may need to check fan status |
[production] |
| 07:20 |
<_joe_> |
powercycling mw1192, blank console, unresponsive |
[production] |
| 07:02 |
<springle> |
removed all-but-latest large slow logs on elastic1004 and elastic1014 |
[production] |
| 06:22 |
<springle> |
removed txt files filling up db1047 /tmp, looked like analytics SELECT INTO OUTFILE, dated mid-August |
[production] |
| 05:58 |
<springle> |
dump s3 db1035 to db1069:3313 |
[production] |
| 05:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 08s) |
[production] |
| 04:59 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 07s) |
[production] |
| 03:35 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s) |
[production] |
| 03:16 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 2 03:15:22 UTC 2014 (duration 15m 21s) |
[production] |
| 02:53 |
<springle> |
restarted dbstore1002 mysqld for upgrade |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf19) at 2014-09-02 02:25:36+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-09-02 02:14:30+00:00 |
[production] |
|
2014-09-01
§
|
| 23:00 |
<Krinkle> |
Running extensions/GlobalCssJs/removeOldManualUserPages.php per m:GlobalCssJs |
[production] |
| 21:50 |
<ori> |
disabled gerrit account Caothu9669; spam |
[production] |
| 19:12 |
<Reedy> |
Deleted php-1.24wmf[6-8] from apaches via dsh |
[production] |
| 19:01 |
<reedy> |
Purged l10n cache for 1.24wmf13 |
[production] |
| 19:01 |
<reedy> |
Purged l10n cache for 1.24wmf14 |
[production] |
| 19:00 |
<reedy> |
Purged l10n cache for 1.24wmf15 |
[production] |
| 18:59 |
<reedy> |
Purged l10n cache for 1.24wmf16 |
[production] |
| 18:58 |
<reedy> |
Purged l10n cache for 1.24wmf17 |
[production] |
| 16:44 |
<ottomata> |
removed some large slow query logs from elastic* nodes, need to look into this... |
[production] |
| 12:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1044, take 2 (duration: 00m 06s) |
[production] |
| 12:04 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1044 (duration: 00m 06s) |
[production] |
| 11:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: pool db1044, warm up (duration: 00m 06s) |
[production] |
| 09:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1027 (duration: 00m 07s) |
[production] |
| 07:27 |
<godog> |
deploy latest ring to swift eqiad-prod |
[production] |
| 07:11 |
<godog> |
powercycle ms-be1010 "cpu soft lockup" on console |
[production] |
| 05:28 |
<springle> |
xtrabackup clone db1027 to db1044 |
[production] |
| 05:26 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1027 while cloning (duration: 00m 07s) |
[production] |
| 03:15 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Sep 1 03:14:22 UTC 2014 (duration 14m 21s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.24wmf19) at 2014-09-01 02:27:39+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-09-01 02:16:18+00:00 |
[production] |
|
2014-08-30
§
|
| 08:18 |
<godog> |
restart mailman on sodium, pending https://gerrit.wikimedia.org/r/#/c/156766/ |
[production] |
| 08:01 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1071, warm up (duration: 00m 07s) |
[production] |
| 06:33 |
<jgage> |
analytics1021 back in service after election |
[production] |
| 06:14 |
<jgage> |
upgraded & rebooted analytics1021 |
[production] |
| 06:01 |
<springle> |
Synchronized wmf-config/db-eqiad.php: return db1037 to normal load (duration: 00m 06s) |
[production] |
| 05:03 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1001 (duration: 00m 06s) |
[production] |
| 04:06 |
<springle> |
upgrade es1001 to mariadb 10 |
[production] |
| 03:56 |
<springle> |
xtrabackup clone db1037 to db1071 |
[production] |
| 03:55 |
<springle> |
Synchronized wmf-config/db-eqiad.php: reduce db1037 load while cloning (duration: 00m 06s) |
[production] |
| 03:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: pool db1073, depool db1071 (duration: 00m 07s) |
[production] |
| 03:19 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Aug 30 03:17:57 UTC 2014 (duration 17m 56s) |
[production] |