|
2015-08-31
§
|
| 14:43 |
<ebernhardson> |
elasticsearch cluster.routing.allocation.disk.watermark.high set to 75% to force elastic1022 to reduce its disk usage |
[production] |
| 14:41 |
<urandom> |
bouncing Cassandra on restbase1001 to apply temporary GC setting |
[production] |
| 14:06 |
<akosiaris> |
rebooted krypton. was reporting 100% cpu steal time |
[production] |
| 13:40 |
<paravoid> |
running puppet on newly-installed mc2001 |
[production] |
| 13:40 |
<paravoid> |
restarting hhvm on mw1065 |
[production] |
| 11:10 |
<moritzm> |
restart salt-master on palladium |
[production] |
| 10:45 |
<paravoid> |
reenabling asw2-a5-eqiad:xe-0/0/36 (T107635) |
[production] |
| 10:36 |
<godog> |
repool ms-fe1004 |
[production] |
| 10:32 |
<godog> |
repool ms-fe1003 and depool ms-fe1004 for firewall changes |
[production] |
| 10:19 |
<godog> |
update graphite retention policy on files with previous retention and older than 30d T96662 |
[production] |
| 10:18 |
<godog> |
repool ms-fe1002 and depool ms-fe1003 for firewall changes |
[production] |
| 10:05 |
<godog> |
depool ms-fe1002 to apply firewall changes |
[production] |
| 09:55 |
<jynus> |
cloning es1007 mysql data into es1013 (ETA: 5h30m) |
[production] |
| 09:51 |
<godog> |
repool ms-fe1001 |
[production] |
| 09:35 |
<godog> |
depool ms-fe1001 in preparation for ferm changes |
[production] |
| 09:27 |
<godog> |
update graphite retention policy on files with previous retention and older than 60d T96662 |
[production] |
| 09:25 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1007 for maintenance (duration: 00m 13s) |
[production] |
| 08:33 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 12s) |
[production] |
| 04:34 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Aug 31 04:34:14 UTC 2015 (duration 34m 13s) |
[production] |
| 04:05 |
<bblack> |
disabled ipv6 autoconf on neon, flushed old dynamic addr |
[production] |
| 02:32 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-31 02:32:25+00:00 |
[production] |
| 02:29 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 42s) |
[production] |
|
2015-08-30
§
|
| 12:58 |
<godog> |
lvchange -ay labstore/others on labstore1002 |
[production] |
| 12:52 |
<godog> |
start-nfs on labstore1002 |
[production] |
| 12:31 |
<godog> |
lvchange -ay labstore/tools on labstore1002 |
[production] |
| 12:30 |
<godog> |
also disabled puppet on labstore1002 while investigating |
[production] |
| 12:15 |
<godog> |
trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose |
[production] |
| 11:18 |
<YuviPanda> |
powered labstore1002 back up |
[production] |
| 11:17 |
<YuviPanda> |
shut down labstore1002, going to powercycle from mgmt |
[production] |
| 10:34 |
<YuviPanda> |
disabled backups on labstore1002 to prevent overwriting of good backups on 2001 |
[production] |
| 10:07 |
<YuviPanda> |
run start-nfs in labstore1002 |
[production] |
| 09:54 |
<YuviPanda> |
rebooted labstore1002 |
[production] |
| 04:16 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s) |
[production] |
| 02:23 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00 |
[production] |
| 02:20 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s) |
[production] |
|
2015-08-29
§
|
| 15:26 |
<jynus> |
killing idle mysql connections from phabricator and setting wait and interactive timeout to 60 |
[production] |
| 09:30 |
<jynus> |
SCAP failed, cannot depool db1028 |
[production] |
| 09:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) |
[production] |
| 09:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) |
[production] |
| 09:05 |
<jynus> |
about to depool db1028 due to disk issue |
[production] |
| 04:17 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s) |
[production] |
| 02:24 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00 |
[production] |
| 02:21 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s) |
[production] |