|
2015-06-08
§
|
| 19:18 |
<jynus> |
RAID degradation (disk failure) on s5 master (db1058), no production impact, replacement on the way |
[production] |
| 17:13 |
<ottomata> |
restarted eventlogging services on eventlog1001 after disabling kafka pieces |
[production] |
| 16:13 |
<_joe_> |
powercycling tmh1001, console blank, unresponsive to pings |
[production] |
| 16:00 |
<thcipriani> |
Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia, for real [[gerrit:216719]] (duration: 01m 07s) |
[production] |
| 15:58 |
<thcipriani> |
Synchronized commonsuploads.dblist: SWAT: Revert Temporarily re-enable uploads on Marathi Wikipedia [[gerrit:216719]] (duration: 01m 08s) |
[production] |
| 15:40 |
<thcipriani> |
Synchronized php-1.26wmf8/extensions/Cite: SWAT: Revert Do all of Cite's real work during unstrip and followup [[gerrit:216715]] (duration: 01m 08s) |
[production] |
| 15:19 |
<Coren> |
T96063: process halted for now as store/backup is unmovable and on slice5 |
[production] |
| 15:17 |
<thcipriani> |
Synchronized w/static/images/project-logos/pflwiki.png: SWAT: Fix transparency of pflwiki logo [[gerrit:216595]] (duration: 01m 08s) |
[production] |
| 15:15 |
<akosiaris> |
disabled ircecho on neon for a while |
[production] |
| 14:53 |
<Coren> |
T96063: starting pvmove from slice5 to slice2 |
[production] |
| 14:48 |
<Coren> |
T96063: dropped volume slice1 from vg store |
[production] |
| 14:46 |
<Coren> |
T96063: dropped store/project |
[production] |
| 14:44 |
<Coren> |
starting https://phabricator.wikimedia.org/T96063 on labstore1001 |
[production] |
| 14:24 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool es1005 (duration: 01m 08s) |
[production] |
| 14:23 |
<Coren> |
rsync in progress between labstore1001:store/backup and labstore1002:backup/backup (at ionice idle) |
[production] |
| 14:13 |
<Coren> |
created store/backup snapshot on labstore1001 for backup copy |
[production] |
| 14:12 |
<hashar> |
clearing disk space on trusty 1011 and 1012 |
[releng] |
| 14:12 |
<hashar> |
clearing disk space on trusty 1011 and 1012 |
[releng] |
| 13:03 |
<moritzm> |
added strongswan_5.3.0-1+wmf2 to jessie-wikimedia on carbon |
[production] |
| 11:42 |
<_joe_> |
purging squid cache on carbon |
[production] |
| 11:26 |
<moritzm> |
updated mc2* to 2:2.8.17-1+deb8u1 |
[production] |
| 10:55 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool es1007 (duration: 01m 08s) |
[production] |
| 10:27 |
<akosiaris> |
disabled puppet on uranium, investigating ganglia problems |
[production] |
| 10:05 |
<akosiaris> |
ganglia gmetad problems |
[production] |
| 08:56 |
<hashar> |
rebooted trusty-1013 trusty-1015 ( https://phabricator.wikimedia.org/T101658 ) and repooled them in Jenkins |
[releng] |
| 08:48 |
<hashar> |
rebooting integration-slave-trusty-1012 (stalled can't login) |
[releng] |
| 05:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 8 05:24:08 UTC 2015 (duration 24m 7s) |
[production] |
| 04:30 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/216520 |
[releng] |
| 02:26 |
<LocalisationUpdate> |
completed (1.26wmf8) at 2015-06-08 02:25:12+00:00 |
[production] |
| 02:21 |
<l10nupdate> |
Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 07s) |
[production] |
| 00:40 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/216600 |
[releng] |
|
2015-06-07
§
|
| 23:27 |
<godog> |
reboot ms-be2008 sdg failed, xfs unhappy |
[production] |
| 20:43 |
<Krinkle> |
Rebooting integration-slave-trusty-1015 to see if it comes back so we can inspect logs (T101658) |
[releng] |
| 20:16 |
<Krinkle> |
Per Yuvi's advice, disabled "Shared project storage" (/data/project NFS mount) for the integration project. Mostly unused. Two existing directories were archived to /home/krinkle/integration-nfs-data-project/ |
[releng] |
| 17:51 |
<Krinkle> |
integration-slave-trusty-1012, trusty-1013 and 1015 unresponsive to pings or ssh. Other trusty slaves still reachable. |
[releng] |
| 07:03 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1073, warm up (duration: 01m 09s) |
[production] |
| 05:16 |
<andrewbogott> |
we did a whole lot of things to labstore1001 while morebots was away |
[production] |
| 05:14 |
<andrewbogott> |
service nfs-kernel-server restart on labstore1001 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.26wmf8) at 2015-06-07 02:25:13+00:00 |
[production] |
| 02:21 |
<l10nupdate> |
Synchronized php-1.26wmf8/cache/l10n: (no message) (duration: 07m 09s) |
[production] |