|
2014-10-20
§
|
| 13:26 |
<paravoid> |
cr2-ulsfo: "request chassis mic {off,on}line fpc-slot 1 mic-slot 1" to reboot broken card |
[production] |
| 12:28 |
<apergos> |
upgraded salt master (plus minion) on palladium to 2014.1.11, all neww precise installs will get this version now, other minion upgrades to follow shortly |
[production] |
| 11:51 |
<godog> |
temporarily stopped ircecho/icinga-wm on neon, shower of alarms |
[production] |
| 11:42 |
<godog> |
killed stray/old copy of diamond that was filling up conntrack on virt1000 |
[production] |
| 09:53 |
<akosiaris> |
restarted ocg on ocg1001, ocg1002, ocg1003 |
[production] |
| 07:07 |
<_joe_> |
rolling restart of ocg services |
[production] |
| 04:29 |
<springle> |
removed old /var/log/ocg* on ocg1001 and ocg1003 and forced logrotate, / space critical |
[production] |
| 03:42 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Oct 20 03:42:24 UTC 2014 (duration 42m 23s) |
[production] |
| 02:50 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1042 (duration: 00m 06s) |
[production] |
| 02:34 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1066 (duration: 00m 06s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.25wmf4) at 2014-10-20 02:28:35+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-20 02:16:31+00:00 |
[production] |
|
2014-10-17
§
|
| 23:46 |
<bd808> |
Ran trebuchet to create initial tag for iegreview/iegreview |
[production] |
| 23:35 |
<ori> |
pooled mw1114 (hhvm api server) to test whether new package resolves overload behavior |
[production] |
| 23:29 |
<ori> |
updating hhvm app servers to 3.3.0-20140925+wmf3 |
[production] |
| 22:18 |
<mutante> |
graceful Apache on antimony - svn fixed, gitblit behind varnish |
[production] |
| 22:09 |
<mutante> |
graceful Apache on neon - icinga and tendril done, ishmael = misc-web |
[production] |
| 21:47 |
<mutante> |
graceful Apache on stat1001 |
[production] |
| 21:25 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool es1004, es1007, es1010 (duration: 00m 07s) |
[production] |
| 21:19 |
<mutante> |
graceful Apache on netmon1001 |
[production] |
| 21:13 |
<mutante> |
graceful Apache on puppetmaster |
[production] |
| 20:34 |
<andrewbogott> |
removed some stray .zip files from /tmp on ocg1002 |
[production] |
| 20:16 |
<ori> |
disabled puppet on osmium to debug hhvm |
[production] |
| 15:36 |
<paravoid> |
killed tampa config remnants on all cr1/cr2s |
[production] |
| 15:30 |
<bblack> |
restarted puppetmasters |
[production] |
| 14:44 |
<_joe_> |
load test on hhvm done |
[production] |
| 14:26 |
<_joe_> |
load test on the hhvm cluster |
[production] |
| 10:41 |
<akosiaris> |
uploaded apertium 3.3 on apt.wikimedia.org (trusty-wikimedia) |
[production] |
| 09:17 |
<_joe_> |
manually killed long-running stuck processes on ocg1001, moving to the rest of the cluster |
[production] |
| 08:56 |
<_joe_> |
restarted the ocg cluster |
[production] |
| 08:44 |
<akosiaris> |
uploaded cg3 and lttoolbox on apt.wikimedia.org |
[production] |
| 05:37 |
<_joe_> |
depooling both hhvm api appservers |
[production] |
| 04:52 |
<springle> |
xtrabackup clone es1010 to es2008 |
[production] |
| 04:51 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Oct 17 04:51:02 UTC 2014 (duration 51m 1s) |
[production] |
| 04:17 |
<springle> |
Synchronized wmf-config/db-eqiad.php: reduce load on db1059 (duration: 00m 21s) |
[production] |
| 04:05 |
<springle> |
upgrade es1010 to trusty (clone failed, needs trusty) |
[production] |
| 03:54 |
<springle> |
xtrabackup clone es1010 to es2008 |
[production] |
| 03:53 |
<springle> |
xtrabackup clone es1007 to es2006 |
[production] |
| 03:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1007 and es1010 (duration: 00m 09s) |
[production] |
| 03:13 |
<LocalisationUpdate> |
completed (1.25wmf4) at 2014-10-17 03:13:33+00:00 |
[production] |
| 02:39 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-17 02:39:26+00:00 |
[production] |
| 01:16 |
<mutante> |
powering up server formerly known as cp1001 |
[production] |