|
2014-06-19
§
|
| 14:30 |
<cmjohnson1> |
replacing failed disk slot3 es1006 |
[production] |
| 13:01 |
<_joe_> |
re-enable puppet on lvs1003 |
[production] |
| 11:26 |
<reedy> |
Synchronized wmf-config/: touch (duration: 00m 15s) |
[production] |
| 11:25 |
<reedy> |
Synchronized commonsuploads.dblist: (no message) (duration: 00m 15s) |
[production] |
| 11:00 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s) |
[production] |
| 10:53 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 15s) |
[production] |
| 10:53 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 15s) |
[production] |
| 10:28 |
<Reedy> |
manually ran sync-common tin on fenari |
[production] |
| 10:09 |
<reedy> |
Synchronized docroot/noc: (no message) (duration: 00m 15s) |
[production] |
| 10:07 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 14s) |
[production] |
| 10:04 |
<reedy> |
Synchronized wmf-config/: I248fa7b98a8a0eea943c6643d1bf9c2ed36296b8 (duration: 00m 15s) |
[production] |
| 03:34 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jun 19 03:33:36 UTC 2014 (duration 33m 35s) |
[production] |
| 02:46 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-19 02:45:51+00:00 |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-19 02:23:42+00:00 |
[production] |
|
2014-06-18
§
|
| 23:09 |
<awight> |
update crm from 26460d6eaec26861661322df8e9f07a8b0519677 to f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c |
[production] |
| 23:05 |
<maxsem> |
Synchronized php-1.24wmf9/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/140563/ (duration: 00m 03s) |
[production] |
| 23:03 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/140250/ (duration: 00m 04s) |
[production] |
| 22:30 |
<bblack> |
rebooting lvs1004 + lvs1005 |
[production] |
| 22:10 |
<bblack> |
turning lvs1003 pybal back on |
[production] |
| 21:52 |
<bblack> |
disable pybal on lvs1003, since 1006 seems to have all its interfaces :P |
[production] |
| 21:34 |
<bblack> |
rebooting lvs1003 for kernel/bios stuff |
[production] |
| 21:00 |
<bblack> |
rebooting lvs1006 for kernel/bios stuff |
[production] |
| 20:23 |
<subbu> |
deployed Parsoid 88a61f81 (deploy repo sha 470a5ef2) |
[production] |
| 17:39 |
<yurik> |
Synchronized docroot/bits/WikipediaMobileFirefoxOS/: (no message) (duration: 01m 09s) |
[production] |
| 17:36 |
<yurik> |
Synchronized php-1.24wmf9/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 14s) |
[production] |
| 17:32 |
<yurik> |
Synchronized php-1.24wmf8/extensions/: Updating JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 15s) |
[production] |
| 17:26 |
<yurik> |
Synchronized docroot/bits/WikipediaMobileFirefoxOS/: (no message) (duration: 01m 04s) |
[production] |
| 17:10 |
<RobH> |
magnesium back to proper function |
[production] |
| 17:09 |
<RobH> |
apache2ctl restart on magnesium, racktables wasn't working |
[production] |
| 16:24 |
<bblack> |
rebooting lvs4001 for kenerl + num_queues |
[production] |
| 16:19 |
<bblack> |
rebooting lvs4002 for kenerl + num_queues |
[production] |
| 15:20 |
<bblack> |
rebooting lvs4003 for kernel / num_queues updates |
[production] |
| 15:17 |
<bblack> |
rebooting lvs4004 for kernel / num_queues updates |
[production] |
| 15:10 |
<anomie> |
Synchronized php-1.24wmf9/extensions/Scribunto/engines/LuaCommon/SiteLibrary.php: SWAT: Fix Scribunto-related exceptions on testwiki [[gerrit:140370]] (duration: 00m 14s) |
[production] |
| 13:40 |
<_joe_> |
restarted profiler-to-carbon, stuck (again) waiting for mwprof |
[production] |
| 13:25 |
<springle> |
script rt-7708.pl hitting m2-master eventlogging from terbium for RT #7708. fine to kill if necessary |
[production] |
| 10:01 |
<hashar> |
Updated our Jenkins job builder fork: 8cbc93a..416ee7d |
[production] |
| 08:26 |
<_joe_> |
disk is gone, powering down ms-be1007, opening ticket for disk replacement |
[production] |
| 08:24 |
<_joe_> |
stopped swift on ms-be1007, unmounting volume to check for repair |
[production] |
| 06:01 |
<springle> |
restarted gmetad on nickel while unbreaking the mysql graphs I broke on ganglia |
[production] |
| 04:30 |
<ori> |
enabled puppet on polonium (was disabled but nothing in SAL) |
[production] |
| 02:59 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 18 02:58:22 UTC 2014 (duration 58m 21s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-18 02:25:03+00:00 |
[production] |
| 02:23 |
<MaxSem> |
searchidx1001 outta sync - running sync-common |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-18 02:13:34+00:00 |
[production] |
| 02:05 |
<Krinkle> |
Nevermind, graphite.wikimedia.org going down is due to overload which recovers eventually (it just has). Has become SNAFU/FIXME. |
[production] |
| 02:02 |
<Krinkle> |
graphite.wikimedia.org is down with HTTP 502 Bad Gateway errors |
[production] |
| 01:49 |
<ori> |
puppet freshness on tungsten and stat1001 can be fixed with https://gerrit.wikimedia.org/r/#/c/140269/ |
[production] |