|
2014-07-17
§
|
| 18:30 |
<springle> |
db1021 raid write-cache failure, BBU at 9% |
[production] |
| 18:15 |
<springle> |
db1021 disabled sync_binlog, thread tied up on fsync |
[production] |
| 18:11 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf13 |
[production] |
| 18:09 |
<reedy> |
Synchronized wmf-config/: De-pool db1021 due to increasing replag (duration: 00m 14s) |
[production] |
| 17:40 |
<reedy> |
Finished scap: testwiki to 1.24wmf14 take 2 (duration: 33m 02s) |
[production] |
| 17:30 |
<Jeff_Green> |
payments1002 dist upgrade & reboot |
[production] |
| 17:21 |
<mutante> |
nickel (ganglia) apt-get upgrading packages |
[production] |
| 17:14 |
<Jeff_Green> |
dist-upgrade and reboot payments1003 |
[production] |
| 17:07 |
<reedy> |
Started scap: testwiki to 1.24wmf14 take 2 |
[production] |
| 17:03 |
<RobH> |
payments4 is kernel updating (per jgreen) |
[production] |
| 17:02 |
<reedy> |
Started scap: testwiki to 1.24wmf14 |
[production] |
| 15:05 |
<manybubbles> |
Synchronized php-1.24wmf13/extensions/MultimediaViewer/: SWAT - Moving repo icon back to the right-hand side in Media Viewer (duration: 00m 05s) |
[production] |
| 15:03 |
<manybubbles> |
Synchronized wmf-config/CommonSettings-labs.php: SWAT deploy to keep us synced, but this is a noop in prod. only anything in beta. (duration: 00m 05s) |
[production] |
| 07:27 |
<springle> |
mariadb 10 on labsdb1002:3309 cloning s5 from sanitarium db1054:3308 |
[production] |
| 03:33 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 17 03:32:25 UTC 2014 (duration 32m 24s) |
[production] |
| 02:47 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-17 02:46:24+00:00 |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-17 02:23:08+00:00 |
[production] |
|
2014-07-16
§
|
| 23:55 |
<maxsem> |
Synchronized private: Clean up old mobile cruft (duration: 00m 05s) |
[production] |
| 23:18 |
<yurik> |
Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 03s) |
[production] |
| 23:13 |
<yurik> |
Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 14s) |
[production] |
| 22:34 |
<andrewbogott> |
temporarily fixed puppet on tin by restarting salt-master and salt-minion. A proper fix would involve upgrading to a salt version that fixes https://github.com/saltstack/salt/issues/6306 |
[production] |
| 22:29 |
<yurik> |
Synchronized php-1.24wmf13/extensions/ZeroBanner: (no message) (duration: 03m 55s) |
[production] |
| 22:27 |
<ori> |
restarted jobrunner service on all job runners |
[production] |
| 22:18 |
<yurik> |
Synchronized php-1.24wmf12/extensions/ZeroBanner: (no message) (duration: 04m 31s) |
[production] |
| 21:50 |
<AaronSchulz> |
Updated job runners to 186b9b33 |
[production] |
| 21:08 |
<legoktm> |
clearing Magog the Ogre's watchlist on enwp per request (173668 entries) |
[production] |
| 21:01 |
<yurik> |
Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 53s) |
[production] |
| 20:56 |
<yurik> |
Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 54s) |
[production] |
| 20:22 |
<subbu> |
deploy parsoid 060dcb54 |
[production] |
| 19:56 |
<ottomata> |
reenabling puppet on analytics1027 |
[production] |
| 19:21 |
<ottomata> |
temp disabling puppet on analytics1027 |
[production] |
| 17:57 |
<akosiaris> |
clean puppet stored config database for osm-db100{1,2}.eqiad.wmnet, updating icinga |
[production] |
| 16:49 |
<Reedy> |
Restarted jenkins again |
[production] |
| 16:12 |
<Reedy> |
Restarted jenkins |
[production] |
| 16:11 |
<Reedy> |
Killed jenkins |
[production] |
| 14:34 |
<_joe_> |
moving the stale conf-enabled directory away on jobrunners, or when we upgrade to trusty all hell will break loose |
[production] |
| 13:06 |
<oblivian> |
gracefulled all apaches |
[production] |
| 12:15 |
<oblivian> |
gracefulled all apaches |
[production] |
| 12:01 |
<_joe_> |
removed stale files from /etc/apache2/conf-enabled on all mw hosts |
[production] |
| 11:25 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: Take Cirrus as default from more wikis while we figure out load issues (duration: 00m 06s) |
[production] |
| 10:32 |
<_joe_> |
releasing a new apache config to all mediawikis |
[production] |
| 08:54 |
<godog> |
repool ms-fe1004 |
[production] |
| 08:52 |
<godog> |
repool ms-fe1003 and depool ms-fe1004 |
[production] |
| 08:46 |
<godog> |
repool ms-fe1002 and depool ms-fe1003 |
[production] |
| 08:39 |
<godog> |
depool ms-fe1002 for swift upgrade |
[production] |
| 05:54 |
<springle> |
resuming page content model schema changes, osc_host.sh processes on terbium ok to kill in emergency |
[production] |
| 04:23 |
<springle> |
restarted gitblit on antimony |
[production] |
| 03:04 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 16 03:03:41 UTC 2014 (duration 3m 40s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-16 02:26:12+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-16 02:14:32+00:00 |
[production] |