|
2014-07-18
§
|
| 17:25 |
<mutante> |
temp. stopped icinga-wm to avoid channel spam |
[production] |
| 17:24 |
<mutante> |
puppetmaster on strontium had 'Unexpected error in mod_passenger" causing puppet fails all over the place with error 500 on master, resumed normal after graceful |
[production] |
| 17:21 |
<mutante> |
graceful'ed apache on strontium |
[production] |
| 14:37 |
<godog> |
rolling reload of proxy-server on swift ms-fe1* to pick up changes |
[production] |
| 13:19 |
<_joe_> |
re-enabling puppet, applying on a sample of hosts created no change according to my tests. |
[production] |
| 13:13 |
<_joe_> |
temporarily disabling puppet on mw servers, will re-enable when I'm done with testing (again) the change |
[production] |
| 11:20 |
<godog> |
restart proxy-server on ms-fe1003, as suspected it wasn't running the latest version |
[production] |
| 11:14 |
<godog> |
restart proxy-server on ms-fe1003, double checking for a change in numbers reported to graphite |
[production] |
| 10:04 |
<godog> |
stagger reload swift {account,object,container} server in ms-be.eqiad to pick up recon changes |
[production] |
| 06:01 |
<AaronSchulz> |
Updated /srv/deployment/jobrunner to 4cddd5033efadf431e138c399b5d86542e32f196 |
[production] |
| 03:55 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jul 18 03:53:55 UTC 2014 (duration 53m 54s) |
[production] |
| 03:22 |
<ori> |
Updated jobrunner to d9520c9 and restarted service on all jobrunners |
[production] |
| 03:09 |
<LocalisationUpdate> |
completed (1.24wmf14) at 2014-07-18 03:08:02+00:00 |
[production] |
| 02:45 |
<springle> |
Synchronized wmf-config/db-eqiad.php: Repool db1021, context RT 7916, warm up (duration: 00m 08s) |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-18 02:36:54+00:00 |
[production] |
|
2014-07-17
§
|
| 23:49 |
<mwalker> |
Finished scap: SWAT for {{gerrit|146651}}, {{gerrit|147102}}, {{gerrit|146925}}, {{gerrit|147331}}, {{gerrit|147332}}, and {{gerrit|147206}} |
[production] |
| 23:19 |
<mwalker> |
Started scap: SWAT for {{gerrit|146651}}, {{gerrit|147102}}, {{gerrit|146925}}, {{gerrit|147331}}, {{gerrit|147332}}, and {{gerrit|147206}} |
[production] |
| 21:02 |
<csteipp> |
deployed fix for bug68187 |
[production] |
| 20:29 |
<ori> |
updated jobrunner to 71d84ea18d and restarted service |
[production] |
| 18:36 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf14 |
[production] |
| 18:30 |
<springle> |
db1021 raid write-cache failure, BBU at 9% |
[production] |
| 18:15 |
<springle> |
db1021 disabled sync_binlog, thread tied up on fsync |
[production] |
| 18:11 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf13 |
[production] |
| 18:09 |
<reedy> |
Synchronized wmf-config/: De-pool db1021 due to increasing replag (duration: 00m 14s) |
[production] |
| 17:40 |
<reedy> |
Finished scap: testwiki to 1.24wmf14 take 2 (duration: 33m 02s) |
[production] |
| 17:30 |
<Jeff_Green> |
payments1002 dist upgrade & reboot |
[production] |
| 17:21 |
<mutante> |
nickel (ganglia) apt-get upgrading packages |
[production] |
| 17:14 |
<Jeff_Green> |
dist-upgrade and reboot payments1003 |
[production] |
| 17:07 |
<reedy> |
Started scap: testwiki to 1.24wmf14 take 2 |
[production] |
| 17:03 |
<RobH> |
payments4 is kernel updating (per jgreen) |
[production] |
| 17:02 |
<reedy> |
Started scap: testwiki to 1.24wmf14 |
[production] |
| 15:05 |
<manybubbles> |
Synchronized php-1.24wmf13/extensions/MultimediaViewer/: SWAT - Moving repo icon back to the right-hand side in Media Viewer (duration: 00m 05s) |
[production] |
| 15:03 |
<manybubbles> |
Synchronized wmf-config/CommonSettings-labs.php: SWAT deploy to keep us synced, but this is a noop in prod. only anything in beta. (duration: 00m 05s) |
[production] |
| 07:27 |
<springle> |
mariadb 10 on labsdb1002:3309 cloning s5 from sanitarium db1054:3308 |
[production] |
| 03:33 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 17 03:32:25 UTC 2014 (duration 32m 24s) |
[production] |
| 02:47 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-17 02:46:24+00:00 |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-17 02:23:08+00:00 |
[production] |
|
2014-07-16
§
|
| 23:55 |
<maxsem> |
Synchronized private: Clean up old mobile cruft (duration: 00m 05s) |
[production] |
| 23:18 |
<yurik> |
Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 03s) |
[production] |
| 23:13 |
<yurik> |
Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 14s) |
[production] |
| 22:34 |
<andrewbogott> |
temporarily fixed puppet on tin by restarting salt-master and salt-minion. A proper fix would involve upgrading to a salt version that fixes https://github.com/saltstack/salt/issues/6306 |
[production] |
| 22:29 |
<yurik> |
Synchronized php-1.24wmf13/extensions/ZeroBanner: (no message) (duration: 03m 55s) |
[production] |
| 22:27 |
<ori> |
restarted jobrunner service on all job runners |
[production] |
| 22:18 |
<yurik> |
Synchronized php-1.24wmf12/extensions/ZeroBanner: (no message) (duration: 04m 31s) |
[production] |
| 21:50 |
<AaronSchulz> |
Updated job runners to 186b9b33 |
[production] |
| 21:08 |
<legoktm> |
clearing Magog the Ogre's watchlist on enwp per request (173668 entries) |
[production] |
| 21:01 |
<yurik> |
Synchronized php-1.24wmf13/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 53s) |
[production] |
| 20:56 |
<yurik> |
Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 04m 54s) |
[production] |
| 20:22 |
<subbu> |
deploy parsoid 060dcb54 |
[production] |
| 19:56 |
<ottomata> |
reenabling puppet on analytics1027 |
[production] |