|
2014-07-19
§
|
| 05:02 |
<Krinkle> |
Ungracefully restarting Zuul to clear the items stuck in the queue (picked a moment with no real items waiting in the queue). |
[production] |
| 03:25 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1001 (duration: 00m 06s) |
[production] |
| 02:57 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jul 19 02:56:30 UTC 2014 (duration 56m 29s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf14) at 2014-07-19 02:26:26+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-19 02:15:20+00:00 |
[production] |
| 01:09 |
<awight> |
Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s) |
[production] |
| 01:08 |
<awight> |
Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s) |
[production] |
| 01:07 |
<awight> |
Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 4) (duration: 00m 04s) |
[production] |
| 00:36 |
<awight> |
Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s) |
[production] |
| 00:36 |
<awight> |
Synchronized php-1.24wmf13/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s) |
[production] |
| 00:36 |
<awight> |
Synchronized php-1.24wmf12/extensions/FundraisingTranslateWorkflow: update FundraisingTranslateWorkflow submodule content (take 3) (duration: 00m 04s) |
[production] |
|
2014-07-18
§
|
| 22:04 |
<awight> |
Synchronized php-1.24wmf14: update FundraisingTranslateWorkflow submodule (take 2) (duration: 00m 58s) |
[production] |
| 22:03 |
<awight> |
updated /a/common/php-1.24wmf14 to {{Gerrit|I1036dae02}}: Update mediawiki/core/vendor to head to 1.24wmf14 |
[production] |
| 21:00 |
<awight> |
Synchronized php-1.24wmf13: update FundraisingTranslateWorkflow submodule (duration: 01m 04s) |
[production] |
| 20:59 |
<awight> |
for the record, I actually updated to ade90e0e22492d87e6069db3a359b22ef56401a6 |
[production] |
| 20:57 |
<awight> |
updated /a/common/php-1.24wmf13 to {{Gerrit|Id3462554b}}: Made --maxtime a soft limit again |
[production] |
| 20:50 |
<awight> |
Synchronized php-1.24wmf12: update FundraisingTranslateWorkflow submodule (duration: 00m 49s) |
[production] |
| 20:49 |
<awight> |
Synchronized php-1.24wmf12: update FundraisingTranslateWorkflow submodule (duration: 00m 21s) |
[production] |
| 20:48 |
<awight> |
updated /a/common/php-1.24wmf12 to {{Gerrit|Idf3f49941}}: Updating ZeroBanner |
[production] |
| 20:41 |
<MaxSem> |
Load testing GeoData |
[production] |
| 19:11 |
<mutante> |
restarted apache on strontium.. sigh |
[production] |
| 18:17 |
<aaron> |
Synchronized php-1.24wmf13/maintenance/runJobs.php: ae053860dc36a07f05ab9e31299f2da0d2f66e85 (duration: 00m 03s) |
[production] |
| 18:16 |
<aaron> |
Synchronized php-1.24wmf14/maintenance/runJobs.php: 684c21c325370aa3baac631ae9a006fc8861b952 (duration: 00m 03s) |
[production] |
| 18:05 |
<aaron> |
Synchronized wmf-config/jobqueue-eqiad.php: Set "daemonized" flag for the redis job queue (duration: 00m 04s) |
[production] |
| 17:33 |
<cmjohnson> |
replacing disk 2 es1005 |
[production] |
| 17:25 |
<mutante> |
temp. stopped icinga-wm to avoid channel spam |
[production] |
| 17:24 |
<mutante> |
puppetmaster on strontium had 'Unexpected error in mod_passenger" causing puppet fails all over the place with error 500 on master, resumed normal after graceful |
[production] |
| 17:21 |
<mutante> |
graceful'ed apache on strontium |
[production] |
| 14:37 |
<godog> |
rolling reload of proxy-server on swift ms-fe1* to pick up changes |
[production] |
| 13:19 |
<_joe_> |
re-enabling puppet, applying on a sample of hosts created no change according to my tests. |
[production] |
| 13:13 |
<_joe_> |
temporarily disabling puppet on mw servers, will re-enable when I'm done with testing (again) the change |
[production] |
| 11:20 |
<godog> |
restart proxy-server on ms-fe1003, as suspected it wasn't running the latest version |
[production] |
| 11:14 |
<godog> |
restart proxy-server on ms-fe1003, double checking for a change in numbers reported to graphite |
[production] |
| 10:04 |
<godog> |
stagger reload swift {account,object,container} server in ms-be.eqiad to pick up recon changes |
[production] |
| 06:01 |
<AaronSchulz> |
Updated /srv/deployment/jobrunner to 4cddd5033efadf431e138c399b5d86542e32f196 |
[production] |
| 03:55 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jul 18 03:53:55 UTC 2014 (duration 53m 54s) |
[production] |
| 03:22 |
<ori> |
Updated jobrunner to d9520c9 and restarted service on all jobrunners |
[production] |
| 03:09 |
<LocalisationUpdate> |
completed (1.24wmf14) at 2014-07-18 03:08:02+00:00 |
[production] |
| 02:45 |
<springle> |
Synchronized wmf-config/db-eqiad.php: Repool db1021, context RT 7916, warm up (duration: 00m 08s) |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.24wmf13) at 2014-07-18 02:36:54+00:00 |
[production] |
|
2014-07-17
§
|
| 23:49 |
<mwalker> |
Finished scap: SWAT for {{gerrit|146651}}, {{gerrit|147102}}, {{gerrit|146925}}, {{gerrit|147331}}, {{gerrit|147332}}, and {{gerrit|147206}} |
[production] |
| 23:19 |
<mwalker> |
Started scap: SWAT for {{gerrit|146651}}, {{gerrit|147102}}, {{gerrit|146925}}, {{gerrit|147331}}, {{gerrit|147332}}, and {{gerrit|147206}} |
[production] |
| 21:02 |
<csteipp> |
deployed fix for bug68187 |
[production] |
| 20:29 |
<ori> |
updated jobrunner to 71d84ea18d and restarted service |
[production] |
| 18:36 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf14 |
[production] |
| 18:30 |
<springle> |
db1021 raid write-cache failure, BBU at 9% |
[production] |
| 18:15 |
<springle> |
db1021 disabled sync_binlog, thread tied up on fsync |
[production] |
| 18:11 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf13 |
[production] |
| 18:09 |
<reedy> |
Synchronized wmf-config/: De-pool db1021 due to increasing replag (duration: 00m 14s) |
[production] |
| 17:40 |
<reedy> |
Finished scap: testwiki to 1.24wmf14 take 2 (duration: 33m 02s) |
[production] |