|
2015-05-13
§
|
| 15:21 |
<manybubbles> |
I think the elasticsearch cluster got stuck with alloation disabled after the rolling restart. Funky. Haven't seen that one before. Probably a problem with our instructions. Anyway, unstuck it and recovery is going faster now |
[production] |
| 15:17 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: didn't work, undoing previous sync (duration: 00m 12s) |
[production] |
| 15:15 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: trying something (duration: 00m 12s) |
[production] |
| 14:53 |
<manybubbles> |
elasticsearch restart on elastic1001 going well. cluster still in recovering state as expect. I'll give it an hour to soak. |
[production] |
| 14:48 |
<manybubbles> |
ok - time to start the rolling restart. I'm going to to elastic1001 first non-automated and watch it |
[production] |
| 14:36 |
<manybubbles> |
s/gitfit/gitfat/ oh well |
[production] |
| 14:35 |
<manybubbles> |
first attempt at syncing elasticsearch plugins didn't work 100%. syncing again. gitfit/gitdeploy is betraying me |
[production] |
| 14:32 |
<manybubbles> |
syncing new versions of elsaticsearch plugins to prod. no restarts yet. |
[production] |
| 14:04 |
<aude> |
Synchronized wmf-config/InitialiseSettings.php: Enable usage tracking for Wikisource (duration: 00m 14s) |
[production] |
| 13:57 |
<aude> |
added wbc_entity_usage table on all Wikibase Client wikis |
[production] |
| 13:56 |
<jynus> |
jcrespo Disabling puppet agent in db1009.eqiad in preparation for reinstall |
[production] |
| 13:45 |
<aude> |
Synchronized php-1.26wmf5/extensions/Wikidata: Update maintenance script (duration: 00m 20s) |
[production] |
| 12:45 |
<springle> |
xtrabackup clone db1060 to db1018 |
[production] |
| 12:39 |
<springle> |
upgrade and restart db1060 |
[production] |
| 09:20 |
<jamesofur> |
inserting FDC election encryption key |
[production] |
| 06:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed May 13 06:19:59 UTC 2015 (duration 19m 58s) |
[production] |
| 05:53 |
<springle> |
reinstall db1018 |
[production] |
| 04:50 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1018 (duration: 00m 12s) |
[production] |
| 03:11 |
<LocalisationUpdate> |
completed (1.26wmf5) at 2015-05-13 03:10:31+00:00 |
[production] |
| 03:07 |
<l10nupdate> |
Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 43s) |
[production] |
| 02:46 |
<LocalisationUpdate> |
completed (1.26wmf4) at 2015-05-13 02:45:28+00:00 |
[production] |
| 02:39 |
<l10nupdate> |
Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 10m 08s) |
[production] |
| 01:56 |
<damagecat> |
Started 'jobs' screen in tin to drain refreshLinks for enwiki using --nothrottle (T98621) |
[production] |
| 01:29 |
<legoktm> |
Synchronized wmf-config/CommonSettings.php: Hardcode UploadWizard max upload size - T98933 (duration: 00m 12s) |
[production] |
| 01:23 |
<legoktm> |
Synchronized php-1.26wmf5/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s) |
[production] |
| 01:21 |
<legoktm> |
Synchronized php-1.26wmf4/extensions/GWToolset/: Check php max_file_size limit directly from PHP $_FILES (duration: 00m 12s) |
[production] |
| 01:07 |
<gwicke> |
added commons to supported projects in RESTBase API |
[production] |
| 00:16 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I5ebedfdfb: Set $wgGadgetsCacheType to CACHE_ACCEL (duration: 00m 12s) |
[production] |
| 00:13 |
<ori> |
Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: 914d71f3cc: Temporary hack to drain excess refreshLinks jobs (duration: 00m 14s) |
[production] |
| 00:12 |
<ori> |
Synchronized php-1.26wmf4/extensions/Gadgets: 7539873979: Update Gadgets for cherry-pick (duration: 00m 12s) |
[production] |
| 00:11 |
<ori> |
Synchronized php-1.26wmf5/extensions/Gadgets: cbb9b1e475: Update Gadgets for cherry-pick (duration: 00m 12s) |
[production] |
|
2015-05-12
§
|
| 23:40 |
<ori> |
Upgraded all Apaches to HHVM 3.6.1+dfsg1-1+wm2 and Apache 2.4.7-1ubuntu4.4 |
[production] |
| 23:26 |
<demon> |
Synchronized php-1.26wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 12s) |
[production] |
| 23:24 |
<demon> |
Synchronized php-1.26wmf4/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 11s) |
[production] |
| 23:23 |
<demon> |
Synchronized php-1.26wmf5/includes/jobqueue/jobs/RefreshLinksJob.php: (no message) (duration: 00m 12s) |
[production] |
| 23:23 |
<demon> |
Synchronized php-1.26wmf5/includes/media/DjVu.php: (no message) (duration: 00m 12s) |
[production] |
| 23:18 |
<ori> |
Upgrading more HHVMs; DPKG alerts likely but they will be transient. |
[production] |
| 23:11 |
<demon> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 11s) |
[production] |
| 23:03 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: swat (duration: 00m 12s) |
[production] |
| 21:48 |
<kaldari> |
Finished scap: updating i18n for Gather (1.26wmf5) (duration: 23m 17s) |
[production] |
| 21:25 |
<kaldari> |
Started scap: updating i18n for Gather (1.26wmf5) |
[production] |
| 21:24 |
<kaldari> |
Synchronized php-1.26wmf5/extensions/Gather: Updating Gather for 1.26wmf5 (duration: 00m 12s) |
[production] |
| 21:06 |
<apergos> |
manually installed trigger-trebuchet update on tin after accidental salt upgrade there woops :-D |
[production] |
| 20:56 |
<mutante> |
upgrading salt packages on tin |
[production] |
| 19:50 |
<ori> |
Upgrading several app servers to new version of HHVM, expect transient 'DPKG CRITICAL' alerts |
[production] |
| 18:19 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Group1 wikis to 1.26wmf5 |
[production] |
| 17:38 |
<ori> |
Synchronized wmf-config/CommonSettings.php: Ie4641b6e4: Set $wgWMEStatsdBaseUri to host-relative beacon/ path (duration: 00m 12s) |
[production] |
| 16:24 |
<yurik> |
graphoid service synced, now supports Cache Control headers |
[production] |
| 16:19 |
<ori> |
restarted HHVM on mw1061; T89912 |
[production] |
| 15:20 |
<thcipriani> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Add *.sl.nsw.gov.au to wgCopyUploadsDomains [[gerrit:210356]] (duration: 00m 11s) |
[production] |