|
2015-07-15
§
|
| 14:33 |
<legoktm> |
Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthStrict = true; (duration: 00m 12s) |
[production] |
| 14:22 |
<legoktm> |
sync failed on mw1090.eqiad.wmnet, read only filesystem |
[production] |
| 14:20 |
<legoktm> |
Synchronized php-1.26wmf13/extensions/CentralAuth/includes/CentralAuthPlugin.php: Add log entry for $wgCentralAuthStrict failures if SULMigration is enabled (duration: 00m 13s) |
[production] |
| 13:55 |
<dcausse> |
es1.6 upgrade: upgrade elastic1018 |
[production] |
| 13:24 |
<springle> |
entry below not mw1216 fault, but r/o filesystem error on mw1090 |
[production] |
| 13:15 |
<springle> |
sync-common on mw1216 after sync-file from tin failed non-zero exit status 12 |
[production] |
| 13:12 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1022 T105879 (duration: 00m 12s) |
[production] |
| 11:43 |
<dcausse> |
es1.6 upgrade: upgrade elastic1017 |
[production] |
| 08:27 |
<dcausse> |
es1.6 upgrade: upgrade elastic1016 |
[production] |
| 06:31 |
<dcausse> |
es1.6 upgrade: upgrade elastic1015 |
[production] |
| 05:40 |
<dcausse> |
es1.6 upgrade: upgrade elastic1014 |
[production] |
| 05:10 |
<springle> |
db1030 busy removing table partitioning |
[production] |
| 04:28 |
<manybubbles> |
es1.6 upgrade: lowered the shard transfer settings back to our normal rate. going to bed. |
[production] |
| 04:12 |
<manybubbles> |
es1.6 upgrade: upgrade elastic1013 |
[production] |
| 03:49 |
<springle> |
upgrade db1030 trusty |
[production] |
| 03:29 |
<manybubbles> |
es1.6 upgrade: upgrade elastic1012 |
[production] |
| 03:14 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-15 03:14:21+00:00 |
[production] |
| 03:10 |
<reedy> |
Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 13m 32s) |
[production] |
| 03:03 |
<manybubbles> |
es1.6 upgrade: raised limits on shard migration rate - should speed up the restart. we should lower it before we do restarts during europe's morning |
[production] |
| 02:10 |
<Reedy> |
Running LU manually to see what's wrong with it |
[production] |
| 02:07 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 15 02:07:48 UTC 2015 (duration 7m 47s) |
[production] |
| 02:02 |
<LocalisationUpdate> |
failed (1.26wmf13) at 2015-07-15 02:02:55+00:00 |
[production] |
|
2015-07-14
§
|
| 23:46 |
<manybubbles> |
es1.6 upgrade: upgraded elastic1011 |
[production] |
| 23:22 |
<bblack> |
updating nginx to 1.9.3-1+wmf1 on cp* |
[production] |
| 23:17 |
<bblack> |
reprepro: nginx for jessie-wikimedia/main bumped to 1.9.3-1+wmf1 |
[production] |
| 22:22 |
<ejegg> |
updated civicrm from 04efc7d5c7bbb068f907125f2184692aee676123 to 6560cefa8d7e68e35e30b310d6691ab57798a4c9 |
[production] |
| 21:30 |
<Reedy> |
mw1090 fs is ro |
[production] |
| 21:28 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Fix testwiki |
[production] |
| 21:05 |
<_joe|AFK> |
depooling mw1090, ext4 errors in syslog, filesystem mounted read-only |
[production] |
| 21:01 |
<twentyafterfour> |
Synchronized wmf-config/CommonSettings.php: revert LCStoreStaticArray (duration: 00m 12s) |
[production] |
| 20:59 |
<twentyafterfour> |
Finished scap: testwiki to 1.26wmf14 and rebuild localization cache (duration: 72m 45s) |
[production] |
| 20:42 |
<bblack> |
undoing LCStoreStaticArray because appservers look unhealthy, using ori's command: 'salt -G deployment_target:scap/scap cmd.run "rm /etc/lcstore"' |
[production] |
| 19:46 |
<twentyafterfour> |
Started scap: testwiki to 1.26wmf14 and rebuild localization cache |
[production] |
| 19:23 |
<manybubbles> |
es1.6 step iforget: upgrade elasticsearch on elastic1010 |
[production] |
| 17:41 |
<mutante> |
terbium: /usr/local/bin/foreachwiki extensions/Echo/maintenance/processEchoEmailBatch.php |
[production] |
| 17:10 |
<dcausse> |
es1.6 step 10: upgrade elastic1009 |
[production] |
| 16:23 |
<mutante> |
bromine - apt-get upgrade |
[production] |
| 15:08 |
<manybubbles> |
Synchronized php-1.26wmf13/extensions/UniversalLanguageSelector/: SWAT add some hooks to extension.json (duration: 00m 13s) |
[production] |
| 14:34 |
<gwicke> |
started RESTBase revision thin-out script for html and data-parsoid on wikimedia domains |
[production] |
| 14:01 |
<dcausse> |
es1.6 step 9: upgrade elastic1008 |
[production] |
| 12:48 |
<_joe_> |
reimaging mw1155 |
[production] |
| 12:17 |
<ori> |
Logging a message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log. |
[production] |
| 11:28 |
<dcausse> |
es1.6 step 8: upgrade elastic1007 |
[production] |
| 11:25 |
<_joe_> |
repooling mw1154 with HHVM |
[production] |
| 10:12 |
<_joe_> |
stopped poolcounter on mw1154 |
[production] |
| 10:06 |
<_joe_> |
reimaging mw1154 |
[production] |
| 07:49 |
<dcausse> |
es1.6 step 7: upgrade elastic1006 |
[production] |
| 07:09 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jul 14 07:09:10 UTC 2015 (duration 9m 9s) |
[production] |
| 06:48 |
<dcausse> |
es1.6 step 6: upgrade elastic1005 |
[production] |
| 06:41 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I9c9bf0f4: Use LCStoreStaticArray unconditionally (duration: 03m 02s) |
[production] |