|
2015-07-15
§
|
| 16:45 |
<andrewbogott> |
rebooting labvirt1005 |
[production] |
| 16:43 |
<mutante> |
accepting unaccepted salt keys for ganeti VMs ,planet, bromine, krypton |
[production] |
| 16:39 |
<mutante> |
krypton - signing puppet cert, initial run |
[production] |
| 16:26 |
<andrewbogott> |
woo, first try! |
[production] |
| 16:23 |
<andrewbogott> |
trying to kill labvirt1005 via repeated instance suspend/resume |
[production] |
| 16:04 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/224420/ (duration: 00m 12s) |
[production] |
| 16:03 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224420/ (duration: 00m 12s) |
[production] |
| 16:01 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/224808/ (duration: 00m 12s) |
[production] |
| 15:58 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/222581/ (duration: 00m 11s) |
[production] |
| 15:35 |
<krenair> |
Synchronized database lists: (no message) (duration: 00m 11s) |
[production] |
| 15:29 |
<krenair> |
Synchronized docroot/noc/createTxtFileSymlinks.sh: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 12s) |
[production] |
| 15:27 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 12s) |
[production] |
| 15:20 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/139326/ (duration: 00m 11s) |
[production] |
| 14:33 |
<legoktm> |
Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthStrict = true; (duration: 00m 12s) |
[production] |
| 14:22 |
<legoktm> |
sync failed on mw1090.eqiad.wmnet, read only filesystem |
[production] |
| 14:20 |
<legoktm> |
Synchronized php-1.26wmf13/extensions/CentralAuth/includes/CentralAuthPlugin.php: Add log entry for $wgCentralAuthStrict failures if SULMigration is enabled (duration: 00m 13s) |
[production] |
| 13:55 |
<dcausse> |
es1.6 upgrade: upgrade elastic1018 |
[production] |
| 13:24 |
<springle> |
entry below not mw1216 fault, but r/o filesystem error on mw1090 |
[production] |
| 13:15 |
<springle> |
sync-common on mw1216 after sync-file from tin failed non-zero exit status 12 |
[production] |
| 13:12 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1022 T105879 (duration: 00m 12s) |
[production] |
| 11:43 |
<dcausse> |
es1.6 upgrade: upgrade elastic1017 |
[production] |
| 08:27 |
<dcausse> |
es1.6 upgrade: upgrade elastic1016 |
[production] |
| 06:31 |
<dcausse> |
es1.6 upgrade: upgrade elastic1015 |
[production] |
| 05:40 |
<dcausse> |
es1.6 upgrade: upgrade elastic1014 |
[production] |
| 05:10 |
<springle> |
db1030 busy removing table partitioning |
[production] |
| 04:28 |
<manybubbles> |
es1.6 upgrade: lowered the shard transfer settings back to our normal rate. going to bed. |
[production] |
| 04:12 |
<manybubbles> |
es1.6 upgrade: upgrade elastic1013 |
[production] |
| 03:49 |
<springle> |
upgrade db1030 trusty |
[production] |
| 03:29 |
<manybubbles> |
es1.6 upgrade: upgrade elastic1012 |
[production] |
| 03:14 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-15 03:14:21+00:00 |
[production] |
| 03:10 |
<reedy> |
Synchronized php-1.26wmf13/cache/l10n: (no message) (duration: 13m 32s) |
[production] |
| 03:03 |
<manybubbles> |
es1.6 upgrade: raised limits on shard migration rate - should speed up the restart. we should lower it before we do restarts during europe's morning |
[production] |
| 02:10 |
<Reedy> |
Running LU manually to see what's wrong with it |
[production] |
| 02:07 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 15 02:07:48 UTC 2015 (duration 7m 47s) |
[production] |
| 02:02 |
<LocalisationUpdate> |
failed (1.26wmf13) at 2015-07-15 02:02:55+00:00 |
[production] |
|
2015-07-14
§
|
| 23:46 |
<manybubbles> |
es1.6 upgrade: upgraded elastic1011 |
[production] |
| 23:22 |
<bblack> |
updating nginx to 1.9.3-1+wmf1 on cp* |
[production] |
| 23:17 |
<bblack> |
reprepro: nginx for jessie-wikimedia/main bumped to 1.9.3-1+wmf1 |
[production] |
| 22:22 |
<ejegg> |
updated civicrm from 04efc7d5c7bbb068f907125f2184692aee676123 to 6560cefa8d7e68e35e30b310d6691ab57798a4c9 |
[production] |
| 21:30 |
<Reedy> |
mw1090 fs is ro |
[production] |
| 21:28 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Fix testwiki |
[production] |
| 21:05 |
<_joe|AFK> |
depooling mw1090, ext4 errors in syslog, filesystem mounted read-only |
[production] |
| 21:01 |
<twentyafterfour> |
Synchronized wmf-config/CommonSettings.php: revert LCStoreStaticArray (duration: 00m 12s) |
[production] |
| 20:59 |
<twentyafterfour> |
Finished scap: testwiki to 1.26wmf14 and rebuild localization cache (duration: 72m 45s) |
[production] |
| 20:42 |
<bblack> |
undoing LCStoreStaticArray because appservers look unhealthy, using ori's command: 'salt -G deployment_target:scap/scap cmd.run "rm /etc/lcstore"' |
[production] |
| 19:46 |
<twentyafterfour> |
Started scap: testwiki to 1.26wmf14 and rebuild localization cache |
[production] |
| 19:23 |
<manybubbles> |
es1.6 step iforget: upgrade elasticsearch on elastic1010 |
[production] |
| 17:41 |
<mutante> |
terbium: /usr/local/bin/foreachwiki extensions/Echo/maintenance/processEchoEmailBatch.php |
[production] |
| 17:10 |
<dcausse> |
es1.6 step 10: upgrade elastic1009 |
[production] |
| 16:23 |
<mutante> |
bromine - apt-get upgrade |
[production] |