|
2016-04-22
§
|
| 13:57 |
<mobrovac> |
restbase bringing the service back up on restbase1007 |
[production] |
| 13:43 |
<godog> |
bootstrap restbase1015-a T128107 |
[production] |
| 13:34 |
<mobrovac> |
restbase stopping restbase on rb1007 to manually inspect why is it flapping |
[production] |
| 13:27 |
<akosiaris> |
upload 5.3.10-1ubuntu3.22+wmf1 on apt.wikimedia.org |
[production] |
| 13:03 |
<godog> |
depool restbase1007, 400s from restbase self check |
[production] |
| 09:46 |
<moritzm> |
installing PHP security updates |
[production] |
| 09:25 |
<moritzm> |
installing ircbalance bugfix updates (preventing massive logspam on some systems) |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Apr 22 02:31:45 UTC 2016 (duration 9m 15s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.21) (duration: 10m 12s) |
[production] |
|
2016-04-21
§
|
| 20:34 |
<gehel> |
lowering elasticsearch high watermark to rebalance disk space across cluster |
[production] |
| 20:07 |
<gehel> |
reindex elasticsearch updates for the duration of the switch back from codfw, just in case |
[production] |
| 18:57 |
<mutante> |
rutherfordium - on 4.4.0-1-amd64 now (T131928) |
[production] |
| 18:56 |
<mutante> |
people.wm.org / rutherfordium , very short downtime, reboot |
[production] |
| 18:43 |
<mutante> |
rutherfordium (people.wm.org) - installing package upgrades |
[production] |
| 18:29 |
<mutante> |
bromine - on 4.4.0-1-amd64 now (T131928) |
[production] |
| 18:25 |
<mutante> |
bromine (annualreport,bz-static,transparency,releases) reboot for kernel upgrade |
[production] |
| 18:15 |
<mutante> |
bromine - apt-get dist-upgrade |
[production] |
| 18:12 |
<mutante> |
planet1001 - on 4.4.0-1-amd64 now (T131928) |
[production] |
| 18:10 |
<mutante> |
planet1001 - reboot for upgrade |
[production] |
| 17:44 |
<mutante> |
planet1001 - apt-get dist-upgrade (libc6, apache, ..) |
[production] |
| 16:10 |
<marktraceur@tin> |
Synchronized php-1.27.0-wmf.21/resources/src/mediawiki/api/upload.js: Unbreak finishing stash uploads in upload dialog (duration: 00m 27s) |
[production] |
| 16:10 |
<jynus> |
disabling event scheduler on db1041 |
[production] |
| 15:34 |
<_joe_> |
services switchover done |
[production] |
| 15:30 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Adust weights for shard s7 - T133205 (duration: 00m 32s) |
[production] |
| 15:28 |
<_joe_> |
[switchback services] moving traffic for restbase/citoid/cxserver back to eqiad |
[production] |
| 15:23 |
<ostriches> |
ytterbium: puppet re-enabled for gerrit host |
[production] |
| 15:23 |
<filippo@tin> |
Synchronized wmf-config/filebackend-production.php: [switchover swift #1] async writes to codfw (duration: 00m 28s) |
[production] |
| 15:15 |
<godog> |
[switchover swift #2] upload backends from codfw to eqiad |
[production] |
| 15:09 |
<godog> |
[switchover swift #3] upload codfw to eqiad |
[production] |
| 15:05 |
<godog> |
[switchover swift #4] repool upload eqiad in dns |
[production] |
| 15:04 |
<gehel> |
starting reindex of lost elasticsearch updates during activation of SSL (T132762) |
[production] |
| 15:01 |
<godog> |
[switchover swift #5] upload esams to eqiad |
[production] |
| 14:57 |
<godog> |
[switchover swift #6] upload eqiad to 'direct' |
[production] |
| 14:48 |
<gehel@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s) |
[production] |
| 14:45 |
<gehel> |
reverting elasticsearch traffic back to eqiad |
[production] |
| 14:39 |
<jynus> |
enabling replication lag alerts for all dbs |
[production] |
| 14:28 |
<paravoid> |
traffic/mediawiki codfw->eqiad switchover is done |
[production] |
| 14:26 |
<jynus> |
deployed dns updated records for new eqiad masters |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#3] Running rebuildEntityPerPage.php |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#2] Enabling crons in eqiad |
[production] |
| 14:24 |
<ori> |
[switchover #8/#1] Starting the jobqueue in eqiad |
[production] |
| 14:23 |
<volans> |
[switchover #8/#3] Re-enable puppet on all eqiad and codfw databases masters |
[production] |
| 14:21 |
<paravoid> |
wikis are read-write again |
[production] |
| 14:21 |
<ori@tin> |
Synchronized wmf-config/db-eqiad.php: [switchover #7/#1] Iac92c8bc6b: Put eqiad in read-write mode for datacenter switchover to eqiad (duration: 00m 39s) |
[production] |
| 14:20 |
<volans> |
[switchover #6/#1] Set database masters RW in eqiad for s1-7, es2-3 and x1 |
[production] |
| 14:19 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - DONE (confirmed) |
[production] |
| 14:18 |
<Krinkle> |
apache-fast-test warmup finished |
[production] |
| 14:18 |
<godog> |
[switchover #5/#7] finished swift-proxy roll restart |
[production] |
| 14:15 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - starting |
[production] |
| 14:14 |
<mobrovac> |
[switchover #5/2] restbase {{done}} |
[production] |