|
2018-06-26
§
|
| 08:52 |
<mobrovac@deploy1001> |
scap failed: RuntimeError scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 04m 27s) |
[production] |
| 08:52 |
<mobrovac@deploy1001> |
scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 08:48 |
<mobrovac@deploy1001> |
Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327 |
[production] |
| 08:47 |
<mobrovac@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s) |
[production] |
| 08:47 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s) |
[production] |
| 08:46 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 |
[production] |
| 07:58 |
<vgutierrez> |
Replaced acamar and achernar with dns200[12] as main lvs name servers in codfw - T196493 |
[production] |
| 07:13 |
<_joe_> |
restarting ircecho, was not reporting alerts |
[production] |
| 07:05 |
<marostegui> |
Deploy schema change on db1103:3312 T191316 T192926 T89737 T195193 |
[production] |
| 07:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 56s) |
[production] |
| 06:42 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1122 after alter table (duration: 00m 57s) |
[production] |
| 05:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 55s) |
[production] |
| 05:57 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 57s) |
[production] |
| 05:52 |
<marostegui> |
Stop MySQL on db1054 as it is going to be decommissioned - T197063 |
[production] |
| 05:28 |
<SMalyshev> |
testing fix for T197447 on wdqs1009 |
[production] |
| 05:15 |
<marostegui> |
Deploy schema change on db1122 T191316 T192926 T89737 T195193 |
[production] |
| 05:15 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1122 for alter table (duration: 00m 59s) |
[production] |
| 03:05 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.999) (duration: 12m 59s) |
[production] |
| 02:33 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 26s) |
[production] |
|
2018-06-25
§
|
| 23:40 |
<thcipriani> |
restarting jenkins for updates |
[production] |
| 23:30 |
<twentyafterfour> |
After monitoring logstash, everything appears stable. Evening SWAT is complete. |
[production] |
| 23:23 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440878/ for SWAT: no-op (duration: 00m 56s) |
[production] |
| 23:10 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/Wikibase-production.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440420/ for SWAT (duration: 00m 57s) |
[production] |
| 21:04 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52 (duration: 05m 41s) |
[production] |
| 20:58 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52 |
[production] |
| 20:09 |
<mdholloway> |
deploying mobileapps to canary (scb2001) failed, rolling back to investigate |
[production] |
| 20:07 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0 (duration: 01m 34s) |
[production] |
| 20:06 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0 |
[production] |
| 20:02 |
<cscott> |
Completed deploy of Parsoid to version b068bb51 (T197949, T197702, T196799) |
[production] |
| 19:53 |
<cscott@deploy1001> |
Finished deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51 (duration: 06m 57s) |
[production] |
| 19:46 |
<cscott@deploy1001> |
Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51 |
[production] |
| 19:22 |
<cscott@deploy1001> |
Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51 |
[production] |
| 18:33 |
<thcipriani@deploy1001> |
Synchronized portals: SWAT: [[gerrit:441913|Bumping portals to master]] T128546 (duration: 00m 56s) |
[production] |
| 18:32 |
<thcipriani@deploy1001> |
Synchronized portals/wikipedia.org/assets: SWAT: [[gerrit:441913|Bumping portals to master]] T128546 (duration: 00m 57s) |
[production] |
| 18:25 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:441069|Add import sources to ta.wiktionary]] T196445 (duration: 00m 58s) |
[production] |
| 18:13 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:441901|Enable logging for Schema:CitationUsage]] T191086 (noop beta-only sync) (duration: 00m 57s) |
[production] |
| 18:07 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (duration: 08m 19s) |
[production] |
| 17:58 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater |
[production] |
| 17:56 |
<gehel@deploy1001> |
Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 24s) |
[production] |
| 17:56 |
<gehel@deploy1001> |
Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only) |
[production] |
| 17:36 |
<ottomata> |
Re-enable job and change-prop kafka topic mirroring from main-codfw -> main-eqiad |
[production] |
| 16:45 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Beta only [[gerrit:441909]] (duration: 00m 57s) |
[production] |
| 16:19 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=achernar.wikimedia.org |
[production] |
| 16:13 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=acamar.wikimedia.org |
[production] |
| 16:04 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org |
[production] |
| 16:00 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org |
[production] |
| 15:20 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: REVERT: [[gerrit:438005|Load WikibaseLexeme on testwiki (again again)]] T197454 (duration: 00m 57s) |
[production] |
| 15:16 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:438005|Load WikibaseLexeme on testwiki (again again)]] T197454 (duration: 00m 55s) |
[production] |
| 15:05 |
<addshore> |
FileImporter slot done, moving onto Lexeme slot |
[production] |
| 15:03 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:441014|Enable FileExpoter on ar-, de- and fa-wiki]] T196969 T197066 (duration: 00m 57s) |
[production] |