3001-3050 of 10000 results (65ms)
2018-06-26 §
12:15 <marostegui> Deploy schema change on db1105:3312 T191316 T192926 T89737 T195193 [production]
12:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 57s) [production]
12:07 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 57s) [production]
11:09 <godog> roll-upgrade thumbor in codfw/eqiad [production]
11:08 <godog> upload thumbor 2.0-2 to apt.wikimedia.org [production]
10:40 <godog> upgrade thumbor to 2.0-1 on thumbor1001 [production]
10:10 <godog> test upgrade thumbor 2.0 on thumbor2001 [production]
10:10 <akosiaris> proceed with apertium-apy upgrade on all scb hosts. T194342 [production]
10:01 <akosiaris> downgrade erroneously upgraded apertium-fra-cat on scb1001, scb1002 from 1.3.0~r84327-1+wmf1 to 1.2.0~r78602-1+wmf2 T194342 [production]
09:40 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw [production]
09:39 <akosiaris> re-enable icinga-wm that I 've previously silenced [production]
09:37 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad [production]
09:36 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=appserver [production]
09:33 <mobrovac@deploy1001> Finished scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 (duration: 05m 07s) [production]
09:31 <akosiaris> upgrade apertium-apy, apertium-fra-cat, python3-streamparser, enable puppet, run puppet on scb1002 as a last test before full cluster upgrade. T194342 [production]
09:31 <vgutierrez> updating librdkafka1 and restarting varnishkafka-webrequest on cache::misc nodes - T182993 [production]
09:29 <dcausse> elastic@eqiad: forcemerge (only_expunge_deletes=true) on commonswiki_file [production]
09:28 <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 [production]
09:25 <akosiaris> manually install python3-streamparser on scb1001 for testing. T194342 [production]
09:19 <mobrovac@deploy1001> Finished deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626 (duration: 01m 14s) [production]
09:18 <mobrovac@deploy1001> Started deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626 [production]
09:11 <mobrovac@deploy1001> Finished scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 (duration: 08m 34s) [production]
09:09 <vgutierrez> upload librdkafka_0.11.3-1~bpo8+1+wikimedia2 to apt.w.o jessie-wikimedia - T182993 [production]
09:08 <akosiaris> upgrade scb1001 apertium-apy, apertium-fra-cat, enable puppet and run puppet [production]
09:06 <akosiaris> T194342. Disable puppet on scb hosts for apertium-apy upgrade [production]
09:04 <dcausse> unbanning elastic1042 [production]
09:03 <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 [production]
08:52 <mobrovac@deploy1001> scap failed: RuntimeError scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 04m 27s) [production]
08:52 <mobrovac@deploy1001> scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) [production]
08:48 <mobrovac@deploy1001> Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327 [production]
08:47 <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s) [production]
08:47 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s) [production]
08:46 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 [production]
07:58 <vgutierrez> Replaced acamar and achernar with dns200[12] as main lvs name servers in codfw - T196493 [production]
07:13 <_joe_> restarting ircecho, was not reporting alerts [production]
07:05 <marostegui> Deploy schema change on db1103:3312 T191316 T192926 T89737 T195193 [production]
07:05 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 56s) [production]
06:42 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1122 after alter table (duration: 00m 57s) [production]
05:58 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 55s) [production]
05:57 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 57s) [production]
05:52 <marostegui> Stop MySQL on db1054 as it is going to be decommissioned - T197063 [production]
05:28 <SMalyshev> testing fix for T197447 on wdqs1009 [production]
05:15 <marostegui> Deploy schema change on db1122 T191316 T192926 T89737 T195193 [production]
05:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1122 for alter table (duration: 00m 59s) [production]
03:05 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 12m 59s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 26s) [production]
2018-06-25 §
23:40 <thcipriani> restarting jenkins for updates [production]
23:30 <twentyafterfour> After monitoring logstash, everything appears stable. Evening SWAT is complete. [production]
23:23 <twentyafterfour@deploy1001> Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440878/ for SWAT: no-op (duration: 00m 56s) [production]
23:10 <twentyafterfour@deploy1001> Synchronized wmf-config/Wikibase-production.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440420/ for SWAT (duration: 00m 57s) [production]