3351-3400 of 10000 results (69ms)
2018-11-15 ยง
13:50 <banyek> Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T85757 [production]
13:48 <moritzm> installing qemu security updates (which also backport support for SSBD passthrough) on ganeti clusters [production]
13:48 <moritzm> installing qemu security updates (which also backport support for SSBD passthrough) [production]
13:12 <mobrovac@deploy1001> Finished deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 (duration: 19m 56s) [production]
13:00 <Lucas_WMDE> EU SWAT finished [production]
12:59 <tarrow@deploy1001> Synchronized php-1.33.0-wmf.4/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: [[gerrit:473710]] Fix (accidentally?) reversed blue and yellow lines SWAT T208238 T162119 again (duration: 00m 55s) [production]
12:57 <_joe_> upping pm.maxworkers to 40 on mw1261 on php7.2-fpm, benchmarking T206341 [production]
12:56 <tarrow@deploy1001> scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
12:53 <moritzm> installing nginx security updates [production]
12:52 <mobrovac@deploy1001> Started deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 [production]
12:47 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:471738|Remove wgWBQualityConstraintsCacheCheckConstraintsResults (T207854)]] (duration: 00m 54s) [production]
12:42 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:473224|Prod: increase Schema.org page split test to 25% sampling]] T208755 (duration: 00m 53s) [production]
12:40 <arturo> T207377 downtime and reboot labmon1001 [production]
12:38 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: [[gerrit:473710|Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)]] (duration: 00m 54s) [production]
12:37 <lucaswerkmeister-wmde@deploy1001> sync aborted: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js [[gerrit:473710|Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)]] (duration: 00m 11s) [production]
12:37 <lucaswerkmeister-wmde@deploy1001> Started scap: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js [[gerrit:473710|Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)]] [production]
12:30 <tarrow@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:473716]] Read WikibaseStringLimit in Wikibase.php T154660 (duration: 00m 53s) [production]
12:30 <moritzm> draining ganeti1001 for reboot/kernel security update [production]
12:26 <tarrow@deploy1001> sync-file aborted: [[gerrit:473716]] Read WikibaseStringLimit in Wikibase.php (duration: 00m 01s) [production]
12:21 <tarrow@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:473694]] Set Wikibase string-limits for wikidata dblist T154660 (duration: 00m 54s) [production]
12:08 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:473440|Make AdvancedSearch the default on de-, fa-, ar-, and hu-wiki]] T207640 (duration: 00m 55s) [production]
11:34 <_joe_> depooling mw1261 for early benchmarks of php7.2-fpm [production]
11:04 <akosiaris> enable puppet across the fleet. puppetdb1001 reboot done, ganeti migration_downtime setting applied [production]
10:49 <akosiaris> disable puppet across the fleet for puppetdb1001 reboot [production]
10:49 <moritzm> fail over ganeti master in eqiad to ganeti1003 [production]
10:34 <akosiaris> set migration_downtime=2000 for puppetdb1001. Should help with migration stalls [production]
10:15 <banyek> sanitizing db1124 ( T205714 T207584 T205713 T206916 ) [production]
10:07 <banyek> sanitizing db2094 ( T205714 T207584 T205713 T206916 ) [production]
09:57 <moritzm> draining ganeti1002 for reboot/kernel security update [production]
09:40 <volans> restarting icinga on icinga1001 [production]
09:32 <vgutierrez> restarting icinga on icinga1001 [production]
09:31 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s) [production]
09:22 <moritzm> draining ganeti1003 for reboot/kernel security update [production]
09:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 53s) [production]
09:16 <marostegui> Stop MySQL on db1110 for upgrade [production]
09:08 <moritzm> reset failed debmonitor session in ms-be2038 [production]
09:08 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T85757: repool db1088 (duration: 00m 53s) [production]
09:03 <banyek> repooling db1088 (T85757) [production]
08:58 <ema> upload fifo-log-demux 0.1 to stretch-wikimedia T204225 [production]
08:56 <banyek> Deploy schema change on db1088 (T85757) [production]
08:49 <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T85757: depool db1088 (duration: 00m 53s) [production]
08:45 <banyek> depooling db1088 due a schema change (T85757) [production]
08:21 <moritzm> draining ganeti1004 for reboot/kernel security update [production]
07:42 <marostegui> Drop site_stats.ss_total_views from labswiki - T86339 [production]
07:08 <elukey> memcached on mc1019 restarted to apply -R 200 - T208844 [production]
06:57 <marostegui> Stop MySQL on db2071 to upgrade MySQL and kernel [production]
06:57 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 54s) [production]
06:06 <marostegui> Stop MySQL on pc2006 to clone pc2009 - T208383 [production]
06:06 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool pc2006 - T208383 (duration: 00m 53s) [production]
06:05 <marostegui@deploy1001> sync-file aborted: Dool pc2006 - T208383 (duration: 00m 00s) [production]