451-500 of 10000 results (52ms)
2018-10-04 §
06:57 <jynus> starting multisource replication of s3 from s5 at eqiad master [production]
06:51 <jynus> reenabling consistency configuration on s5 replica databases [production]
06:24 <jynus> create manual backup of databases on eqiad s6, s7, s8, x1 [production]
05:36 <marostegui> Deploy schema change on db2048 (s1 master) - T205913 [production]
05:35 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 56s) [production]
05:30 <marostegui> Deploy schema change on db2062 - T205913 [production]
05:30 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 57s) [production]
04:04 <SMalyshev> repooled wdqs2003 [production]
03:22 <SMalyshev> depool wdqs2003 to let it catch up [production]
03:21 <SMalyshev> repooled wdqs2001 [production]
03:16 <ejegg> re-enabled PayPal EC orphan rectifier [production]
03:06 <ejegg> updated CiviCRM from 80cb98e33e to ddf4865650 [production]
02:43 <SMalyshev> depooled wdqs2001 to see if it catches up faster [production]
01:54 <ejegg> updated payments-wiki from 8b673cfb4f to d623de9494 [production]
2018-10-03 §
23:54 <mutante> scheduled downtime for wdqs as it's flapping and already known [production]
23:44 <catrope@deploy1001> Synchronized php-1.32.0-wmf.23/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081); ext.visualEditor.mwlanguage: Actually load all of the code (T205834) (duration: 00m 57s) [production]
23:41 <catrope@deploy1001> Synchronized php-1.32.0-wmf.24/extensions/VisualEditor/: Require Parsoid HTML 2.0.0, and handle its <audio> tags (T201081) (duration: 00m 59s) [production]
23:29 <catrope@deploy1001> Synchronized php-1.32.0-wmf.23/extensions/PageTriage/: Hide copyvio AFC filter option behind flag (T205918) (duration: 00m 57s) [production]
23:23 <catrope@deploy1001> Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: Make UID clock drift error have more details (T94522) (duration: 00m 58s) [production]
23:20 <XenoRyet> shut off Paypal orphan rectifier [production]
23:12 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Bump Minerva A/B test rates to 100% on jawiki, ruwiki, fawiki (T200792) (duration: 00m 56s) [production]
22:49 <shdubsh> re-enable puppet on einsteinium [production]
22:45 <shdubsh> einsteinium: setting enable_notifications=1 and reloading icinga [production]
22:36 <herron> herron@neodymium:~$ sudo cumin -b 15 -p 95 '*' 'run-puppet-agent -q --failed-only' [production]
22:20 <shdubsh> einsteinium: setting enable_notifications=0 and starting icinga [production]
22:06 <herron> herron@neodymium:~$ sudo cumin -b 40 -p 95 'R:file = /etc/nagios/nrpe_local.cfg' run-puppet-agent [production]
22:02 <mutante> mw2242 - started nagios-nrpe-server [production]
22:01 <shdubsh> icinga stopped manually [production]
21:57 <mutante> einstienium - disabling puppet [production]
21:25 <bblack> upgraded gdnsd to 2.99.9161 on authdns1001 [production]
21:17 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.32.0-wmf.24 (duration: 00m 55s) [production]
21:16 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.24 [production]
21:12 <dduvall@deploy1001> Synchronized php-1.32.0-wmf.24/extensions/WikibaseQualityConstraints/src/ServiceWiring.php: deploying fix to 1.32.0-wmf.24 for T206161 (duration: 00m 57s) [production]
20:28 <marxarelli> deployed proposed WikibaseQualityConstraints fix and wikiversions bump for wikidatawiki to mwdebug1001 and mwdebug1002 for verification (T206161) [production]
20:18 <robh> optic swap on cr4-ulsfo:et-0/0/1 [production]
20:03 <bblack> upgraded gdnsd to 2.99.9161 on multatuli [production]
19:40 <bblack> upgraded gdnsd to 2.99.9161 on authdns2001 [production]
19:35 <bblack> uploaded 2.99.9161-beta-1+wmf1 to stretch-wikimedia [production]
19:33 <mateusbs17> running initial osm import in maps1004 [production]
19:23 <dduvall@deploy1001> Synchronized php: rollback group1 to 1.32.0-wmf.23 (duration: 00m 54s) [production]
19:18 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: rollback group1 to 1.32.0-wmf.23 [production]
19:15 <marxarelli> rolling back group1 after rapid rise in fatals [production]
19:14 <dduvall@deploy1001> scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
18:49 <RoanKattouw> Deployed patches for T206130 [production]
18:36 <papaul> reinstalling OS on lvs2010 [production]
18:16 <mutante> lvs2010 - schduled downtime for host and services for 12 hours for reinstall [production]
18:09 <mutante> lvs2009 - schedule downtime in icinga for 4 hours, reinstall in progress [production]
18:08 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@d5bab41]: Bump cirrusSearchLinksUpdate concurrency to 20 (duration: 00m 57s) [production]
18:07 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@d5bab41]: Bump cirrusSearchLinksUpdate concurrency to 20 [production]
18:07 <XioNoX> disable ulsfo Zayo transit/transport links [production]