651-700 of 10000 results (57ms)
2019-07-24 §
08:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1082 after upgrade (duration: 00m 54s) [production]
08:40 <marostegui> Stop MySQL on db1082 for upgrade [production]
08:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1082 for upgrade (duration: 00m 57s) [production]
08:34 <marostegui> Drop abuse_filter_log.afl_log_id in s2 codfw (lag will appear on codfw) - T226851 [production]
07:57 <marostegui> Drop abuse_filter_log.afl_log_id from wikidata in eqiad - T226851 [production]
07:21 <marostegui> Stop MySQL on db1117:3322 to check dbproxy1013 notifications - T202367 [production]
07:10 <marostegui> Deploy grants for dbproxy1013 in m2 - T202367 [production]
05:00 <marostegui> Stop puppet on dbprov2001 to generate s5 mysqldump manually [production]
04:52 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.14/includes/MediaWiki.php: T227700 (duration: 00m 54s) [production]
04:51 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.14/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) [production]
04:50 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.14/includes/MediaWiki.php: T227700 (duration: 00m 53s) [production]
04:49 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.14/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) [production]
04:46 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/MediaWiki.php: T227700 (duration: 00m 54s) [production]
04:45 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) [production]
04:42 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/MediaWiki.php: T227700 (duration: 00m 54s) [production]
04:40 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/specials/SpecialGoToInterwiki.php: (no justification provided) (duration: 00m 56s) [production]
03:41 <tstarling@deploy1001> Synchronized w/fatal-error.php: Adding post-send exception test for T228462 (duration: 00m 54s) [production]
03:39 <tstarling@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Adding DeferredUpdates log channel (T228462) (duration: 00m 56s) [production]
02:01 <eileen> payments-wiki revision changed from 224c6b2d7b to a28ad541ed, config revision is 8dcb77cf22 [production]
2019-07-23 §
23:44 <eileen> civicrm revision changed from 88e9f24893 to 875ab97742, config revision is 4006d3bdc5 [production]
23:43 <shdubsh> reverting logstash mitigations and re-enable puppet [production]
23:42 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.15/includes/diff/DifferenceEngine.php: T228766 Don't double wrap rollback links (duration: 00m 56s) [production]
23:31 <mutante> mw1267 - rm -rf /srv/mediawiki/php-1.33.0-wmf.23 ; rm -rf /srv/mediawiki/php-1.32.0-wmf.3 ; scap pull [production]
23:25 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet [production]
22:36 <mutante> rolling out scap 3.11.1-1 on mw-eqiad servers [production]
22:14 <mutante> continuing rollout of new scap version 3.11.1-1, starting with kafka-all followed by other cumin-alias groups (T228328) [production]
22:06 <herron> puppet temporarily disabled on eqiad/codfw logstash collectors while catching up with backlog. see /etc/logstash/conf.d/01-filter_temp_drops.conf [production]
21:52 <herron> logstash - temporarily dropping logs matching [message] =~ /^SlowTimer/ due to UTF-8 parsing errors that are stopping the logstash processing pipeline. will re-enable after logstash has caught up with the backlog [production]
20:59 <shdubsh> temporarily disable input-kafka-rsyslog-shipper and drop memcached logs on logstash nodes [production]
20:08 <paravoid> asw2-a-eqiad: request virtual-chassis vc-port set interface member 6 vcp-255/1/0 disable [production]
19:58 <eileen> process-control config revision is 4006d3bdc5 - disabled drush fill donor totals job [production]
19:49 <mutante> mwdebug1002 - restarting hhvm - mw1312 - restarted apache [production]
19:44 <andrewbogott> restarting rabbitmq-server on cloudcontrol1003 and 1004 [production]
19:40 <mutante> restarting hhvm on mw1312 [production]
19:28 <cdanis> depool all appservers in eqiad A7 cdanis@cumin1001.eqiad.wmnet ~ 🍵 sudo cumin 'mw12[67-83]*' 'depool' [production]
19:11 <bblack> repool lvs1013 - T227143 [production]
19:10 <bblack> repool cp1077 + cp1078 - T227143 [production]
19:09 <elukey> depool mw1261 for investigation [production]
19:06 <herron> restarting logstash on logstash100[789] [production]
18:53 <robh> mw1271 had power loss event due to pdu swap via T227143 [production]
18:45 <mutante> rolling out scap 3.11.1-1 on all mw codfw servers (T228328) [production]
18:43 <mutante> rolling out scap 3.11.1-1 on mw canary servers (T228328) [production]
18:13 <robh> started depooling servers in a7-eqiad for pdu work via T227143 [production]
18:11 <cdanis> depool mw1267 [production]
18:10 <cdanis> cdanis@mw1267.eqiad.wmnet /srv/mediawiki ☕ scap pull [production]
18:09 <cdanis> cdanis@mw1267.eqiad.wmnet ~ ☕ sudo apt install python-concurrent.futures [production]
18:08 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.15/includes/export/XmlDumpWriter.php: T228720 Make XmlDumpwriter resilient to blob store corruption (duration: 00m 54s) [production]
18:07 <James_F> Belay that, error on mw1267. [production]
18:06 <James_F> Sync error on mw1314.eqiad.wmnet, No module named concurrent.futures [production]
18:05 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.14/includes/export/XmlDumpWriter.php: T228720 Make XmlDumpwriter resilient to blob store corruption (duration: 00m 57s) [production]