|
2019-07-24
§
|
| 15:11 |
<herron> |
resume ingesting [message] =~ /^SlowTimer/ logs on logstash1007 (as a canary) |
[production] |
| 15:02 |
<XioNoX> |
re-enable vc link between asw2-a6 and asw2-a7 - T228823 |
[production] |
| 14:58 |
<jeh> |
unmounting dumps NFS clients from labstore1007.wikimedia.org T224228 |
[production] |
| 14:54 |
<XioNoX> |
cleared vc ports stats on asw2-a-eqiad - T228823 |
[production] |
| 14:43 |
<marostegui> |
Drop abuse_filter_log.afl_log_id in s5 eqiad - T226851 |
[production] |
| 14:40 |
<marostegui> |
Drop abuse_filter_log.afl_log_id in s5 codfw (lag will appear on codfw) - T226851 |
[production] |
| 14:31 |
<tarrow@> |
helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . |
[production] |
| 13:49 |
<robh> |
rebooting cloudvirt1015 into OS, memory error confirmed. new memory replacement dispatch entered via T220853 |
[production] |
| 13:31 |
<marostegui> |
Drop abuse_filter_log.afl_log_id in s2 eqiad - T226851 |
[production] |
| 13:25 |
<robh> |
rebooting cloudvirt1015 into memtest for dell support repair via T220853 |
[production] |
| 13:06 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.15 (duration: 00m 54s) |
[production] |
| 13:05 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.15 |
[production] |
| 12:19 |
<marostegui> |
Stop haproxy on dbproxy1004 and dbproxy1009 (m4 - eventlogging) - T228768 |
[production] |
| 11:23 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:525254|Disable FileImporter source wiki edits (T228851)]] (duration: 00m 54s) |
[production] |
| 11:12 |
<awight@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:514672|Remove Content Translation event logging config]] (part 2/2) (duration: 00m 54s) |
[production] |
| 11:10 |
<awight@deploy1001> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:514672|Remove Content Translation event logging config]] (part 1/2) (duration: 00m 59s) |
[production] |
| 10:04 |
<marostegui> |
Drop abuse_filter_log.afl_log_id from labswiki (wikitech) and labtestwiki - T226851 |
[production] |
| 09:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1082 (duration: 00m 55s) |
[production] |
| 08:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1082 into API after upgrade (duration: 00m 55s) |
[production] |
| 08:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1082 after upgrade (duration: 00m 54s) |
[production] |
| 08:40 |
<marostegui> |
Stop MySQL on db1082 for upgrade |
[production] |
| 08:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1082 for upgrade (duration: 00m 57s) |
[production] |
| 08:34 |
<marostegui> |
Drop abuse_filter_log.afl_log_id in s2 codfw (lag will appear on codfw) - T226851 |
[production] |
| 07:57 |
<marostegui> |
Drop abuse_filter_log.afl_log_id from wikidata in eqiad - T226851 |
[production] |
| 07:21 |
<marostegui> |
Stop MySQL on db1117:3322 to check dbproxy1013 notifications - T202367 |
[production] |
| 07:10 |
<marostegui> |
Deploy grants for dbproxy1013 in m2 - T202367 |
[production] |
| 05:00 |
<marostegui> |
Stop puppet on dbprov2001 to generate s5 mysqldump manually |
[production] |
| 04:52 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/includes/MediaWiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:51 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:50 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/includes/MediaWiki.php: T227700 (duration: 00m 53s) |
[production] |
| 04:49 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:46 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.15/includes/MediaWiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:45 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.15/includes/specials/SpecialGoToInterwiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:42 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.15/includes/MediaWiki.php: T227700 (duration: 00m 54s) |
[production] |
| 04:40 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.15/includes/specials/SpecialGoToInterwiki.php: (no justification provided) (duration: 00m 56s) |
[production] |
| 03:41 |
<tstarling@deploy1001> |
Synchronized w/fatal-error.php: Adding post-send exception test for T228462 (duration: 00m 54s) |
[production] |
| 03:39 |
<tstarling@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Adding DeferredUpdates log channel (T228462) (duration: 00m 56s) |
[production] |
| 02:01 |
<eileen> |
payments-wiki revision changed from 224c6b2d7b to a28ad541ed, config revision is 8dcb77cf22 |
[production] |
|
2019-07-23
§
|
| 23:44 |
<eileen> |
civicrm revision changed from 88e9f24893 to 875ab97742, config revision is 4006d3bdc5 |
[production] |
| 23:43 |
<shdubsh> |
reverting logstash mitigations and re-enable puppet |
[production] |
| 23:42 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.15/includes/diff/DifferenceEngine.php: T228766 Don't double wrap rollback links (duration: 00m 56s) |
[production] |
| 23:31 |
<mutante> |
mw1267 - rm -rf /srv/mediawiki/php-1.33.0-wmf.23 ; rm -rf /srv/mediawiki/php-1.32.0-wmf.3 ; scap pull |
[production] |
| 23:25 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet |
[production] |
| 22:36 |
<mutante> |
rolling out scap 3.11.1-1 on mw-eqiad servers |
[production] |
| 22:14 |
<mutante> |
continuing rollout of new scap version 3.11.1-1, starting with kafka-all followed by other cumin-alias groups (T228328) |
[production] |
| 22:06 |
<herron> |
puppet temporarily disabled on eqiad/codfw logstash collectors while catching up with backlog. see /etc/logstash/conf.d/01-filter_temp_drops.conf |
[production] |
| 21:52 |
<herron> |
logstash - temporarily dropping logs matching [message] =~ /^SlowTimer/ due to UTF-8 parsing errors that are stopping the logstash processing pipeline. will re-enable after logstash has caught up with the backlog |
[production] |
| 20:59 |
<shdubsh> |
temporarily disable input-kafka-rsyslog-shipper and drop memcached logs on logstash nodes |
[production] |
| 20:08 |
<paravoid> |
asw2-a-eqiad: request virtual-chassis vc-port set interface member 6 vcp-255/1/0 disable |
[production] |
| 19:58 |
<eileen> |
process-control config revision is 4006d3bdc5 - disabled drush fill donor totals job |
[production] |