2951-3000 of 10000 results (45ms)
2017-11-03 §
08:58 <mobrovac@tin> Started deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418 [production]
08:46 <ema> pool cp4021 [production]
08:16 <elukey> drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 from analytics dbs (db1046/db1047/db1108/dbstore1002) - data archived on HDFS - T166712 [production]
08:12 <moritzm> installing openjdk-8 security updates on stretch [production]
07:57 <marostegui> Deploy alter table on s4.db2037 - T174569 [production]
07:29 <elukey> depooled mw1191 and powercycle - host down with 'CPU 1 check errors' in racadm getsel [production]
06:39 <marostegui> Stop MySQL on db2089.s5 to copy it to db2085 - T178359 [production]
06:25 <marostegui> Stop MySQL on db1103 to copy it to dbstore1001 and reimage it - T178359 [production]
2017-11-02 §
23:36 <krinkle@tin> Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s) [production]
23:10 <herron> disabled puppet master debug logging on puppetmaster2001 via /usr/share/puppet/rack/puppet-master/config.ru [production]
22:48 <volans> cleaned daemon.log from puppet spam logs on puppetmaster2001 (15GB) - cc herron [production]
22:39 <halfak@tin> Started deploy [ores/deploy@82a13ae]: T179621 [production]
22:02 <Krinkle> Mass-resizing Graphite/Whisper files on graphite1001 and graphite2001 for T179622 (frontend.* namespace) [production]
21:53 <ayounsi@tin> Finished deploy [librenms/librenms@ad48b4d]: (no justification provided) (duration: 00m 05s) [production]
21:52 <ayounsi@tin> Started deploy [librenms/librenms@ad48b4d]: (no justification provided) [production]
21:50 <XioNoX> upgrading librenms [production]
21:38 <hoo> Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds [production]
20:56 <madhuvishy> Powercycling labvirt1015 [production]
20:18 <chasemp> simulate load for labvirt1015 'sudo cumin "name:labvirt1015stresstest*" "stress-ng --cpu 1 --io 2 --vm 1 --vm-bytes 1G &"' [production]
19:45 <mepps> updated payments-wiki from b88b805c3b223404578c3ef8bb675e710611aa2d to a539d275377b251762f4935c3014978253f39c08 [production]
19:28 <demon@tin> rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.6 [production]
18:20 <gehel> restarting blazegraph on wdqs1003 [production]
18:13 <gehel> restarting stuck updater on wdqs1003 [production]
17:55 <demon@tin> Synchronized php-1.31.0-wmf.6/includes/Message.php: logging fixes (duration: 00m 48s) [production]
17:53 <demon@tin> Synchronized php-1.31.0-wmf.6/includes/libs/rdbms: logging improvements (duration: 00m 54s) [production]
17:47 <moritzm> uploaded openssl 1.1.0g-1+wmf1 to jessie-wikimedia [production]
17:42 <awight@tin> Finished deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 (duration: 33m 50s) [production]
17:08 <awight@tin> Started deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 [production]
16:14 <ema> restart varnish-be on cp4026 due to mbox lag [production]
16:04 <mobrovac@tin> Synchronized wmf-config/jobqueue.php: Use only EventBus for processing updateBetaFeatureUserCount - T175210 (duration: 00m 51s) [production]
13:46 <chasemp> ugrade dnsmasq-base on labcontrol* [production]
13:36 <zeljkof> EU SWAT finished [production]
13:35 <zfilipin@tin> Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: [[gerrit:387996|CX1: Check for template adaptation failures before publishing (T154116)]] (duration: 00m 50s) [production]
13:11 <zfilipin@tin> Synchronized wmf-config/ProductionServices.php: SWAT: [[gerrit:383355|use the logstash LVS endpoint (T175242)]] (duration: 00m 51s) [production]
12:09 <moritzm> removed obsolete packages from jessie-wikimedia/experimental [production]
12:06 <mobrovac@tin> Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s) [production]
11:57 <mobrovac@tin> Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 [production]
11:43 <elukey> drop table log.PageContentSaveComplete_5588433 from db1046,db1047,db1108,dbstore1002, archived on hdfs - T177101 [production]
11:31 <moritzm> remove obsolete packages from stretch-wikimedia/experimental, now empty [production]
11:29 <mobrovac@tin> Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s) [production]
11:26 <elukey> drop log.MediaViewer_10867062_15423246 from db1047,db1108 since already archived in hdfs - T168303 [production]
11:21 <mobrovac@tin> Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 [production]
10:54 <gehel@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=logstash1003.eqiad.wmnet [production]
10:53 <gehel@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=logstash1002.eqiad.wmnet [production]
10:53 <gehel@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=logstash1001.eqiad.wmnet [production]
10:53 <gehel> depooling logstash100[123] in preparation for decommission - T175830 [production]
10:38 <gehel> rolling restart of wdqs nodes for GC tuning - T175919 [production]
10:13 <marostegui> Deploy alter table on db2019 - T174569 [production]
09:41 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T161088 (duration: 00m 50s) [production]
09:06 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s) [production]