951-1000 of 10000 results (43ms)
2018-10-15 §
12:45 <banyek> rebooting db2096 [production]
12:44 <gehel> reseting kafka offsets on wdqs public cluster [production]
12:44 <elukey> complete rolling restart of eventbus on kafka[12]00[1-3] for python security upgrades (only codfw was done) [production]
12:41 <elukey> upgrade prometheus-memcached-exporter on swift and thumbor [production]
11:57 <Amir1> start of mwscript deleteLocalPasswords.php --delete --batch-size 200 on all wikis [production]
11:38 <zeljkof> EU SWAT finished [production]
11:29 <hoo> Started rebuildItemsPerSite on mwmaint1002 (T44325). Can be killed at any time, if necessary. [production]
11:26 <zfilipin@deploy1001> scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
11:09 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:467307|Enable reading from ct_tag_id in s7 (T194164)]] (duration: 00m 49s) [production]
10:57 <moritzm> installing ghostscript security updates for jessie [production]
10:47 <moritzm> installing tomcat7 security updates [production]
10:43 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:467300|Bumping portals to master (T128546)]] (duration: 00m 49s) [production]
10:42 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:467300|Bumping portals to master (T128546)]] (duration: 00m 49s) [production]
09:46 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1092 for recloning - T206743 (duration: 00m 49s) [production]
09:45 <marostegui> Stop MySQL on db1116:3318 to reclone db1092 [production]
09:41 <banyek> max_binlog_size is set back to 1048576000 on ParseCache hosts (T206740) [production]
09:23 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Restore original weight for db1104 (duration: 00m 49s) [production]
09:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 (duration: 00m 48s) [production]
08:58 <banyek@deploy1001> Synchronized wmf-config/db-codfw.php: T206593: depooling db2069 (duration: 00m 48s) [production]
08:50 <elukey> restart hadoop yarn resource managers on an-master* to pick up new jvm settings [production]
08:49 <XioNoX> repool eqsin - T206861 [production]
08:48 <banyek> depooling db2033 (T206593) [production]
08:46 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T206743 (duration: 00m 49s) [production]
08:17 <moritzm> installing imagemagick security update [production]
07:57 <godog> reformat ms-be2040 with crc=1 finobt=0 - T199198 [production]
07:32 <banyek> reimaging db2096(T206593) [production]
07:31 <banyek> reimaging db2096 [production]
07:24 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 48s) [production]
07:15 <marostegui> Stop MySQL at db1116:3318 to clone db1104 [production]
07:12 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 49s) [production]
07:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1109 (duration: 00m 49s) [production]
06:55 <XioNoX> add v6 monitoring for mr1-ulsfo OOB - T206778 [production]
06:49 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 (duration: 00m 49s) [production]
06:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1109 (duration: 00m 50s) [production]
05:20 <kartik@deploy1001> Finished deploy [cxserver/deploy@fd74c3b]: Update cxserver to b51f363 (T203077, T99934, T203550) (duration: 04m 25s) [production]
05:16 <kartik@deploy1001> Started deploy [cxserver/deploy@fd74c3b]: Update cxserver to b51f363 (T203077, T99934, T203550) [production]
05:16 <marostegui> Stop MySQL on db1109 for recloning - T206743 [production]
05:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 50s) [production]
05:11 <marostegui> Stop MySQL on db1116:3318 to use it to clone db1109 [production]
03:18 <kartik@deploy1001> Finished deploy [cxserver/deploy@5a70ef1]: Update cxserver to 47a864b (T205420, T203077, T205700, T205616) (duration: 04m 44s) [production]
03:14 <kartik@deploy1001> Started deploy [cxserver/deploy@5a70ef1]: Update cxserver to 47a864b (T205420, T203077, T205700, T205616) [production]
00:45 <krinkle@deploy1001> Synchronized multiversion/MWRealm.php: I79fb3d194a58: use env.php (duration: 00m 49s) [production]
00:08 <krinkle@deploy1001> Synchronized wmf-config/: I79fb3d194a: add env.php file (not yet used) (duration: 00m 50s) [production]
2018-10-14 §
23:42 <krinkle@deploy1001> Synchronized multiversion/getMWVersion: Ice9a74e73481 no-op (duration: 00m 49s) [production]
23:21 <krinkle@deploy1001> Synchronized wmf-config/ProductionServices.php: If4d8faa4 (duration: 00m 48s) [production]
21:48 <krinkle@deploy1001> Synchronized multiversion/MWMultiVersion.php: I83b2bdd53c13e (duration: 00m 50s) [production]
20:47 <krinkle@deploy1001> Synchronized wmf-config/import.php: beta-only (duration: 00m 54s) [production]
16:34 <volans> forcing a puppet run on all eqsin hosts with batch 1 to clear most of the alarms - T206861 [production]
08:54 <elukey> restart Yarn resource manager on an-master1002 to force an-master1001 to take the leadership back - T206943 [production]
08:34 <elukey> powercycle restbase1015 (frozen, no ssh, no metrics, no root console via serial available) [production]