|
2019-11-07
ยง
|
| 20:21 |
<herron> |
performing rolling reboots of kafka-main hosts for security updates |
[production] |
| 20:17 |
<onimisionipe> |
cluster restart for cloudelastic to pick JVM upgrade |
[production] |
| 20:08 |
<eileen> |
civicrm revision changed from f1ce5c86f7 to bfa53ee611, config revision is 72d2692743 |
[production] |
| 19:54 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision: Enqueue annotation job on upload complete (duration: 05m 19s) |
[production] |
| 18:31 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: MachineVision: Disable retrying annotation requests (duration: 05m 17s) |
[production] |
| 18:25 |
<ebernhardson> |
restart mjolnir-kafka-bulk-daemon and mjolnir-kafka-msearch-daemon across `cirrus` dsh group |
[production] |
| 18:20 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@afd41d7]: bulk_daemon: Adjust glent configuration (duration: 05m 49s) |
[production] |
| 18:14 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@afd41d7]: bulk_daemon: Adjust glent configuration |
[production] |
| 17:44 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 17:39 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) |
[production] |
| 17:38 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 17:37 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 17:30 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 17:25 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Drop currently unsupported external dependencies (T227349) (duration: 05m 19s) |
[production] |
| 17:10 |
<XioNoX> |
Homer push - forwarding-options - to all cr |
[production] |
| 17:09 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 17:08 |
<XioNoX> |
add sampling stanza (disabled) to cr2-esams |
[production] |
| 17:00 |
<mutante> |
wtp2020 - 2 hours downtime - shut down (T205712) - go ahead @papaul |
[production] |
| 17:00 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) |
[production] |
| 16:58 |
<mutante> |
wtp2020 - depooled for T205712 |
[production] |
| 16:58 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp2020.codfw.wmnet |
[production] |
| 16:42 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: some alphasorted config (duration: 01m 00s) |
[production] |
| 16:34 |
<XioNoX> |
Homer push on cr2-knams: Sampling (disabled), enhanced-hash-key, ospf interfaces re-ordering (noop), policy-statement BGP_from_LVS (unused), lo0 term allow_vmhost |
[production] |
| 16:32 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1080 at 100%', diff saved to https://phabricator.wikimedia.org/P9553 and previous config saved to /var/cache/conftool/dbconfig/20191107-163235-jynus.json |
[production] |
| 16:20 |
<XioNoX> |
add BGP sessions to AS64050 in eqiad |
[production] |
| 16:15 |
<XioNoX> |
add BGP sessions to AS57695 in esams and eqiad |
[production] |
| 16:12 |
<XioNoX> |
clear v4 BGP sessions to AS7713 in eqsin (hit max prefix limit) |
[production] |
| 16:02 |
<mutante> |
mw2225 restart cron (T236799) |
[production] |
| 15:58 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: beta logging (duration: 01m 00s) |
[production] |
| 15:41 |
<XioNoX> |
remove BGP to AS3491 on eqiad (left the IX) |
[production] |
| 15:40 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 14:53 |
<jbond42> |
rebuilding compiler1001 |
[production] |
| 13:50 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1080 at 50%', diff saved to https://phabricator.wikimedia.org/P9551 and previous config saved to /var/cache/conftool/dbconfig/20191107-135018-jynus.json |
[production] |
| 12:47 |
<Urbanecm> |
EU SWAT done |
[production] |
| 12:47 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 8e71601: a36ed85: GrowthExperiments: Configure testwiki for suggested edits testing + follow up patch (T237634) (duration: 00m 59s) |
[production] |
| 12:31 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 19034af: GrowthExperiments: Configure intro links for suggested edits (T235723) (duration: 01m 00s) |
[production] |
| 12:16 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 2be3f86: [cirrus] remove cross_cluster_single_shard_search quirk (duration: 01m 02s) |
[production] |
| 12:04 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 5253dec: Give commonswiki filemovers `suppressredirect` rights (T236348) (duration: 01m 03s) |
[production] |
| 11:57 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: repool es1016 fully (duration: 01m 01s) |
[production] |
| 11:54 |
<jbond42> |
update puppet_version used by CI 545289 |
[production] |
| 11:50 |
<jbond42> |
rebuilding compiler1002 |
[production] |
| 11:36 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool db1080 at 10%', diff saved to https://phabricator.wikimedia.org/P9550 and previous config saved to /var/cache/conftool/dbconfig/20191107-113611-jynus.json |
[production] |
| 11:16 |
<jynus> |
stop and upgrade db1080 |
[production] |
| 10:58 |
<moritzm> |
installing Java security updates on kafka-main/logstash |
[production] |
| 10:50 |
<moritzm> |
installing Java security updates on wdqs/maps |
[production] |
| 10:46 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Fully depool db1080', diff saved to https://phabricator.wikimedia.org/P9549 and previous config saved to /var/cache/conftool/dbconfig/20191107-104618-jynus.json |
[production] |
| 10:28 |
<moritzm> |
upgrading mw1277-1279 servers to PHP 7.2.24 T237239 |
[production] |
| 10:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Reduce db1080 weight', diff saved to https://phabricator.wikimedia.org/P9548 and previous config saved to /var/cache/conftool/dbconfig/20191107-102747-jynus.json |
[production] |
| 09:41 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: repool es1016 with low weight (duration: 01m 02s) |
[production] |
| 09:30 |
<jynus> |
stop and upgrade es1016 |
[production] |