851-900 of 10000 results (55ms)
2020-01-20 §
12:18 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
12:09 <moritzm> removing actinium in Ganeti T224551 [production]
12:08 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
12:07 <jmm@cumin2001> START - Cookbook sre.hosts.decommission [production]
11:43 <moritzm> removing alsafi in Ganeti T224551 [production]
11:41 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
11:40 <jmm@cumin2001> START - Cookbook sre.hosts.decommission [production]
11:32 <jbond42_> reverting untill joes change is finished - add SSL validation to conftool/etcd expected no-op (https://gerrit.wikimedia.org/r/c/operations/puppet/+/561817) [production]
11:30 <jbond42_> add SSL validation to conftool/etcd expected no-op (https://gerrit.wikimedia.org/r/c/operations/puppet/+/561817) [production]
11:14 <vgutierrez> deploying wikiworkshop TLS certificate on the text cluster - T242374 [production]
10:06 <moritzm> removing alcyone/aluminium in Ganeti T224551 [production]
10:06 <moritzm> removing alcyone/aluminium in Ganeti [production]
10:04 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
10:04 <jmm@cumin2001> START - Cookbook sre.hosts.decommission [production]
10:01 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
10:01 <jmm@cumin2001> START - Cookbook sre.hosts.decommission [production]
09:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1129', diff saved to https://phabricator.wikimedia.org/P10225 and previous config saved to /var/cache/conftool/dbconfig/20200120-094445-marostegui.json [production]
09:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10224 and previous config saved to /var/cache/conftool/dbconfig/20200120-093603-marostegui.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10223 and previous config saved to /var/cache/conftool/dbconfig/20200120-092642-marostegui.json [production]
09:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1129', diff saved to https://phabricator.wikimedia.org/P10222 and previous config saved to /var/cache/conftool/dbconfig/20200120-091929-marostegui.json [production]
09:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P10221 and previous config saved to /var/cache/conftool/dbconfig/20200120-090850-marostegui.json [production]
09:06 <marostegui> Upgrade db1129 [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129', diff saved to https://phabricator.wikimedia.org/P10220 and previous config saved to /var/cache/conftool/dbconfig/20200120-090617-marostegui.json [production]
09:05 <moritzm> restarting CAS to pick up Java security updates [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10219 and previous config saved to /var/cache/conftool/dbconfig/20200120-090336-marostegui.json [production]
09:01 <moritzm> installing Java security updates on an-conf* [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10218 and previous config saved to /var/cache/conftool/dbconfig/20200120-085537-marostegui.json [production]
08:51 <marostegui> Upgrade db1139:3311 db1139:3316 [production]
08:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P10217 and previous config saved to /var/cache/conftool/dbconfig/20200120-084908-marostegui.json [production]
08:44 <marostegui> Upgrade db1094 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094', diff saved to https://phabricator.wikimedia.org/P10216 and previous config saved to /var/cache/conftool/dbconfig/20200120-084408-marostegui.json [production]
08:10 <marostegui> Compare data on db2085:3318 - T243148 [production]
08:07 <ema> powercycle cp3061 T238305 [production]
07:15 <marostegui> Remove partitions from revision on db2084:3314 T239453 [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2084 T239453', diff saved to https://phabricator.wikimedia.org/P10215 and previous config saved to /var/cache/conftool/dbconfig/20200120-071513-marostegui.json [production]
07:10 <marostegui> Stop MySQL on es2020 to clone es2021 - T243052 [production]
06:09 <marostegui> Stop replication on db1107 [production]
06:08 <marostegui> Compress db1121 - T232446 [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121, pool db1084 into vslow T232446', diff saved to https://phabricator.wikimedia.org/P10214 and previous config saved to /var/cache/conftool/dbconfig/20200120-060759-marostegui.json [production]
2020-01-19 §
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3311, db2085:3318 T243148', diff saved to https://phabricator.wikimedia.org/P10210 and previous config saved to /var/cache/conftool/dbconfig/20200119-120236-marostegui.json [production]
11:20 <elukey> restart-php-fpm on mw2181 to rule out temporary php-related issues in codfw [production]
00:46 <cdanis> T238305 cp3053.mgmt /admin1-> racadm serveraction hardreset [production]
2020-01-18 §
11:35 <volans|off> upgraded spicerack to 0.0.29 on cumin hosts [production]
09:00 <dcausse> repool wdqs1007 (T242453) [production]
07:05 <marostegui> Remove partitions from enwiki.revision on db2085 T239453 [production]
04:15 <cdanis> cp3065.mgmt: /admin1-> racadm serveraction hardreset T238305 [production]
2020-01-17 §
21:56 <urandom> bootstrapping restbase2023-c — T243000 [production]
21:17 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
20:07 <urandom> bootstrapping restbase2023-b — T243000 [production]