|
2020-11-27
§
|
| 10:49 |
<jayme> |
updated helm to 2.17.0-1 on deploy*,contint*,chartmuseum* |
[production] |
| 10:29 |
<elukey> |
restart eventlogging_to_druid_editattemptstep_hourly on an-launcher1002 (failed) to see if the hive metastore works |
[analytics] |
| 10:27 |
<elukey> |
restart oozie and presto-server on an-coord1001 for openjdk upgrades |
[analytics] |
| 10:27 |
<elukey> |
restart hive server and metastore on an-coord1001 - openjdk upgrades + problem with high GC caused by a job |
[analytics] |
| 10:06 |
<jayme> |
updated helm and helmfile on deploy2001 |
[production] |
| 10:04 |
<jayme@deploy2001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 10:00 |
<jayme> |
imported helm 2.17.0 into buster-wikimedia and stretch-wikimedia |
[production] |
| 08:55 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:05 |
<elukey> |
roll restart druid public cluster for openjdk upgrades |
[production] |
| 08:05 |
<elukey> |
roll restart druid public cluster for openjdk upgrades |
[analytics] |
| 08:04 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 06:39 |
<marostegui> |
Stop mysql on es1015 T268810 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove es1015 from dbctl', diff saved to https://phabricator.wikimedia.org/P13454 and previous config saved to /var/cache/conftool/dbconfig/20201127-063846-marostegui.json |
[production] |
| 06:30 |
<marostegui> |
Remove es1016 from tendril and zarcillo T268812 |
[production] |
| 06:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1015 for decommissioning T268810', diff saved to https://phabricator.wikimedia.org/P13453 and previous config saved to /var/cache/conftool/dbconfig/20201127-061929-marostegui.json |
[production] |
|
2020-11-26
§
|
| 22:58 |
<andrewbogott> |
deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x. We need log rotation here it seems. |
[admin] |
| 17:18 |
<jayme> |
downgrade helmfile to 0.125.2-1 on deploy* |
[production] |
| 17:05 |
<jayme> |
updated helm-diff and helmfile on deploy100* and deploy200* |
[production] |
| 16:34 |
<jayme> |
imported helm-diff 3.1.3-1 into buster-wikimedia and stretch-wikimedia |
[production] |
| 15:53 |
<dcaro> |
Created private flavor g2.cores8.ram16.disk1120 (T268190) |
[wikidumpparse] |
| 15:53 |
<dcaro> |
Created private flavor g2.cores8.ram16.disk1120 for wikidumpparse (T268190) |
[admin] |
| 15:01 |
<moritzm> |
installing libonig security updates |
[production] |
| 14:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13452 and previous config saved to /var/cache/conftool/dbconfig/20201126-144446-root.json |
[production] |
| 14:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 14:36 |
<moritzm> |
installing zeromq3 security updates for stretch |
[production] |
| 14:35 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:35 |
<jbond42> |
failing idp back to idp2001 |
[production] |
| 14:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13451 and previous config saved to /var/cache/conftool/dbconfig/20201126-142942-root.json |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 14:23 |
<moritzm> |
remove labtestpuppetmaster2001 from debmonitor T258103 |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13450 and previous config saved to /var/cache/conftool/dbconfig/20201126-141439-root.json |
[production] |
| 13:52 |
<elukey> |
roll restart druid daemons on druid analytics to pick up new openjdk upgrades |
[production] |
| 13:52 |
<elukey> |
roll restart druid daemons on druid analytics to pick up new openjdk upgrades |
[analytics] |
| 13:52 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:52 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:52 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 13:50 |
<moritzm> |
installing python3.5 security updates |
[production] |
| 13:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 for schema change', diff saved to https://phabricator.wikimedia.org/P13449 and previous config saved to /var/cache/conftool/dbconfig/20201126-133204-marostegui.json |
[production] |
| 13:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13448 and previous config saved to /var/cache/conftool/dbconfig/20201126-132918-root.json |
[production] |
| 13:19 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13447 and previous config saved to /var/cache/conftool/dbconfig/20201126-131414-root.json |
[production] |
| 13:08 |
<elukey> |
force umount/mount of all /mnt/hdfs mountpoints to pick up opendjdk upgrades |
[analytics] |
| 13:07 |
<hnowlan> |
testing depooling kartotherian on maps2004 to reduce load |
[production] |
| 13:07 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:01 |
<jbond42> |
update puppet_compiler on compiler1003 |
[production] |
| 12:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13446 and previous config saved to /var/cache/conftool/dbconfig/20201126-125911-root.json |
[production] |
| 12:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1109 for schema change', diff saved to https://phabricator.wikimedia.org/P13445 and previous config saved to /var/cache/conftool/dbconfig/20201126-124253-marostegui.json |
[production] |