|
2020-11-26
§
|
| 17:18 |
<jayme> |
downgrade helmfile to 0.125.2-1 on deploy* |
[production] |
| 17:05 |
<jayme> |
updated helm-diff and helmfile on deploy100* and deploy200* |
[production] |
| 16:34 |
<jayme> |
imported helm-diff 3.1.3-1 into buster-wikimedia and stretch-wikimedia |
[production] |
| 15:01 |
<moritzm> |
installing libonig security updates |
[production] |
| 14:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13452 and previous config saved to /var/cache/conftool/dbconfig/20201126-144446-root.json |
[production] |
| 14:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 14:36 |
<moritzm> |
installing zeromq3 security updates for stretch |
[production] |
| 14:35 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:35 |
<jbond42> |
failing idp back to idp2001 |
[production] |
| 14:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13451 and previous config saved to /var/cache/conftool/dbconfig/20201126-142942-root.json |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 14:23 |
<moritzm> |
remove labtestpuppetmaster2001 from debmonitor T258103 |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13450 and previous config saved to /var/cache/conftool/dbconfig/20201126-141439-root.json |
[production] |
| 13:52 |
<elukey> |
roll restart druid daemons on druid analytics to pick up new openjdk upgrades |
[production] |
| 13:52 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:52 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:52 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 13:50 |
<moritzm> |
installing python3.5 security updates |
[production] |
| 13:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 for schema change', diff saved to https://phabricator.wikimedia.org/P13449 and previous config saved to /var/cache/conftool/dbconfig/20201126-133204-marostegui.json |
[production] |
| 13:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13448 and previous config saved to /var/cache/conftool/dbconfig/20201126-132918-root.json |
[production] |
| 13:19 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13447 and previous config saved to /var/cache/conftool/dbconfig/20201126-131414-root.json |
[production] |
| 13:07 |
<hnowlan> |
testing depooling kartotherian on maps2004 to reduce load |
[production] |
| 13:07 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:01 |
<jbond42> |
update puppet_compiler on compiler1003 |
[production] |
| 12:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13446 and previous config saved to /var/cache/conftool/dbconfig/20201126-125911-root.json |
[production] |
| 12:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1109 for schema change', diff saved to https://phabricator.wikimedia.org/P13445 and previous config saved to /var/cache/conftool/dbconfig/20201126-124253-marostegui.json |
[production] |
| 12:31 |
<jbond42> |
fail over idp.wikimedia.org |
[production] |
| 11:55 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:53 |
<moritzm> |
rebooting seaborgium for kernel update |
[production] |
| 11:53 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:40 |
<marostegui> |
Deploy schema change on s8 codfw - there will be lag on s8 codfw - T268004 |
[production] |
| 11:16 |
<moritzm> |
restarting archiva to pick up Java security update |
[production] |
| 10:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13442 and previous config saved to /var/cache/conftool/dbconfig/20201126-104324-root.json |
[production] |
| 10:41 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 10:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13441 and previous config saved to /var/cache/conftool/dbconfig/20201126-102820-root.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13440 and previous config saved to /var/cache/conftool/dbconfig/20201126-101317-root.json |
[production] |
| 09:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13439 and previous config saved to /var/cache/conftool/dbconfig/20201126-095813-root.json |
[production] |
| 09:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136 for schema change', diff saved to https://phabricator.wikimedia.org/P13438 and previous config saved to /var/cache/conftool/dbconfig/20201126-094729-marostegui.json |
[production] |
| 09:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1094 after schema change', diff saved to https://phabricator.wikimedia.org/P13437 and previous config saved to /var/cache/conftool/dbconfig/20201126-094702-marostegui.json |
[production] |
| 09:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1094 for schema change', diff saved to https://phabricator.wikimedia.org/P13436 and previous config saved to /var/cache/conftool/dbconfig/20201126-094639-marostegui.json |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13435 and previous config saved to /var/cache/conftool/dbconfig/20201126-094538-root.json |
[production] |
| 09:38 |
<marostegui> |
Stop mysql on es1016 for decommission |
[production] |