|
2020-11-27
§
|
| 10:06 |
<jayme> |
updated helm and helmfile on deploy2001 |
[production] |
| 10:04 |
<jayme@deploy2001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 10:00 |
<jayme> |
imported helm 2.17.0 into buster-wikimedia and stretch-wikimedia |
[production] |
| 08:55 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:05 |
<elukey> |
roll restart druid public cluster for openjdk upgrades |
[production] |
| 08:04 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 06:39 |
<marostegui> |
Stop mysql on es1015 T268810 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove es1015 from dbctl', diff saved to https://phabricator.wikimedia.org/P13454 and previous config saved to /var/cache/conftool/dbconfig/20201127-063846-marostegui.json |
[production] |
| 06:30 |
<marostegui> |
Remove es1016 from tendril and zarcillo T268812 |
[production] |
| 06:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1015 for decommissioning T268810', diff saved to https://phabricator.wikimedia.org/P13453 and previous config saved to /var/cache/conftool/dbconfig/20201127-061929-marostegui.json |
[production] |
|
2020-11-26
§
|
| 17:18 |
<jayme> |
downgrade helmfile to 0.125.2-1 on deploy* |
[production] |
| 17:05 |
<jayme> |
updated helm-diff and helmfile on deploy100* and deploy200* |
[production] |
| 16:34 |
<jayme> |
imported helm-diff 3.1.3-1 into buster-wikimedia and stretch-wikimedia |
[production] |
| 15:01 |
<moritzm> |
installing libonig security updates |
[production] |
| 14:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13452 and previous config saved to /var/cache/conftool/dbconfig/20201126-144446-root.json |
[production] |
| 14:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 14:36 |
<moritzm> |
installing zeromq3 security updates for stretch |
[production] |
| 14:35 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:35 |
<jbond42> |
failing idp back to idp2001 |
[production] |
| 14:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13451 and previous config saved to /var/cache/conftool/dbconfig/20201126-142942-root.json |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet |
[production] |
| 14:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 14:23 |
<moritzm> |
remove labtestpuppetmaster2001 from debmonitor T258103 |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1126 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13450 and previous config saved to /var/cache/conftool/dbconfig/20201126-141439-root.json |
[production] |
| 13:52 |
<elukey> |
roll restart druid daemons on druid analytics to pick up new openjdk upgrades |
[production] |
| 13:52 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:52 |
<root@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:52 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 13:50 |
<moritzm> |
installing python3.5 security updates |
[production] |
| 13:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 for schema change', diff saved to https://phabricator.wikimedia.org/P13449 and previous config saved to /var/cache/conftool/dbconfig/20201126-133204-marostegui.json |
[production] |
| 13:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13448 and previous config saved to /var/cache/conftool/dbconfig/20201126-132918-root.json |
[production] |
| 13:19 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13447 and previous config saved to /var/cache/conftool/dbconfig/20201126-131414-root.json |
[production] |
| 13:07 |
<hnowlan> |
testing depooling kartotherian on maps2004 to reduce load |
[production] |
| 13:07 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet |
[production] |
| 13:01 |
<jbond42> |
update puppet_compiler on compiler1003 |
[production] |
| 12:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13446 and previous config saved to /var/cache/conftool/dbconfig/20201126-125911-root.json |
[production] |
| 12:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1109 for schema change', diff saved to https://phabricator.wikimedia.org/P13445 and previous config saved to /var/cache/conftool/dbconfig/20201126-124253-marostegui.json |
[production] |
| 12:31 |
<jbond42> |
fail over idp.wikimedia.org |
[production] |
| 11:55 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:53 |
<moritzm> |
rebooting seaborgium for kernel update |
[production] |
| 11:53 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:40 |
<marostegui> |
Deploy schema change on s8 codfw - there will be lag on s8 codfw - T268004 |
[production] |
| 11:16 |
<moritzm> |
restarting archiva to pick up Java security update |
[production] |
| 10:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13442 and previous config saved to /var/cache/conftool/dbconfig/20201126-104324-root.json |
[production] |
| 10:41 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 10:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13441 and previous config saved to /var/cache/conftool/dbconfig/20201126-102820-root.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13440 and previous config saved to /var/cache/conftool/dbconfig/20201126-101317-root.json |
[production] |