1-50 of 10000 results (21ms)
2020-11-27 §
08:55 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:05 <elukey> roll restart druid public cluster for openjdk upgrades [production]
08:04 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
06:39 <marostegui> Stop mysql on es1015 T268810 [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es1015 from dbctl', diff saved to https://phabricator.wikimedia.org/P13454 and previous config saved to /var/cache/conftool/dbconfig/20201127-063846-marostegui.json [production]
06:30 <marostegui> Remove es1016 from tendril and zarcillo T268812 [production]
06:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:25 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1015 for decommissioning T268810', diff saved to https://phabricator.wikimedia.org/P13453 and previous config saved to /var/cache/conftool/dbconfig/20201127-061929-marostegui.json [production]
2020-11-26 §
17:18 <jayme> downgrade helmfile to 0.125.2-1 on deploy* [production]
17:05 <jayme> updated helm-diff and helmfile on deploy100* and deploy200* [production]
16:34 <jayme> imported helm-diff 3.1.3-1 into buster-wikimedia and stretch-wikimedia [production]
15:01 <moritzm> installing libonig security updates [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13452 and previous config saved to /var/cache/conftool/dbconfig/20201126-144446-root.json [production]
14:38 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
14:36 <moritzm> installing zeromq3 security updates for stretch [production]
14:35 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet [production]
14:35 <jbond42> failing idp back to idp2001 [production]
14:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13451 and previous config saved to /var/cache/conftool/dbconfig/20201126-142942-root.json [production]
14:24 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet [production]
14:24 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet [production]
14:23 <moritzm> remove labtestpuppetmaster2001 from debmonitor T258103 [production]
14:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13450 and previous config saved to /var/cache/conftool/dbconfig/20201126-141439-root.json [production]
13:52 <elukey> roll restart druid daemons on druid analytics to pick up new openjdk upgrades [production]
13:52 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:52 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:52 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
13:50 <moritzm> installing python3.5 security updates [production]
13:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126 for schema change', diff saved to https://phabricator.wikimedia.org/P13449 and previous config saved to /var/cache/conftool/dbconfig/20201126-133204-marostegui.json [production]
13:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13448 and previous config saved to /var/cache/conftool/dbconfig/20201126-132918-root.json [production]
13:19 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet [production]
13:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1109 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13447 and previous config saved to /var/cache/conftool/dbconfig/20201126-131414-root.json [production]
13:07 <hnowlan> testing depooling kartotherian on maps2004 to reduce load [production]
13:07 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet [production]
13:01 <jbond42> update puppet_compiler on compiler1003 [production]
12:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1109 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13446 and previous config saved to /var/cache/conftool/dbconfig/20201126-125911-root.json [production]
12:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1109 for schema change', diff saved to https://phabricator.wikimedia.org/P13445 and previous config saved to /var/cache/conftool/dbconfig/20201126-124253-marostegui.json [production]
12:31 <jbond42> fail over idp.wikimedia.org [production]
11:55 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
11:53 <moritzm> rebooting seaborgium for kernel update [production]
11:53 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
11:40 <marostegui> Deploy schema change on s8 codfw - there will be lag on s8 codfw - T268004 [production]
11:16 <moritzm> restarting archiva to pick up Java security update [production]
10:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13442 and previous config saved to /var/cache/conftool/dbconfig/20201126-104324-root.json [production]
10:41 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13441 and previous config saved to /var/cache/conftool/dbconfig/20201126-102820-root.json [production]
10:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13440 and previous config saved to /var/cache/conftool/dbconfig/20201126-101317-root.json [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13439 and previous config saved to /var/cache/conftool/dbconfig/20201126-095813-root.json [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1136 for schema change', diff saved to https://phabricator.wikimedia.org/P13438 and previous config saved to /var/cache/conftool/dbconfig/20201126-094729-marostegui.json [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1094 after schema change', diff saved to https://phabricator.wikimedia.org/P13437 and previous config saved to /var/cache/conftool/dbconfig/20201126-094702-marostegui.json [production]