2951-3000 of 10000 results (35ms)
2020-11-30 §
04:26 <kart_> Updated cxserver to 2020-11-23-050106-production (T262253, T268410) [production]
04:18 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:14 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
04:11 <kartik@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
2020-11-27 §
17:30 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:30 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:50 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:50 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:13 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
15:06 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
14:56 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
14:50 <elukey> roll restart zookeeper on druid* nodes for openjdk upgrades [production]
14:50 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
10:52 <jayme> updated helmfile to 0.135.0-1 on deploy*,contint* [production]
10:51 <jayme> updated helm-diff to 3.1.3-1 on contint* [production]
10:49 <jayme> updated helm to 2.17.0-1 on deploy*,contint*,chartmuseum* [production]
10:06 <jayme> updated helm and helmfile on deploy2001 [production]
10:04 <jayme@deploy2001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
10:00 <jayme> imported helm 2.17.0 into buster-wikimedia and stretch-wikimedia [production]
08:55 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:05 <elukey> roll restart druid public cluster for openjdk upgrades [production]
08:04 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
06:39 <marostegui> Stop mysql on es1015 T268810 [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es1015 from dbctl', diff saved to https://phabricator.wikimedia.org/P13454 and previous config saved to /var/cache/conftool/dbconfig/20201127-063846-marostegui.json [production]
06:30 <marostegui> Remove es1016 from tendril and zarcillo T268812 [production]
06:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:25 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1015 for decommissioning T268810', diff saved to https://phabricator.wikimedia.org/P13453 and previous config saved to /var/cache/conftool/dbconfig/20201127-061929-marostegui.json [production]
2020-11-26 §
17:18 <jayme> downgrade helmfile to 0.125.2-1 on deploy* [production]
17:05 <jayme> updated helm-diff and helmfile on deploy100* and deploy200* [production]
16:34 <jayme> imported helm-diff 3.1.3-1 into buster-wikimedia and stretch-wikimedia [production]
15:01 <moritzm> installing libonig security updates [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13452 and previous config saved to /var/cache/conftool/dbconfig/20201126-144446-root.json [production]
14:38 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
14:36 <moritzm> installing zeromq3 security updates for stretch [production]
14:35 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet [production]
14:35 <jbond42> failing idp back to idp2001 [production]
14:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 66%: After schema change', diff saved to https://phabricator.wikimedia.org/P13451 and previous config saved to /var/cache/conftool/dbconfig/20201126-142942-root.json [production]
14:24 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2001.codfw.wmnet [production]
14:24 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet [production]
14:23 <moritzm> remove labtestpuppetmaster2001 from debmonitor T258103 [production]
14:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1126 (re)pooling @ 33%: After schema change', diff saved to https://phabricator.wikimedia.org/P13450 and previous config saved to /var/cache/conftool/dbconfig/20201126-141439-root.json [production]
13:52 <elukey> roll restart druid daemons on druid analytics to pick up new openjdk upgrades [production]
13:52 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:52 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:52 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
13:50 <moritzm> installing python3.5 security updates [production]
13:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126 for schema change', diff saved to https://phabricator.wikimedia.org/P13449 and previous config saved to /var/cache/conftool/dbconfig/20201126-133204-marostegui.json [production]
13:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13448 and previous config saved to /var/cache/conftool/dbconfig/20201126-132918-root.json [production]
13:19 <hnowlan@puppetmaster1001> conftool action : set/pooled=no:weight=0; selector: dc=codfw,cluster=maps,service=kartotherian,name=maps2004.codfw.wmnet [production]