4201-4250 of 10000 results (40ms)
2021-02-26 ยง
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'staging' . [production]
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'production' . [production]
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
10:33 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
10:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
10:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
10:31 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
10:31 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:31 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:31 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . [production]
10:31 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'production' . [production]
10:28 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2003-dev.codfw.wmnet [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14505 and previous config saved to /var/cache/conftool/dbconfig/20210226-102254-root.json [production]
10:18 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1096.eqiad.wmnet with reason: REIMAGE [production]
10:16 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1096.eqiad.wmnet with reason: REIMAGE [production]
10:14 <dcaro@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcephosd2003-dev.codfw.wmnet [production]
10:09 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2002-dev.codfw.wmnet [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 85%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14504 and previous config saved to /var/cache/conftool/dbconfig/20210226-100750-root.json [production]
10:06 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudweb2001-dev.wikimedia.org [production]
10:05 <dcaro@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcephosd2002-dev.codfw.wmnet [production]
09:59 <aborrero@cumin2001> START - Cookbook sre.hosts.reboot-single for host cloudweb2001-dev.wikimedia.org [production]
09:59 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2003-dev.wikimedia.org [production]
09:55 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2002-dev.wikimedia.org [production]
09:54 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2001-dev.codfw.wmnet [production]
09:52 <aborrero@cumin2001> START - Cookbook sre.hosts.reboot-single for host cloudservices2003-dev.wikimedia.org [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14503 and previous config saved to /var/cache/conftool/dbconfig/20210226-095247-root.json [production]
09:50 <aborrero@cumin2001> START - Cookbook sre.hosts.reboot-single for host cloudservices2002-dev.wikimedia.org [production]
09:50 <dcaro@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcephosd2001-dev.codfw.wmnet [production]
09:48 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2001-dev.codfw.wmnet [production]
09:43 <dcaro@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcephosd2001-dev.codfw.wmnet [production]
09:41 <aborrero@cumin2001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudcontrol2001-dev.wikimedia.org [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 65%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14502 and previous config saved to /var/cache/conftool/dbconfig/20210226-093743-root.json [production]
09:33 <aborrero@cumin2001> START - Cookbook sre.hosts.reboot-single for host cloudcontrol2001-dev.wikimedia.org [production]
09:28 <root@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:24 <root@cumin1001> START - Cookbook sre.dns.netbox [production]
09:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14501 and previous config saved to /var/cache/conftool/dbconfig/20210226-092240-root.json [production]
09:13 <jbond42> pupet enabled post sudoers fix, running puppet fleet wide with cumin -b 15 '*' 'run-puppet-agent ' [production]
09:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 40%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14500 and previous config saved to /var/cache/conftool/dbconfig/20210226-090736-root.json [production]
08:55 <jbond42> disabled puppet pending rollback of https://gerrit.wikimedia.org/r/666899 [production]
08:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14498 and previous config saved to /var/cache/conftool/dbconfig/20210226-085233-root.json [production]
08:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 15%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14497 and previous config saved to /var/cache/conftool/dbconfig/20210226-083729-root.json [production]
08:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14496 and previous config saved to /var/cache/conftool/dbconfig/20210226-082226-root.json [production]
08:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1058.eqiad.wmnet with reason: REIMAGE [production]
08:17 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1058.eqiad.wmnet with reason: REIMAGE [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 5%: Repool db1169 after cloning db1134', diff saved to https://phabricator.wikimedia.org/P14495 and previous config saved to /var/cache/conftool/dbconfig/20210226-080722-root.json [production]