2151-2200 of 10000 results (23ms)
2021-01-20 ยง
12:31 <godog> bounce icinga on alert1001 [production]
12:28 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'similar-users' for release 'test' . [production]
12:27 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'similar-users' for release 'main' . [production]
12:27 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'similar-users' for release 'canary' . [production]
12:27 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2035.codfw.wmnet [production]
12:20 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2035.codfw.wmnet [production]
12:19 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2034.codfw.wmnet [production]
12:10 <matthiasmullie> EU config window done [production]
12:10 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2034.codfw.wmnet [production]
12:09 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2033.codfw.wmnet [production]
12:08 <mlitn@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 2fc57b259: Remove MediaSearch survey (duration: 01m 10s) [production]
12:02 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2033.codfw.wmnet [production]
12:01 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2032.codfw.wmnet [production]
11:54 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2032.codfw.wmnet [production]
11:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 100%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13847 and previous config saved to /var/cache/conftool/dbconfig/20210120-112808-root.json [production]
11:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 75%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13846 and previous config saved to /var/cache/conftool/dbconfig/20210120-111305-root.json [production]
10:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 50%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13845 and previous config saved to /var/cache/conftool/dbconfig/20210120-105801-root.json [production]
10:53 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2030.codfw.wmnet [production]
10:51 <XioNoX> Discard the non-whitelisted 172.16.0.0/12 traffic - T209082 [production]
10:49 <arturo> merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 (T209082) [admin]
10:48 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2030.codfw.wmnet [production]
10:46 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2029.codfw.wmnet [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 25%: After moving wikireplicas to another host', diff saved to https://phabricator.wikimedia.org/P13844 and previous config saved to /var/cache/conftool/dbconfig/20210120-104257-root.json [production]
10:37 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2029.codfw.wmnet [production]
10:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2028.codfw.wmnet [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079 to stop replication T272008', diff saved to https://phabricator.wikimedia.org/P13842 and previous config saved to /var/cache/conftool/dbconfig/20210120-103449-marostegui.json [production]
10:26 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2028.codfw.wmnet [production]
10:26 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2027.codfw.wmnet [production]
10:17 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2027.codfw.wmnet [production]
10:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2026.codfw.wmnet [production]
10:07 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2026.codfw.wmnet [production]
10:05 <dcaro> Everything looks ok, created a new vm with a volume in ceph without issues, and on warnings/errors on ceph status, closing (T272303) [admin]
10:05 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2025.codfw.wmnet [production]
09:59 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2025.codfw.wmnet [production]
09:57 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2024.codfw.wmnet [production]
09:55 <dcaro> Eqiad ceph cluster uprgaded, doing sanity checks (T272303) [admin]
09:49 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2024.codfw.wmnet [production]
09:47 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2023.codfw.wmnet [production]
09:46 <dcaro> 75% of the eqiad cluster upgraded... continuing (T272303) [admin]
09:39 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2023.codfw.wmnet [production]
09:39 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2021.codfw.wmnet [production]
09:37 <dcaro> 25% of the eqiad cluster upgraded... continuing (T272303) [admin]
09:32 <moritzm> installing cuminunpriv1001 [production]
09:32 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2021.codfw.wmnet [production]
09:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2020.codfw.wmnet [production]
09:24 <dcaro> Mgr daemons upgraded and running, upgrading osd daemons on servers cloudcephosd1*, this make take a bit longer (T272303) [admin]
09:24 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2020.codfw.wmnet [production]
09:22 <dcaro> Mon daemons upgraded and running, upgrading mgr daemons on servers cloudcephmon1* (T272303) [admin]
09:19 <XioNoX> configure Lumen interfaces [production]
09:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2019.codfw.wmnet [production]