2901-2950 of 10000 results (46ms)
2021-03-10 ยง
11:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 60%: 10', diff saved to https://phabricator.wikimedia.org/P14726 and previous config saved to /var/cache/conftool/dbconfig/20210310-110359-root.json [production]
11:00 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2038.codfw.wmnet [production]
11:00 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1028.eqiad.wmnet [production]
10:55 <kormat@cumin1001> dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14725 and previous config saved to /var/cache/conftool/dbconfig/20210310-105545-kormat.json [production]
10:54 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1023.eqiad.wmnet [production]
10:51 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2037.codfw.wmnet [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3317 (re)pooling @ 30%: 10', diff saved to https://phabricator.wikimedia.org/P14724 and previous config saved to /var/cache/conftool/dbconfig/20210310-104856-root.json [production]
10:47 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2037.codfw.wmnet [production]
10:45 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2036.codfw.wmnet [production]
10:40 <kormat@cumin1001> dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14723 and previous config saved to /var/cache/conftool/dbconfig/20210310-104042-kormat.json [production]
10:40 <effie> upgrade memcached on mc2019, mc1019 [production]
10:39 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2036.codfw.wmnet [production]
10:38 <kormat@cumin1001> dbctl commit (dc=all): 'db1168 depooling: schema change T267767', diff saved to https://phabricator.wikimedia.org/P14722 and previous config saved to /var/cache/conftool/dbconfig/20210310-103836-kormat.json [production]
10:38 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1168.eqiad.wmnet with reason: schema change T267767 [production]
10:38 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1168.eqiad.wmnet with reason: schema change T267767 [production]
10:37 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2035.codfw.wmnet [production]
10:32 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2035.codfw.wmnet [production]
10:29 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudvirt1023.eqiad.wmnet [production]
10:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 for schema change', diff saved to https://phabricator.wikimedia.org/P14721 and previous config saved to /var/cache/conftool/dbconfig/20210310-101922-marostegui.json [production]
10:18 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2034.codfw.wmnet [production]
10:12 <marostegui> Drop testreduce_vd from m5 master - T276787 [production]
10:11 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2034.codfw.wmnet [production]
10:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2033.codfw.wmnet [production]
09:58 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2033.codfw.wmnet [production]
09:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2032.codfw.wmnet [production]
09:52 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2032.codfw.wmnet [production]
09:49 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2031.codfw.wmnet [production]
09:40 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2031.codfw.wmnet [production]
09:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2030.codfw.wmnet [production]
09:30 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2030.codfw.wmnet [production]
09:27 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2029.codfw.wmnet [production]
09:25 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: REIMAGE [production]
09:23 <aborrero@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2003-dev.codfw.wmnet with reason: REIMAGE [production]
09:21 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2029.codfw.wmnet [production]
09:18 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2028.codfw.wmnet [production]
09:12 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be2028.codfw.wmnet [production]
08:39 <marostegui> Upgrade mysql and kernel on db2132 [production]
08:25 <marostegui> Upgrade mysql and kernel on db2078 [production]
08:21 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thorium.eqiad.wmnet [production]
08:20 <moritzm> pruning obsolete kernels from ganeti hosts in eqiad/codfw [production]
08:17 <moritzm> powercycling thorium, stuck on reboot [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1085 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14719 and previous config saved to /var/cache/conftool/dbconfig/20210310-081627-root.json [production]
08:11 <marostegui> Check tables on db1150:3315 - T276742 [production]
08:09 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host thorium.eqiad.wmnet [production]
08:05 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host analytics-tool1001.eqiad.wmnet [production]
08:03 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host analytics-tool1001.eqiad.wmnet [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1085 (re)pooling @ 60%: 10', diff saved to https://phabricator.wikimedia.org/P14718 and previous config saved to /var/cache/conftool/dbconfig/20210310-080123-root.json [production]
07:52 <marostegui> Deploy schema change on s7 codfw (lag will appear) T276150 T276156 [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1085 (re)pooling @ 30%: 10', diff saved to https://phabricator.wikimedia.org/P14717 and previous config saved to /var/cache/conftool/dbconfig/20210310-074618-root.json [production]
07:33 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet [production]