2451-2500 of 10000 results (36ms)
2021-03-12 ยง
18:01 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2244.codfw.wmnet [production]
18:00 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2230.codfw.wmnet [production]
18:00 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2229.codfw.wmnet [production]
17:00 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on aqs1010.eqiad.wmnet with reason: New buster host [production]
17:00 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on aqs1010.eqiad.wmnet with reason: New buster host [production]
16:05 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:01 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:56 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:50 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Repool db1170:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P14818 and previous config saved to /var/cache/conftool/dbconfig/20210312-143450-root.json [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Repool db1170:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P14817 and previous config saved to /var/cache/conftool/dbconfig/20210312-141947-root.json [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Repool db1170:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P14816 and previous config saved to /var/cache/conftool/dbconfig/20210312-140443-root.json [production]
13:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Repool db1170:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P14815 and previous config saved to /var/cache/conftool/dbconfig/20210312-134940-root.json [production]
13:24 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1088.eqiad.wmnet [production]
13:14 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1088.eqiad.wmnet [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1170:3312', diff saved to https://phabricator.wikimedia.org/P14814 and previous config saved to /var/cache/conftool/dbconfig/20210312-131033-marostegui.json [production]
12:12 <vgutierrez> restart ats-tls on cp3051 [production]
11:55 <effie> upgrade memcached on mc1022, mc2022 [production]
11:22 <hnowlan> corrected git_server for logstash-logback-encoder, cassandra/twcs and cassandra/metrics-collector on deploy1002 [production]
09:45 <jayme@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [production]
09:45 <jayme@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [production]
09:44 <jayme@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:43 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
09:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx1001.wikimedia.org [production]
09:25 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host mx1001.wikimedia.org [production]
09:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx2001.wikimedia.org [production]
09:11 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host mx2001.wikimedia.org [production]
09:07 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 01m 35s) [production]
09:05 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
09:00 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 09s) [production]
09:00 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
08:59 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 10s) [production]
08:59 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
08:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pybal-test2003.codfw.wmnet [production]
08:49 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host pybal-test2003.codfw.wmnet [production]
08:47 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pybal-test2002.codfw.wmnet [production]
08:44 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host pybal-test2002.codfw.wmnet [production]
08:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host scandium.eqiad.wmnet [production]
08:20 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host scandium.eqiad.wmnet [production]
08:01 <moritzm> installing openjpeg2 security updates [production]
07:16 <marostegui> Stop mysql on db2108 to clone db2148 [production]
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2108 T276742', diff saved to https://phabricator.wikimedia.org/P14811 and previous config saved to /var/cache/conftool/dbconfig/20210312-071628-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 100%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14810 and previous config saved to /var/cache/conftool/dbconfig/20210312-071400-root.json [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2148 T276742', diff saved to https://phabricator.wikimedia.org/P14809 and previous config saved to /var/cache/conftool/dbconfig/20210312-070219-marostegui.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 60%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14808 and previous config saved to /var/cache/conftool/dbconfig/20210312-065857-root.json [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314 for table checking T276742', diff saved to https://phabricator.wikimedia.org/P14807 and previous config saved to /var/cache/conftool/dbconfig/20210312-065008-marostegui.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 30%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14806 and previous config saved to /var/cache/conftool/dbconfig/20210312-064353-root.json [production]
06:30 <marostegui> Deploy schema change on s2 codfw master, lag will appear - T276150 T276156 [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 10%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14805 and previous config saved to /var/cache/conftool/dbconfig/20210312-062850-root.json [production]
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082 for schema change', diff saved to https://phabricator.wikimedia.org/P14804 and previous config saved to /var/cache/conftool/dbconfig/20210312-061306-marostegui.json [production]