6201-6250 of 10000 results (100ms)
2023-04-05 ยง
11:15 <ladsgroup@deploy2002> Started scap: Backport for [[gerrit:905609|Revert "Revert "Revert "Revert "mwscript: Switch to use run.php"""" (T326800)]] [production]
11:14 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
11:12 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
11:12 <moritzm> installing systemd security updates on buster [production]
11:12 <slyngshede@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host testvm2002.codfw.wmnet with OS bullseye [production]
11:10 <hnowlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
11:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1100 with 1% weight', diff saved to https://phabricator.wikimedia.org/P46053 and previous config saved to /var/cache/conftool/dbconfig/20230405-110717-root.json [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1130 to s5 primary T331302', diff saved to https://phabricator.wikimedia.org/P46052 and previous config saved to /var/cache/conftool/dbconfig/20230405-110530-root.json [production]
11:05 <marostegui> Starting s5 eqiad failover from db1100 to db1130 - T331302 [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 2%: Repooling', diff saved to https://phabricator.wikimedia.org/P46051 and previous config saved to /var/cache/conftool/dbconfig/20230405-110237-root.json [production]
11:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1122 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P46050 and previous config saved to /var/cache/conftool/dbconfig/20230405-110022-root.json [production]
11:00 <cgoubert@deploy2002> helmfile [staging] DONE helmfile.d/services/termbox: apply [production]
11:00 <cgoubert@deploy2002> helmfile [staging] START helmfile.d/services/termbox: apply [production]
10:59 <slyngshede@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
10:59 <hnowlan@deploy2002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
10:56 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
10:56 <slyngshede@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
10:50 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
10:50 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
10:50 <hnowlan@deploy2002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
10:49 <hnowlan@deploy2002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
10:48 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
10:48 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
10:47 <slyngshede@cumin1001> START - Cookbook sre.ganeti.reimage for host testvm2002.codfw.wmnet with OS bullseye [production]
10:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P46049 and previous config saved to /var/cache/conftool/dbconfig/20230405-104732-root.json [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1122 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P46048 and previous config saved to /var/cache/conftool/dbconfig/20230405-104517-root.json [production]
10:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1130 with weight 0 T331302', diff saved to https://phabricator.wikimedia.org/P46047 and previous config saved to /var/cache/conftool/dbconfig/20230405-104422-marostegui.json [production]
10:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T331302 [production]
10:43 <hnowlan@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s5 T331302 [production]
10:43 <hnowlan@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:41 <hnowlan@deploy2002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:40 <hnowlan@deploy2002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
10:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1122 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P46046 and previous config saved to /var/cache/conftool/dbconfig/20230405-103012-root.json [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1120 T326669', diff saved to https://phabricator.wikimedia.org/P46044 and previous config saved to /var/cache/conftool/dbconfig/20230405-102215-marostegui.json [production]
10:20 <slyngshede@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host testvm2002.codfw.wmnet with OS bullseye [production]
10:17 <cgoubert@deploy2002> helmfile [staging] DONE helmfile.d/services/termbox: apply [production]
10:17 <cgoubert@deploy2002> helmfile [staging] START helmfile.d/services/termbox: apply [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1122 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P46043 and previous config saved to /var/cache/conftool/dbconfig/20230405-101507-root.json [production]
10:14 <elukey> restart purged on cp5032, cp1082, cp6004, cp1090 - errors after restart of kafka main eqiad brokers [production]
10:12 <elukey> restart purged on cp6015 to verify if connection to brokers failed are only temporary or not [production]
10:11 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1007.eqiad.wmnet with OS bullseye [production]
10:09 <slyngshede@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
10:06 <slyngshede@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1122 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P46041 and previous config saved to /var/cache/conftool/dbconfig/20230405-100003-root.json [production]
09:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1122', diff saved to https://phabricator.wikimedia.org/P46040 and previous config saved to /var/cache/conftool/dbconfig/20230405-095954-marostegui.json [production]
09:57 <slyngshede@cumin1001> START - Cookbook sre.ganeti.reimage for host testvm2002.codfw.wmnet with OS bullseye [production]
09:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1162 to s2 primary T334067', diff saved to https://phabricator.wikimedia.org/P46039 and previous config saved to /var/cache/conftool/dbconfig/20230405-095600-root.json [production]
09:55 <marostegui> Starting s2 eqiad failover from db1122 to db1162 - T334067 [production]
09:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1007.eqiad.wmnet with reason: host reimage [production]