2401-2450 of 10000 results (87ms)
2024-02-22 ยง
19:58 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
19:57 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
19:56 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:56 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:55 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
19:53 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
19:52 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage [production]
19:52 <jforrester@deploy2002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
19:52 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
19:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P57764 and previous config saved to /var/cache/conftool/dbconfig/20240222-195108-arnaudb.json [production]
19:50 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
19:50 <jforrester@deploy2002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
19:49 <jforrester@deploy2002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
19:40 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host testvm2002.codfw.wmnet with OS bullseye [production]
19:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P57763 and previous config saved to /var/cache/conftool/dbconfig/20240222-193601-arnaudb.json [production]
19:30 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:30 <robh@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cleanup incorrect asset tags - robh@cumin2002" [production]
19:29 <robh@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cleanup incorrect asset tags - robh@cumin2002" [production]
19:27 <robh@cumin2002> START - Cookbook sre.dns.netbox [production]
19:23 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
19:22 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
19:20 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T357189)', diff saved to https://phabricator.wikimedia.org/P57762 and previous config saved to /var/cache/conftool/dbconfig/20240222-192055-arnaudb.json [production]
19:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2161 (T357189)', diff saved to https://phabricator.wikimedia.org/P57761 and previous config saved to /var/cache/conftool/dbconfig/20240222-191834-arnaudb.json [production]
19:18 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.19 refs T354437 [production]
19:18 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
19:18 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
19:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2154 (T357189)', diff saved to https://phabricator.wikimedia.org/P57760 and previous config saved to /var/cache/conftool/dbconfig/20240222-191810-arnaudb.json [production]
19:14 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2385.codfw.wmnet with OS bullseye [production]
19:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P57759 and previous config saved to /var/cache/conftool/dbconfig/20240222-190304-arnaudb.json [production]
18:49 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2385.codfw.wmnet with reason: host reimage [production]
18:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P57758 and previous config saved to /var/cache/conftool/dbconfig/20240222-184757-arnaudb.json [production]
18:46 <hnowlan@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2385.codfw.wmnet with reason: host reimage [production]
18:44 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2384.codfw.wmnet with OS bullseye [production]
18:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2154 (T357189)', diff saved to https://phabricator.wikimedia.org/P57757 and previous config saved to /var/cache/conftool/dbconfig/20240222-183251-arnaudb.json [production]
18:31 <hnowlan@cumin1002> START - Cookbook sre.hosts.reimage for host mw2385.codfw.wmnet with OS bullseye [production]
18:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2154 (T357189)', diff saved to https://phabricator.wikimedia.org/P57756 and previous config saved to /var/cache/conftool/dbconfig/20240222-183030-arnaudb.json [production]
18:30 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance [production]
18:30 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance [production]
18:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2152 (T357189)', diff saved to https://phabricator.wikimedia.org/P57755 and previous config saved to /var/cache/conftool/dbconfig/20240222-183009-arnaudb.json [production]
18:28 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1485.eqiad.wmnet with OS bullseye [production]
18:25 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1467.eqiad.wmnet with OS bullseye [production]
18:24 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1494.eqiad.wmnet with OS bullseye [production]
18:22 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw [production]
18:22 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw [production]
18:22 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1484.eqiad.wmnet with OS bullseye [production]
18:21 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2384.codfw.wmnet with reason: host reimage [production]
18:18 <hnowlan@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2384.codfw.wmnet with reason: host reimage [production]
18:17 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1468.eqiad.wmnet with OS bullseye [production]
18:15 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P57753 and previous config saved to /var/cache/conftool/dbconfig/20240222-181502-arnaudb.json [production]
18:14 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1483.eqiad.wmnet with OS bullseye [production]