701-750 of 10000 results (79ms)
2023-05-15 ยง
17:15 <volans@cumin2002> START - Cookbook sre.network.provision for device ssw1-a1-codfw.mgmt.codfw.wmnet [production]
15:00 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Setup Incomplete [production]
15:00 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Setup Incomplete [production]
14:24 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: testing transferpy cookbook [production]
14:24 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: testing transferpy cookbook [production]
14:21 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye [production]
14:20 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
14:20 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
14:17 <klausman@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
14:16 <klausman@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
14:03 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
14:00 <volans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage [production]
13:56 <volans> re-enabled puppet on the install hosts to deploy changes for T336485 [production]
13:45 <volans@cumin1001> START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS bullseye [production]
13:33 <volans> disabling puppet on the install hosts to deploy changes for T336485 [production]
13:00 <jelto@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
13:00 <jelto@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
12:58 <jelto@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
12:58 <jelto@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
11:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023 (T335845)', diff saved to https://phabricator.wikimedia.org/P48228 and previous config saved to /var/cache/conftool/dbconfig/20230515-111624-ladsgroup.json [production]
11:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023', diff saved to https://phabricator.wikimedia.org/P48227 and previous config saved to /var/cache/conftool/dbconfig/20230515-110118-ladsgroup.json [production]
10:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023', diff saved to https://phabricator.wikimedia.org/P48226 and previous config saved to /var/cache/conftool/dbconfig/20230515-104611-ladsgroup.json [production]
10:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023 (T335845)', diff saved to https://phabricator.wikimedia.org/P48225 and previous config saved to /var/cache/conftool/dbconfig/20230515-103105-ladsgroup.json [production]
10:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1023 (T335845)', diff saved to https://phabricator.wikimedia.org/P48224 and previous config saved to /var/cache/conftool/dbconfig/20230515-102038-ladsgroup.json [production]
10:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1023.eqiad.wmnet with reason: Maintenance [production]
10:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1023.eqiad.wmnet with reason: Maintenance [production]
10:19 <Amir1> Removing db1123 from zarcillo T334910 [production]
10:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1123.eqiad.wmnet [production]
10:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1123.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001" [production]
10:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1020 (T335845)', diff saved to https://phabricator.wikimedia.org/P48223 and previous config saved to /var/cache/conftool/dbconfig/20230515-101329-ladsgroup.json [production]
10:13 <ladsgroup@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1123.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ladsgroup@cumin1001" [production]
10:11 <ladsgroup@cumin1001> START - Cookbook sre.dns.netbox [production]
10:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1123.eqiad.wmnet [production]
09:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1020', diff saved to https://phabricator.wikimedia.org/P48222 and previous config saved to /var/cache/conftool/dbconfig/20230515-095823-ladsgroup.json [production]
09:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Remove db1123 from dbctl T334910', diff saved to https://phabricator.wikimedia.org/P48221 and previous config saved to /var/cache/conftool/dbconfig/20230515-095412-ladsgroup.json [production]
09:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1123 T334910', diff saved to https://phabricator.wikimedia.org/P48220 and previous config saved to /var/cache/conftool/dbconfig/20230515-094938-ladsgroup.json [production]
09:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1020', diff saved to https://phabricator.wikimedia.org/P48219 and previous config saved to /var/cache/conftool/dbconfig/20230515-094317-ladsgroup.json [production]
09:39 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 15802 [production]
09:38 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 15802 [production]
09:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1020 (T335845)', diff saved to https://phabricator.wikimedia.org/P48218 and previous config saved to /var/cache/conftool/dbconfig/20230515-092810-ladsgroup.json [production]
09:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1020 (T335845)', diff saved to https://phabricator.wikimedia.org/P48217 and previous config saved to /var/cache/conftool/dbconfig/20230515-091139-ladsgroup.json [production]
09:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1020.eqiad.wmnet with reason: Maintenance [production]
09:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1020.eqiad.wmnet with reason: Maintenance [production]
09:08 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: sync [production]
09:05 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: sync [production]
08:45 <jelto@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:45 <jelto@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
08:26 <elukey> restart pybal on lvs2010 and lvs2009 to pick up new LVS VIP for ml-staging k8s ingress - T335756 [production]
08:26 <volans> installed spicerack_7.1.0 on cumin1001 [production]