351-400 of 10000 results (76ms)
2024-03-05 §
14:33 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for wqds1025 - cmooney@cumin1002" [production]
14:32 <elukey@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
14:32 <elukey@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
14:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 10%: Cloning done', diff saved to https://phabricator.wikimedia.org/P58477 and previous config saved to /var/cache/conftool/dbconfig/20240305-143154-arnaudb.json [production]
14:31 <jnuche@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.21 refs T354439 (duration: 42m 08s) [production]
14:29 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
14:28 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:28 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:26 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:26 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2151 (re)pooling @ 5%: Cloning done', diff saved to https://phabricator.wikimedia.org/P58476 and previous config saved to /var/cache/conftool/dbconfig/20240305-141649-arnaudb.json [production]
14:13 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2151.codfw.wmnet onto db2217.codfw.wmnet [production]
14:04 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:04 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:49 <jnuche@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.21 refs T354439 [production]
13:35 <jiji@deploy2002> Finished scap: (no justification provided) (duration: 22m 47s) [production]
13:33 <jynus> running refreshImageMetadata.php on commons for Алфавітно-предметний_покажчик_за_1938_рік_до_Збірника_постанов_і_розпоряджень_Уряду_Української_Радянської_Соціалістичної_Республіки.pdf [production]
13:28 <jnuche@deploy2002> Finished deploy [zuul/deploy@bb76c45]: test deployment for new host (duration: 00m 04s) [production]
13:28 <jnuche@deploy2002> Started deploy [zuul/deploy@bb76c45]: test deployment for new host [production]
13:24 <arnaudb@cumin1002> START - Cookbook sre.mysql.clone Will create a clone of db2151.codfw.wmnet onto db2217.codfw.wmnet [production]
13:21 <arnaudb@cumin1002> dbctl commit (dc=all): 'Cloning db2151 in db2217 for T355422', diff saved to https://phabricator.wikimedia.org/P58475 and previous config saved to /var/cache/conftool/dbconfig/20240305-132106-arnaudb.json [production]
13:18 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: provisionning db2217.codfw.wmnet - T355422 [production]
13:18 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: provisionning db2217.codfw.wmnet - T355422 [production]
13:18 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2151.codfw.wmnet with reason: provisionning db2217.codfw.wmnet - T355422 [production]
13:18 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2151.codfw.wmnet with reason: provisionning db2217.codfw.wmnet - T355422 [production]
13:17 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2151.codfw.wmnet with reason: Silence for cloning [production]
13:17 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2151.codfw.wmnet with reason: Silence for cloning [production]
13:12 <jiji@deploy2002> Started scap: (no justification provided) [production]
13:11 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
13:11 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
12:57 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
12:56 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
12:54 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:54 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:54 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
12:54 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
12:52 <eoghan@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts vrts1002.eqiad.wmnet [production]
12:52 <eoghan@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:52 <eoghan@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1002" [production]
12:52 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Optimize revision table T354015 [production]
12:52 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Optimize revision table T354015 [production]
12:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1186', diff saved to https://phabricator.wikimedia.org/P58474 and previous config saved to /var/cache/conftool/dbconfig/20240305-125152-root.json [production]
12:51 <eoghan@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1002" [production]
12:48 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
12:48 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
12:46 <eoghan@cumin1002> START - Cookbook sre.dns.netbox [production]
12:35 <eoghan@cumin1002> START - Cookbook sre.hosts.decommission for hosts vrts1002.eqiad.wmnet [production]
12:32 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
12:22 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
12:17 <cgoubert@deploy2002> scap failed: KeyError 'canaries' (duration: 10m 29s) [production]