151-200 of 10000 results (83ms)
2024-05-16 ยง
13:47 <jsn@deploy1002> Started scap: Backport for [[gerrit:1032429|Make EntitySchemaValue::getArrayValue() match EntityIdValue (T362955 T362001)]], [[gerrit:1032430|Make EntitySchemaValue::getArrayValue() match EntityIdValue (T362955 T362001)]] [production]
13:37 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2176.codfw.wmnet with reason: host reimage [production]
13:34 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2176.codfw.wmnet with reason: host reimage [production]
13:32 <jsn@deploy1002> Finished scap: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]] (duration: 18m 18s) [production]
13:31 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1024.eqiad.wmnet with reason: host reimage [production]
13:27 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on es1024.eqiad.wmnet with reason: host reimage [production]
13:19 <jsn@deploy1002> jsn and hnowlan: Continuing with sync [production]
13:18 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T352010)', diff saved to https://phabricator.wikimedia.org/P62503 and previous config saved to /var/cache/conftool/dbconfig/20240516-131800-ladsgroup.json [production]
13:17 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db2176.codfw.wmnet with OS bookworm [production]
13:16 <jsn@deploy1002> jsn and hnowlan: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:15 <arnaudb@cumin1002> END (ERROR) - Cookbook sre.mysql.upgrade (exit_code=97) for db2176.codfw.wmnet [production]
13:15 <arnaudb@cumin1002> START - Cookbook sre.mysql.upgrade for db2176.codfw.wmnet [production]
13:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2176', diff saved to https://phabricator.wikimedia.org/P62502 and previous config saved to /var/cache/conftool/dbconfig/20240516-131429-arnaudb.json [production]
13:14 <jsn@deploy1002> Started scap: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]] [production]
13:12 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host es1024.eqiad.wmnet with OS bookworm [production]
13:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1024 T364289', diff saved to https://phabricator.wikimedia.org/P62501 and previous config saved to /var/cache/conftool/dbconfig/20240516-131111-root.json [production]
13:02 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P62500 and previous config saved to /var/cache/conftool/dbconfig/20240516-130252-ladsgroup.json [production]
12:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P62499 and previous config saved to /var/cache/conftool/dbconfig/20240516-124743-ladsgroup.json [production]
10:48 <fnegri@cumin1002> END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0) [production]
10:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1202 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P62497 and previous config saved to /var/cache/conftool/dbconfig/20240516-104601-ladsgroup.json [production]
10:43 <claime> New redirects for T25216 T204830 T31186 operational [production]
10:37 <fnegri@cumin1002> START - Cookbook sre.wikireplicas.update-views [production]
10:32 <claime> cumin 'A:all-mw' -b30 "run-puppet-agent -q" - T25216 T204830 T31186 [production]
10:31 <claime> cumin 'A:all-mw' "enable-puppet 'New redirects T25216 T204830 T31186 - cgoubert'" [production]
10:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Test pc4 master switch', diff saved to https://phabricator.wikimedia.org/P62496 and previous config saved to /var/cache/conftool/dbconfig/20240516-103148-marostegui.json [production]
10:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1202 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P62495 and previous config saved to /var/cache/conftool/dbconfig/20240516-103055-ladsgroup.json [production]
10:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Test pc4 master switch', diff saved to https://phabricator.wikimedia.org/P62494 and previous config saved to /var/cache/conftool/dbconfig/20240516-103039-marostegui.json [production]
10:29 <cgoubert@deploy1002> Finished scap: Deploy new redirects to mw-on-k8s - T25216 T204830 T31186 (duration: 08m 06s) [production]
10:22 <cgoubert@deploy1002> Started scap: Deploy new redirects to mw-on-k8s - T25216 T204830 T31186 [production]
10:21 <claime> New redirects ok on mwdebug - T25216 T204830 T31186 [production]
10:19 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:19 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:18 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2015 and pc1015 to pc4 as depooled spares T362786', diff saved to https://phabricator.wikimedia.org/P62493 and previous config saved to /var/cache/conftool/dbconfig/20240516-101829-marostegui.json [production]
10:15 <marostegui@cumin1002> dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P62492 and previous config saved to /var/cache/conftool/dbconfig/20240516-101553-root.json [production]
10:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1202 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P62491 and previous config saved to /var/cache/conftool/dbconfig/20240516-101548-ladsgroup.json [production]
10:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2016 and pc1016 to pc4 T362786', diff saved to https://phabricator.wikimedia.org/P62490 and previous config saved to /var/cache/conftool/dbconfig/20240516-101543-marostegui.json [production]
10:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2014 and pc1014 to pc4 T362786', diff saved to https://phabricator.wikimedia.org/P62489 and previous config saved to /var/cache/conftool/dbconfig/20240516-101122-marostegui.json [production]
10:10 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: post fix repool', diff saved to https://phabricator.wikimedia.org/P62488 and previous config saved to /var/cache/conftool/dbconfig/20240516-101018-arnaudb.json [production]
10:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2013 and pc1013 to pc2 T362786', diff saved to https://phabricator.wikimedia.org/P62487 and previous config saved to /var/cache/conftool/dbconfig/20240516-101009-marostegui.json [production]
10:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2012 and pc1012 to pc2 T362786', diff saved to https://phabricator.wikimedia.org/P62486 and previous config saved to /var/cache/conftool/dbconfig/20240516-100858-marostegui.json [production]
10:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc2011 to pc1 T362786', diff saved to https://phabricator.wikimedia.org/P62485 and previous config saved to /var/cache/conftool/dbconfig/20240516-100744-marostegui.json [production]
10:04 <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc1011 to pc1 T362786', diff saved to https://phabricator.wikimedia.org/P62484 and previous config saved to /var/cache/conftool/dbconfig/20240516-100418-marostegui.json [production]
10:02 <claime> cumin 'A:all-mw' "disable-puppet 'New redirects T25216 T204830 T31186 - cgoubert'" [production]
10:00 <marostegui@cumin1002> dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P62483 and previous config saved to /var/cache/conftool/dbconfig/20240516-100040-root.json [production]
09:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db1202 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P62482 and previous config saved to /var/cache/conftool/dbconfig/20240516-095927-ladsgroup.json [production]
09:58 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1227 (T352010)', diff saved to https://phabricator.wikimedia.org/P62481 and previous config saved to /var/cache/conftool/dbconfig/20240516-095817-ladsgroup.json [production]
09:58 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance [production]
09:57 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance [production]
09:56 <ladsgroup@cumin1002> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]
09:56 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [production]