701-750 of 10000 results (87ms)
2024-05-22 ยง
16:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P62926 and previous config saved to /var/cache/conftool/dbconfig/20240522-165558-root.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62925 and previous config saved to /var/cache/conftool/dbconfig/20240522-165340-arnaudb.json [production]
16:40 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P62924 and previous config saved to /var/cache/conftool/dbconfig/20240522-164052-root.json [production]
16:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 10%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62923 and previous config saved to /var/cache/conftool/dbconfig/20240522-163834-arnaudb.json [production]
16:25 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P62922 and previous config saved to /var/cache/conftool/dbconfig/20240522-162546-root.json [production]
16:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 5%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62921 and previous config saved to /var/cache/conftool/dbconfig/20240522-162327-arnaudb.json [production]
16:19 <kamila@deploy1002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply [production]
16:19 <kamila@deploy1002> helmfile [codfw] START helmfile.d/services/recommendation-api: apply [production]
16:13 <James_F> Running `mwscript extensions/WikiLambda/maintenance/migrateZ16K1StringsToZ61s.php --wiki=wikifunctionswiki --implement` on mwmaint1002 for T287153 [production]
16:10 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P62920 and previous config saved to /var/cache/conftool/dbconfig/20240522-161039-root.json [production]
16:08 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
16:08 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
16:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 2%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62919 and previous config saved to /var/cache/conftool/dbconfig/20240522-160821-arnaudb.json [production]
15:56 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Long schema change [production]
15:56 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Long schema change [production]
15:56 <kamila@deploy1002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply [production]
15:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1226', diff saved to https://phabricator.wikimedia.org/P62918 and previous config saved to /var/cache/conftool/dbconfig/20240522-155621-root.json [production]
15:55 <kamila@deploy1002> helmfile [eqiad] START helmfile.d/services/recommendation-api: apply [production]
15:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P62917 and previous config saved to /var/cache/conftool/dbconfig/20240522-155533-root.json [production]
15:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 1%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62916 and previous config saved to /var/cache/conftool/dbconfig/20240522-155315-arnaudb.json [production]
15:50 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2130.codfw.wmnet with OS bookworm [production]
15:44 <elukey> upload to bookworm-wikimedia dragonfly-{dfdaemon,dfget}, calicoctl, calico-cni - T365253 [production]
15:42 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:42 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:42 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-staging2001.codfw.wmnet with OS bookworm [production]
15:40 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:40 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:39 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:39 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:34 <damilare> civicrm upgraded from 8c5fee40 to b0a3965a [production]
15:32 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
15:32 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
15:27 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2130.codfw.wmnet with reason: host reimage [production]
15:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2130.codfw.wmnet with reason: host reimage [production]
15:22 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-staging2001.codfw.wmnet with reason: host reimage [production]
15:19 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-staging2001.codfw.wmnet with reason: host reimage [production]
15:19 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220 (T352010)', diff saved to https://phabricator.wikimedia.org/P62915 and previous config saved to /var/cache/conftool/dbconfig/20240522-151923-ladsgroup.json [production]
15:16 <vgutierrez> repool upload@drmrs with IPIP encapsulation enabled - T357257 [production]
15:16 <fabfur> enabling puppet on all cp-ulsfo (T365566) [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts contint1003.eqiad.wmnet [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: contint1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002" [production]
15:14 <dzahn@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: contint1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002" [production]
15:10 <dzahn@cumin1002> START - Cookbook sre.dns.netbox [production]
15:10 <vgutierrez> rolling restart of pybal on lvs6003 and lvs6002 - T357257 [production]
15:06 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2130.codfw.wmnet with reason: reimage [production]
15:06 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2130.codfw.wmnet with reason: reimage [production]
15:06 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db2130.codfw.wmnet with OS bookworm [production]
15:05 <dzahn@cumin1002> START - Cookbook sre.hosts.decommission for hosts contint1003.eqiad.wmnet [production]
15:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2130', diff saved to https://phabricator.wikimedia.org/P62914 and previous config saved to /var/cache/conftool/dbconfig/20240522-150516-arnaudb.json [production]