451-500 of 10000 results (89ms)
2024-05-22 ยง
17:58 <swfrench@deploy1002> helmfile [staging] START helmfile.d/services/device-analytics: apply [production]
17:55 <cmooney@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2002.wikimedia.org with OS bookworm [production]
17:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62930 and previous config saved to /var/cache/conftool/dbconfig/20240522-173900-arnaudb.json [production]
17:24 <topranks> Setting DHCP in codfw row A to 'forward-only' mode to troubleshoot DHCP bug T365204 [production]
17:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62929 and previous config saved to /var/cache/conftool/dbconfig/20240522-172354-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62928 and previous config saved to /var/cache/conftool/dbconfig/20240522-170848-arnaudb.json [production]
17:07 <cmooney@cumin1002> START - Cookbook sre.hosts.reimage for host sretest2002.wikimedia.org with OS bookworm [production]
16:58 <ejegg> standalone SmashPig upgraded from a9c5ee43 to edf573bb [production]
16:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P62926 and previous config saved to /var/cache/conftool/dbconfig/20240522-165558-root.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62925 and previous config saved to /var/cache/conftool/dbconfig/20240522-165340-arnaudb.json [production]
16:40 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P62924 and previous config saved to /var/cache/conftool/dbconfig/20240522-164052-root.json [production]
16:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 10%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62923 and previous config saved to /var/cache/conftool/dbconfig/20240522-163834-arnaudb.json [production]
16:25 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P62922 and previous config saved to /var/cache/conftool/dbconfig/20240522-162546-root.json [production]
16:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 5%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62921 and previous config saved to /var/cache/conftool/dbconfig/20240522-162327-arnaudb.json [production]
16:19 <kamila@deploy1002> helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply [production]
16:19 <kamila@deploy1002> helmfile [codfw] START helmfile.d/services/recommendation-api: apply [production]
16:13 <James_F> Running `mwscript extensions/WikiLambda/maintenance/migrateZ16K1StringsToZ61s.php --wiki=wikifunctionswiki --implement` on mwmaint1002 for T287153 [production]
16:10 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P62920 and previous config saved to /var/cache/conftool/dbconfig/20240522-161039-root.json [production]
16:08 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
16:08 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
16:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 2%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62919 and previous config saved to /var/cache/conftool/dbconfig/20240522-160821-arnaudb.json [production]
15:56 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Long schema change [production]
15:56 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Long schema change [production]
15:56 <kamila@deploy1002> helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply [production]
15:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1226', diff saved to https://phabricator.wikimedia.org/P62918 and previous config saved to /var/cache/conftool/dbconfig/20240522-155621-root.json [production]
15:55 <kamila@deploy1002> helmfile [eqiad] START helmfile.d/services/recommendation-api: apply [production]
15:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1192 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P62917 and previous config saved to /var/cache/conftool/dbconfig/20240522-155533-root.json [production]
15:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 1%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P62916 and previous config saved to /var/cache/conftool/dbconfig/20240522-155315-arnaudb.json [production]
15:50 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2130.codfw.wmnet with OS bookworm [production]
15:44 <elukey> upload to bookworm-wikimedia dragonfly-{dfdaemon,dfget}, calicoctl, calico-cni - T365253 [production]
15:42 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:42 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:42 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-staging2001.codfw.wmnet with OS bookworm [production]
15:40 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:40 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:39 <kamila@deploy1002> helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [production]
15:39 <kamila@deploy1002> helmfile [staging] START helmfile.d/services/recommendation-api: apply [production]
15:34 <damilare> civicrm upgraded from 8c5fee40 to b0a3965a [production]
15:32 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/sessionstore: apply [production]
15:32 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/sessionstore: apply [production]
15:27 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2130.codfw.wmnet with reason: host reimage [production]
15:24 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2130.codfw.wmnet with reason: host reimage [production]
15:22 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-staging2001.codfw.wmnet with reason: host reimage [production]
15:19 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-staging2001.codfw.wmnet with reason: host reimage [production]
15:19 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220 (T352010)', diff saved to https://phabricator.wikimedia.org/P62915 and previous config saved to /var/cache/conftool/dbconfig/20240522-151923-ladsgroup.json [production]
15:16 <vgutierrez> repool upload@drmrs with IPIP encapsulation enabled - T357257 [production]
15:16 <fabfur> enabling puppet on all cp-ulsfo (T365566) [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts contint1003.eqiad.wmnet [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:16 <dzahn@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: contint1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1002" [production]