4851-4900 of 10000 results (99ms)
2023-06-20 ยง
14:11 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4051.ulsfo.wmnet} and A:cp [production]
14:07 <vgutierrez@cumin1001> START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4051.ulsfo.wmnet} and A:cp [production]
14:06 <vgutierrez> test HAProxy 2.6.14 on cp4044 and cp4051 [production]
14:03 <vgutierrez> fetch HAProxy 2.6.14 on thirdparty/haproxy26 for bullseye (apt.wm.o) [production]
13:22 <vgutierrez> repooling cp3050 - T339898 [production]
13:22 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
13:18 <moritzm> installing python2.7 security updates [production]
13:15 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts otrs1001.eqiad.wmnet [production]
13:15 <aokoth@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:15 <aokoth@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: otrs1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1001" [production]
13:14 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:930189|Enable Extension:Translate on pt.wikisource.org (T339139)]] (duration: 09m 11s) [production]
13:13 <aokoth@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: otrs1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1001" [production]
13:10 <aokoth@cumin1001> START - Cookbook sre.dns.netbox [production]
13:06 <urbanecm@deploy1002> albertoleoncio and urbanecm: Backport for [[gerrit:930189|Enable Extension:Translate on pt.wikisource.org (T339139)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
13:05 <urbanecm> Create ext:Translate tables on ptwikisource (T339139) [production]
13:04 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:930189|Enable Extension:Translate on pt.wikisource.org (T339139)]] [production]
13:04 <aokoth@cumin1001> START - Cookbook sre.hosts.decommission for hosts otrs1001.eqiad.wmnet [production]
13:04 <urbanecm> Start foreachwikiindblist 'group2 & s1' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all on a tmux in mwmaint1002 (T315510) [production]
12:58 <jclark@cumin1001> START - Cookbook sre.hosts.reboot-single for host parse1002.eqiad.wmnet [production]
12:57 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts parse1002.eqiad.wmnet [production]
12:47 <aokoth@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts otrs1001.eqiad.wmnet [production]
12:46 <aokoth@cumin1001> START - Cookbook sre.hosts.decommission for hosts otrs1001.eqiad.wmnet [production]
12:37 <vgutierrez> depooling cp3050 - T339898 [production]
12:32 <klausman@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [production]
12:32 <klausman@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop: apply [production]
12:26 <klausman@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop: apply [production]
12:25 <klausman@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop: apply [production]
11:27 <jnuche@deploy1002> deploy aborted: (no justification provided) (duration: 01m 32s) [production]
11:26 <jnuche@deploy1002> Started deploy [releng/jenkins-deploy@0c82f2d] (releasing): (no justification provided) [production]
11:15 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:14 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:13 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:13 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:10 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:10 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
10:57 <isaranto@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:30 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:931306|Stop setting wgLegacyEncdoing (T128150 T128151)]] (duration: 08m 06s) [production]
10:23 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:931306|Stop setting wgLegacyEncdoing (T128150 T128151)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
10:22 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:931306|Stop setting wgLegacyEncdoing (T128150 T128151)]] [production]
10:16 <Lucas_WMDE> deployed patches for T339111 [production]
09:35 <isaranto@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
09:23 <ariel@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1003.eqiad.wmnet with OS bullseye [production]
09:20 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
09:02 <jmm@cumin2002> END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:elastic-eqiad [production]
08:39 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1119.eqiad.wmnet with OS bookworm [production]
08:37 <ariel@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1003.eqiad.wmnet with OS bullseye [production]
08:37 <jmm@cumin2002> START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:elastic-eqiad [production]
08:06 <ariel@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1003.eqiad.wmnet with OS bullseye [production]
07:40 <jmm@cumin2002> END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:elastic-codfw [production]
07:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1119.eqiad.wmnet with reason: host reimage [production]