2201-2250 of 10000 results (52ms)
2022-06-28 ยง
20:45 <volans@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host sretest2001.codfw.wmnet [production]
20:45 <volans@cumin2002> START - Cookbook sre.ganeti.makevm for new host sretest2001.codfw.wmnet [production]
20:44 <volans@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host sretest2001.codfw.wmnet [production]
20:44 <volans@cumin2002> START - Cookbook sre.ganeti.makevm for new host sretest2001.codfw.wmnet [production]
20:42 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:37 <volans@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM cuminunpriv1001.eqiad.wmnet [production]
20:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P30589 and previous config saved to /var/cache/conftool/dbconfig/20220628-203701-ladsgroup.json [production]
20:35 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:31 <mforns@deploy1002> Started deploy [analytics/refinery@2f5987d]: Regular analytics weekly train [analytics/refinery@2f5987d] [production]
20:31 <volans@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM cuminunpriv1001.eqiad.wmnet [production]
20:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:23 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts webperf2002.codfw.wmnet [production]
20:23 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P30588 and previous config saved to /var/cache/conftool/dbconfig/20220628-202156-ladsgroup.json [production]
20:19 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
20:16 <mutante> gitlab-runner2004 - fixing /etc/resolv.conf and with that the puppet run, leftover mistake from tests [production]
20:15 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts webperf2002.codfw.wmnet [production]
20:04 <mutante> gitlab-runner* -disabling puppet - deploying firewall change on 2004 first [production]
19:48 <moritzm> restarting etherpad to pick up new nodejs [production]
19:42 <moritzm> restarting turnilo to pick up new nodejs [production]
19:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2005-dev.wikimedia.org with reason: host reimage [production]
19:38 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:38 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2005-dev.wikimedia.org with reason: host reimage [production]
19:36 <moritzm> installing nodejs 12 security updates (as shipped in Debian bullseye) [production]
19:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:22 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcontrol2005-dev.wikimedia.org with OS bullseye [production]
19:14 <ryankemper> T309648 Enabling puppet on just `elastic2053` and running puppet agent. Expecting to see result of https://gerrit.wikimedia.org/r/807623 being that the new s3 user/pass creds are added to the elasticsearch keystore [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:08 <ryankemper> T309648 Disabling puppet across all cirrus hosts in order to test out https://gerrit.wikimedia.org/r/c/operations/puppet/+/807623: `ryankemper@cumin1001:~$ sudo -E cumin 'R:elasticsearch::instance' 'disable-puppet "T309648"'` [production]
19:06 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1497.eqiad.wmnet with OS buster [production]
19:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host clouddb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:06 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1495.eqiad.wmnet with OS buster [production]
19:04 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:03 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host clouddb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:02 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host clouddb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:02 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host clouddb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudgw2003-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:56 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS buster [production]
18:56 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.18 refs T308071 [production]
18:56 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:49 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1498.eqiad.wmnet with OS buster [production]
18:47 <dduvall@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.18 refs T308071 (duration: 27m 34s) [production]
18:45 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1494.eqiad.wmnet with OS buster [production]
18:36 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host cloudgw2003-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:35 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1493.eqiad.wmnet with OS buster [production]
18:35 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host clouddb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]