2701-2750 of 10000 results (85ms)
2023-03-03 §
09:01 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade [production]
08:54 <elukey> restart pybal on lvs2010 (standby) and then on lvs2009 (active) to pick up monitoring change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/893008) [production]
08:48 <elukey> restart pybal on lvs1020 (standby) and then on lvs1019 (active) to pick up monitoring change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/893008) [production]
08:45 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade [production]
08:36 <vgutierrez> restarting ntp in dns1001 [production]
07:29 <elukey> truncate /var/log/auth.log.1 on krb1001 to free space (root partition almost filled up) [production]
01:12 <mutante> releases1002: deleting /usr/local/sbin/sync-srv-org-wikimedia-reprepro-releases1002.eqiad.wmnet which confusingly contains an rsync command to rsync from releases1001 which does not exist anymore T330960 [production]
00:13 <mutante> switching releases.wikimedia.org from eqiad to codfw - T330960 [production]
2023-03-02 §
23:40 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs[2001-2003].codfw.wmnet [production]
23:40 <ryankemper@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:39 <ryankemper@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs[2001-2003].codfw.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" [production]
22:45 <ryankemper@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs[2001-2003].codfw.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002" [production]
22:37 <ryankemper@cumin2002> START - Cookbook sre.dns.netbox [production]
22:11 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts wdqs[2001-2003].codfw.wmnet [production]
21:22 <TheresNoTime> close UTC late backport and config training [production]
21:10 <samtar@deploy2002> Finished scap: Backport for [[gerrit:893810|[itwiki] Assign 'changetags' flag only to sysop/bot/botadmin (T331051)]] (duration: 08m 03s) [production]
21:04 <samtar@deploy2002> superpes and samtar: Backport for [[gerrit:893810|[itwiki] Assign 'changetags' flag only to sysop/bot/botadmin (T331051)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
21:04 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2001.wikimedia.org with OS bullseye [production]
21:02 <samtar@deploy2002> Started scap: Backport for [[gerrit:893810|[itwiki] Assign 'changetags' flag only to sysop/bot/botadmin (T331051)]] [production]
21:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-fe2004.codfw.wmnet with OS bullseye [production]
21:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
20:52 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
20:43 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns1001.wikimedia.org with OS bullseye [production]
20:43 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2001.wikimedia.org with reason: host reimage [production]
20:40 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage [production]
20:39 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns2001.wikimedia.org with reason: host reimage [production]
20:37 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2004.codfw.wmnet with reason: host reimage [production]
20:23 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host dns2001.wikimedia.org with OS bullseye [production]
20:23 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns1001.wikimedia.org with reason: host reimage [production]
20:20 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns1001.wikimedia.org with reason: host reimage [production]
20:12 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host thanos-fe2004.codfw.wmnet with OS bullseye [production]
20:08 <brett@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:07 <brett@cumin2002> START - Cookbook sre.dns.netbox [production]
20:04 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host dns1001.wikimedia.org with OS bullseye [production]
19:59 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2014.codfw.wmnet with OS bullseye [production]
19:59 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:44 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:31 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2014.codfw.wmnet with reason: host reimage [production]
19:27 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2014.codfw.wmnet with reason: host reimage [production]
19:05 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ms-fe2014.codfw.wmnet with OS bullseye [production]
18:10 <bd808@deploy2002> helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [production]
18:10 <bd808@deploy2002> helmfile [codfw] START helmfile.d/services/developer-portal: apply [production]
18:10 <bd808@deploy2002> helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [production]
18:09 <bd808@deploy2002> helmfile [eqiad] START helmfile.d/services/developer-portal: apply [production]
18:09 <bd808@deploy2002> helmfile [staging] DONE helmfile.d/services/developer-portal: apply [production]
18:08 <bd808@deploy2002> helmfile [staging] START helmfile.d/services/developer-portal: apply [production]
17:09 <oblivian@deploy2002> Finished scap: Backport for [[gerrit:893796|filebackend: hotfix - make swift master follow the mediawiki master (T330942)]] (duration: 09m 16s) [production]
17:01 <oblivian@deploy2002> oblivian: Backport for [[gerrit:893796|filebackend: hotfix - make swift master follow the mediawiki master (T330942)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
16:59 <oblivian@deploy2002> Started scap: Backport for [[gerrit:893796|filebackend: hotfix - make swift master follow the mediawiki master (T330942)]] [production]
15:59 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]