5901-5950 of 10000 results (90ms)
2023-07-07 §
08:47 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: xhgui2002.codfw.wmnet [production]
08:46 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: xhgui1002.eqiad.wmnet [production]
08:46 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: xhgui1002.eqiad.wmnet [production]
08:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on kafka-test[1006-1010].eqiad.wmnet with reason: resetting cluster [production]
08:05 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on kafka-test[1006-1010].eqiad.wmnet with reason: resetting cluster [production]
01:55 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
00:28 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
2023-07-06 §
23:14 <mutante> mx1001 - rm /usr/local/bin/otrs_aliases ; rm /lib/systemd/system/generate_otrs_aliases.* after deploying gerrit:932316 which renamed script and timer without absenting them [production]
23:08 <mutante> mx2001 - rm /usr/local/bin/otrs_aliases ; rm /lib/systemd/system/generate_otrs_aliases.* after deploying gerrit:932316 which renamed script and timer without absenting them [production]
21:12 <thcipriani@deploy1002> Finished scap: Clean up font directory [[gerrit:723652]] (duration: 06m 33s) [production]
21:10 <bking@deploy1002> Finished deploy [wdqs/wdqs@dff41b7]: 0.3.124 (duration: 14m 56s) [production]
21:06 <thcipriani@deploy1002> Started scap: Clean up font directory [[gerrit:723652]] [production]
21:04 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:936084|pawikibooks: Install Quiz extension (T340613)]] (duration: 12m 19s) [production]
20:55 <bking@deploy1002> Started deploy [wdqs/wdqs@dff41b7]: 0.3.124 [production]
20:54 <bking@deploy1002> Finished deploy [wdqs/wdqs@dff41b7]: 0.3.124 (duration: 00m 05s) [production]
20:54 <bking@deploy1002> Started deploy [wdqs/wdqs@dff41b7]: 0.3.124 [production]
20:53 <thcipriani@deploy1002> stang and thcipriani: Backport for [[gerrit:936084|pawikibooks: Install Quiz extension (T340613)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
20:51 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:936084|pawikibooks: Install Quiz extension (T340613)]] [production]
20:48 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:935824|Update more logos with available SVGs (T338162)]] (duration: 12m 41s) [production]
20:37 <thcipriani@deploy1002> jdlrobson and thcipriani: Backport for [[gerrit:935824|Update more logos with available SVGs (T338162)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
20:35 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:935824|Update more logos with available SVGs (T338162)]] [production]
20:16 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:936055|Disable purging of old client hint data by default (T340959 T341076)]] (duration: 10m 08s) [production]
20:07 <thcipriani@deploy1002> thcipriani and dreamyjazz: Backport for [[gerrit:936055|Disable purging of old client hint data by default (T340959 T341076)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
20:06 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:936055|Disable purging of old client hint data by default (T340959 T341076)]] [production]
19:24 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:936083|PageView: Fix base URL when using service proxy (T341191)]] (duration: 07m 16s) [production]
19:17 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:936083|PageView: Fix base URL when using service proxy (T341191)]] [production]
19:06 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:03 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:936065|PageView: Route requests through restbase service proxy (T341191)]] (duration: 07m 27s) [production]
18:57 <urbanecm@deploy1002> urbanecm: Backport for [[gerrit:936065|PageView: Route requests through restbase service proxy (T341191)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
18:56 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:936065|PageView: Route requests through restbase service proxy (T341191)]] [production]
17:33 <cstone> tools upgraded from 2ca83336 to 10972e59 [production]
17:24 <sukhe> sudo cumin -b1 -s300 'A:dns-rec' 'systemctl restart ntp.service' [production]
17:17 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:15 <sukhe> homer "mr*" commit "update ntp_servers (add dns1004, remove dns1001)" [production]
17:07 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
17:06 <cstone> SmashPig upgraded from db23b998 to 95181a1b [production]
17:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns1001.wikimedia.org [production]
17:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
17:02 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
17:00 <sukhe@cumin2002> START - Cookbook sre.dns.netbox [production]
16:58 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
16:58 <kharlan@deploy1002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
16:54 <sukhe@cumin2002> START - Cookbook sre.hosts.decommission for hosts dns1001.wikimedia.org [production]
16:49 <sukhe> sudo cumin A:netbox 'run-puppet-agent': removing dns1001 before decomm cookbook [production]
16:44 <sukhe> homer "cr*-eqiad*" commit "decommission DNS host dns1001 (replaced by dns1004)" [production]
16:31 <sukhe> ns0: set routing-options static route 208.80.154.238/32 next-hop [ 208.80.154.6 208.80.155.108 208.80.154.134 ] [production]
16:30 <jiji@deploy1002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
16:30 <jiji@deploy1002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
16:16 <sukhe> homer "cr*-eqiad*" commit "Gerrit: 933917 add new DNS host dns1004" [production]