951-1000 of 10000 results (99ms)
2024-09-12 ยง
14:18 <akosiaris@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet [production]
14:17 <sukhe> sudo cumin "A:dnsbox" "rm /etc/ntp.conf": cleaning up ntpd configuration file to avoid confusion with ntpsec.conf [production]
14:10 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2110.codfw.wmnet with OS bullseye [production]
14:09 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2113 [production]
14:09 <akosiaris@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2113 [production]
14:09 <akosiaris@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2113 [production]
14:09 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2113.codfw.wmnet 63.0.192.10.in-addr.arpa 3.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
14:09 <akosiaris@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker2113.codfw.wmnet 63.0.192.10.in-addr.arpa 3.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
14:09 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:09 <akosiaris@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2113 - akosiaris@cumin1002" [production]
14:09 <akosiaris@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2113 - akosiaris@cumin1002" [production]
14:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P69058 and previous config saved to /var/cache/conftool/dbconfig/20240912-140758-arnaudb.json [production]
14:04 <denisse> Make alert2002 the active host - T372418 [production]
14:04 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2112.codfw.wmnet with reason: host reimage [production]
14:02 <denisse> Disable meta-monitoring for the alert hosts - T372418 [production]
14:02 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2109.codfw.wmnet with OS bullseye [production]
14:01 <denisse> Enable the alert[12]002 hosts as alertmanagers - T372418 [production]
14:01 <denisse> Enable the alert[12]002 hosts as alertmanagers [production]
14:00 <akosiaris@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2112.codfw.wmnet with reason: host reimage [production]
13:56 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2108.codfw.wmnet with OS bullseye [production]
13:56 <cscott> mwscript cleanupTitles enwiki 2>&1 | tee ~/T363538-enwiki-cleanupTitles [production]
13:56 <akosiaris@cumin1002> END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2107.codfw.wmnet [production]
13:56 <akosiaris@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet [production]
13:56 <akosiaris@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet [production]
13:55 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2111.codfw.wmnet with reason: host reimage [production]
13:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T367781)', diff saved to https://phabricator.wikimedia.org/P69057 and previous config saved to /var/cache/conftool/dbconfig/20240912-135251-arnaudb.json [production]
13:52 <akosiaris@cumin1002> START - Cookbook sre.dns.netbox [production]
13:52 <akosiaris@cumin1002> START - Cookbook sre.hosts.move-vlan for host wikikube-worker2113 [production]
13:52 <akosiaris@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2111.codfw.wmnet with reason: host reimage [production]
13:52 <akosiaris@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker2113.codfw.wmnet with OS bullseye [production]
13:51 <akosiaris@cumin1002> START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet [production]
13:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1209 (T367781)', diff saved to https://phabricator.wikimedia.org/P69056 and previous config saved to /var/cache/conftool/dbconfig/20240912-135142-arnaudb.json [production]
13:51 <akosiaris@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2113.codfw.wmnet on all recursors [production]
13:51 <akosiaris@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker2113.codfw.wmnet on all recursors [production]
13:51 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
13:51 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
13:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P69055 and previous config saved to /var/cache/conftool/dbconfig/20240912-135131-arnaudb.json [production]
13:50 <akosiaris> homer cr*codfw* commit 'T372878' [production]
13:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T370903)', diff saved to https://phabricator.wikimedia.org/P69054 and previous config saved to /var/cache/conftool/dbconfig/20240912-135003-ladsgroup.json [production]
13:50 <cscott> mwscript namespaceDupes enwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-suffix=/T363538 --fix 2>&1 | tee ~/T363538-enwiki-namespaceDupes.take2 [production]
13:49 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2110.codfw.wmnet with reason: host reimage [production]
13:49 <cscott> namespaceDupes crashed on MOS:_OVERLINKING, re-running with --add-suffix [production]
13:48 <akosiaris> homer lsw1-a3-codfw* commit 'T372878' [production]
13:47 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2107.codfw.wmnet with OS bullseye [production]
13:47 <akosiaris@cumin1002> END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2113.codfw.wmnet [production]
13:47 <akosiaris@cumin1002> START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet [production]
13:46 <akosiaris@cumin1002> END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2113.codfw.wmnet [production]
13:46 <akosiaris@cumin1002> START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet [production]
13:45 <akosiaris@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2110.codfw.wmnet with reason: host reimage [production]
13:44 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2112 [production]