1701-1750 of 10000 results (105ms)
2024-08-02 ยง
13:52 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on prometheus2008.codfw.wmnet with reason: host reimage [production]
13:50 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host prometheus2008.codfw.wmnet with OS bookworm [production]
13:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2007.codfw.wmnet with OS bookworm [production]
13:44 <sukhe> running authdns-update for CR: 1059362 T371304 [production]
13:44 <sukhe> running authdns-update for CR: T3713041059362 [production]
13:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on prometheus2007.codfw.wmnet with reason: host reimage [production]
13:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host alert2002.wikimedia.org with OS bookworm [production]
13:35 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on prometheus2007.codfw.wmnet with reason: host reimage [production]
13:33 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host prometheus2007.codfw.wmnet with OS bookworm [production]
13:27 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on alert2002.wikimedia.org with reason: host reimage [production]
13:24 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on alert2002.wikimedia.org with reason: host reimage [production]
13:21 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host alert2002.wikimedia.org with OS bookworm [production]
13:11 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "prometheus - ayounsi@cumin1002" [production]
13:10 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "prometheus - ayounsi@cumin1002" [production]
11:03 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:03 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
10:55 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
10:23 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "alert2002 - ayounsi@cumin1002" [production]
10:18 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "alert2002 - ayounsi@cumin1002" [production]
10:18 <elukey> manually start dump_cloud_ip_ranges.service on puppetmaster1001 as test [production]
10:11 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:11 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:23 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:14 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:09 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1195 (T367856)', diff saved to https://phabricator.wikimedia.org/P67203 and previous config saved to /var/cache/conftool/dbconfig/20240802-090649-marostegui.json [production]
09:06 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db1195.eqiad.wmnet with reason: Maintenance [production]
09:06 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 6:00:00 on db1195.eqiad.wmnet with reason: Maintenance [production]
09:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T367856)', diff saved to https://phabricator.wikimedia.org/P67202 and previous config saved to /var/cache/conftool/dbconfig/20240802-090627-marostegui.json [production]
08:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P67201 and previous config saved to /var/cache/conftool/dbconfig/20240802-085119-marostegui.json [production]
08:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P67200 and previous config saved to /var/cache/conftool/dbconfig/20240802-083612-marostegui.json [production]
08:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T367856)', diff saved to https://phabricator.wikimedia.org/P67199 and previous config saved to /var/cache/conftool/dbconfig/20240802-082105-marostegui.json [production]
08:20 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:20 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:19 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:19 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
07:37 <slyngshede@cumin1002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Sbailey out of all services on: 2241 hosts [production]
07:36 <slyngshede@cumin1002> START - Cookbook sre.idm.logout Logging Sbailey out of all services on: 2241 hosts [production]
02:09 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1260 [production]
02:08 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1260 [production]
02:07 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1267.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:07 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1260.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:07 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1261.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:05 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1266.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:04 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1269.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:03 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host wikikube-worker1260.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:01 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1260.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:59 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1263.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:59 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1268.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:57 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1262.mgmt.eqiad.wmnet with reboot policy FORCED [production]