251-300 of 10000 results (69ms)
2023-02-08 ยง
13:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
13:13 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
13:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43807 and previous config saved to /var/cache/conftool/dbconfig/20230208-131348-marostegui.json [production]
13:03 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1002.eqiad.wmnet with OS bullseye [production]
13:00 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 13150 [production]
12:59 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 13150 [production]
12:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P43805 and previous config saved to /var/cache/conftool/dbconfig/20230208-125841-marostegui.json [production]
12:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P43804 and previous config saved to /var/cache/conftool/dbconfig/20230208-124335-marostegui.json [production]
12:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1096.eqiad.wmnet [production]
12:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1096.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" [production]
12:36 <marostegui@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1096.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" [production]
12:34 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
12:29 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1096.eqiad.wmnet [production]
12:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43803 and previous config saved to /var/cache/conftool/dbconfig/20230208-122829-marostegui.json [production]
12:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T328817)', diff saved to https://phabricator.wikimedia.org/P43802 and previous config saved to /var/cache/conftool/dbconfig/20230208-122620-marostegui.json [production]
12:26 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
12:26 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
12:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T328817)', diff saved to https://phabricator.wikimedia.org/P43801 and previous config saved to /var/cache/conftool/dbconfig/20230208-122559-marostegui.json [production]
12:21 <jmm@cumin2002> END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas [production]
12:19 <jmm@cumin2002> START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas [production]
12:18 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage [production]
12:15 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage [production]
12:13 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kafka-stretch2002.codfw.wmnet [production]
12:13 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kafka-stretch2002.codfw.wmnet [production]
12:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P43800 and previous config saved to /var/cache/conftool/dbconfig/20230208-121053-marostegui.json [production]
12:03 <eoghan@cumin1001> START - Cookbook sre.hosts.reimage for host gitlab-runner1002.eqiad.wmnet with OS bullseye [production]
11:59 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:59 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:58 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1097.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:57 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1097.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:57 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-worker1096.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:57 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on an-worker1096.eqiad.wmnet with reason: Attempting to move some GPUs [production]
11:56 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: moss-be1001.eqiad.wmnet [production]
11:56 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: moss-be1001.eqiad.wmnet [production]
11:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P43799 and previous config saved to /var/cache/conftool/dbconfig/20230208-115546-marostegui.json [production]
11:53 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: flowspec1001.eqiad.wmnet [production]
11:53 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: flowspec1001.eqiad.wmnet [production]
11:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T328817)', diff saved to https://phabricator.wikimedia.org/P43798 and previous config saved to /var/cache/conftool/dbconfig/20230208-114040-marostegui.json [production]
11:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1165 (T328817)', diff saved to https://phabricator.wikimedia.org/P43797 and previous config saved to /var/cache/conftool/dbconfig/20230208-113832-marostegui.json [production]
11:38 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
11:38 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
11:38 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance [production]
11:37 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance [production]
11:13 <marostegui> Stop mysql on db1096 (s5,s6) T329147 [production]
11:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43796 and previous config saved to /var/cache/conftool/dbconfig/20230208-110507-marostegui.json [production]
10:57 <zabe@deploy1002> Finished scap: Backport for [[gerrit:887748|Remove cul_reason comment table migration code (T233004 T329151)]] (duration: 08m 05s) [production]
10:51 <zabe@deploy1002> zabe: Backport for [[gerrit:887748|Remove cul_reason comment table migration code (T233004 T329151)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]