451-500 of 10000 results (72ms)
2023-02-13 §
07:41 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host ml-staging-etcd2001.codfw.wmnet with OS bullseye [production]
07:39 <elukey@cumin1001> START - Cookbook sre.k8s.upgrade-cluster Upgrade K8s version: Upgrade ml-staging-codfw cluster to 1.23 [production]
07:37 <marostegui> Deploy schema change on db2151 T329260 [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T329203)', diff saved to https://phabricator.wikimedia.org/P44345 and previous config saved to /var/cache/conftool/dbconfig/20230213-072825-marostegui.json [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2152 (T328817)', diff saved to https://phabricator.wikimedia.org/P44344 and previous config saved to /var/cache/conftool/dbconfig/20230213-072752-marostegui.json [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2123 (T329203)', diff saved to https://phabricator.wikimedia.org/P44343 and previous config saved to /var/cache/conftool/dbconfig/20230213-072535-marostegui.json [production]
07:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
07:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T329203)', diff saved to https://phabricator.wikimedia.org/P44342 and previous config saved to /var/cache/conftool/dbconfig/20230213-072514-marostegui.json [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P44341 and previous config saved to /var/cache/conftool/dbconfig/20230213-071007-marostegui.json [production]
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2152 (T328817)', diff saved to https://phabricator.wikimedia.org/P44340 and previous config saved to /var/cache/conftool/dbconfig/20230213-070717-marostegui.json [production]
07:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
07:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance [production]
06:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2132,2160].codfw.wmnet,db[1117,1164,1176].eqiad.wmnet with reason: Primary switchover m1 T329259 [production]
06:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2132,2160].codfw.wmnet,db[1117,1164,1176].eqiad.wmnet with reason: Primary switchover m1 T329259 [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P44339 and previous config saved to /var/cache/conftool/dbconfig/20230213-065501-marostegui.json [production]
06:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
06:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2100.codfw.wmnet with reason: Maintenance [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1099 from dbctl T329181', diff saved to https://phabricator.wikimedia.org/P44338 and previous config saved to /var/cache/conftool/dbconfig/20230213-064051-marostegui.json [production]
06:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T329203)', diff saved to https://phabricator.wikimedia.org/P44337 and previous config saved to /var/cache/conftool/dbconfig/20230213-063955-marostegui.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2111 (T329203)', diff saved to https://phabricator.wikimedia.org/P44336 and previous config saved to /var/cache/conftool/dbconfig/20230213-063449-marostegui.json [production]
06:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
06:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
06:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
06:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
06:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
06:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
06:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
06:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
06:24 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
06:22 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
06:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
06:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
06:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
06:05 <AndyRussG> DjangoBannerStats upgraded from c9926cfc to 5dc35ea2 on fran1001 [production]
2023-02-11 §
01:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197 (T329203)', diff saved to https://phabricator.wikimedia.org/P44335 and previous config saved to /var/cache/conftool/dbconfig/20230211-015530-marostegui.json [production]
01:41 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2451.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P44334 and previous config saved to /var/cache/conftool/dbconfig/20230211-014023-marostegui.json [production]
01:37 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2451.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2449.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:32 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2450.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:28 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2450.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:28 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2449.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:27 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2448.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:27 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2447.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P44333 and previous config saved to /var/cache/conftool/dbconfig/20230211-012517-marostegui.json [production]
01:10 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2448.mgmt.codfw.wmnet with reboot policy FORCED [production]