5651-5700 of 10000 results (96ms)
2023-08-30 §
05:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1165 (re)pooling @ 1%: Repooling after onsite upgrade', diff saved to https://phabricator.wikimedia.org/P51976 and previous config saved to /var/cache/conftool/dbconfig/20230830-054940-root.json [production]
05:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1165 upgrade to mariadb 10.6', diff saved to https://phabricator.wikimedia.org/P51975 and previous config saved to /var/cache/conftool/dbconfig/20230830-054248-root.json [production]
05:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
05:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
05:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1221 (T343718)', diff saved to https://phabricator.wikimedia.org/P51974 and previous config saved to /var/cache/conftool/dbconfig/20230830-051543-ladsgroup.json [production]
05:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P51973 and previous config saved to /var/cache/conftool/dbconfig/20230830-050036-ladsgroup.json [production]
04:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P51972 and previous config saved to /var/cache/conftool/dbconfig/20230830-044530-ladsgroup.json [production]
04:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1221 (T343718)', diff saved to https://phabricator.wikimedia.org/P51971 and previous config saved to /var/cache/conftool/dbconfig/20230830-043024-ladsgroup.json [production]
01:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1221 (T343718)', diff saved to https://phabricator.wikimedia.org/P51970 and previous config saved to /var/cache/conftool/dbconfig/20230830-014730-ladsgroup.json [production]
01:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
01:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
01:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [production]
01:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [production]
01:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T343718)', diff saved to https://phabricator.wikimedia.org/P51969 and previous config saved to /var/cache/conftool/dbconfig/20230830-014702-ladsgroup.json [production]
01:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P51968 and previous config saved to /var/cache/conftool/dbconfig/20230830-013156-ladsgroup.json [production]
01:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P51967 and previous config saved to /var/cache/conftool/dbconfig/20230830-011650-ladsgroup.json [production]
01:09 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kubernetes2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T343718)', diff saved to https://phabricator.wikimedia.org/P51966 and previous config saved to /var/cache/conftool/dbconfig/20230830-010144-ladsgroup.json [production]
01:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:00 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kubernetes2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2037.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2035.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2039.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:41 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2036.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:30 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2039.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:30 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2038.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:30 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2037.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:29 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2036.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:29 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2035.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:29 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:26 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2033.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:25 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2032.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:21 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2031.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T343718)', diff saved to https://phabricator.wikimedia.org/P51965 and previous config saved to /var/cache/conftool/dbconfig/20230830-002108-ladsgroup.json [production]
00:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2030.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:17 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2029.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:15 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2033.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:14 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kubernetes2028.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:12 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2032.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:10 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2031.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:08 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2030.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P51964 and previous config saved to /var/cache/conftool/dbconfig/20230830-000602-ladsgroup.json [production]
00:05 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2029.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:03 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2028.mgmt.codfw.wmnet with reboot policy FORCED [production]
2023-08-29 §
23:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P51963 and previous config saved to /var/cache/conftool/dbconfig/20230829-235055-ladsgroup.json [production]
23:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T343718)', diff saved to https://phabricator.wikimedia.org/P51962 and previous config saved to /var/cache/conftool/dbconfig/20230829-233549-ladsgroup.json [production]
23:09 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:08 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly racked kubernetes2034 hosts in codfw - jhancock@cumin2002" [production]
23:08 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly racked kubernetes2034 hosts in codfw - jhancock@cumin2002" [production]