1701-1750 of 10000 results (77ms)
2023-02-08 ยง
18:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43858 and previous config saved to /var/cache/conftool/dbconfig/20230208-180216-marostegui.json [production]
18:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
18:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
18:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43857 and previous config saved to /var/cache/conftool/dbconfig/20230208-180153-marostegui.json [production]
17:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43856 and previous config saved to /var/cache/conftool/dbconfig/20230208-175654-marostegui.json [production]
17:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P43855 and previous config saved to /var/cache/conftool/dbconfig/20230208-174647-marostegui.json [production]
17:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43854 and previous config saved to /var/cache/conftool/dbconfig/20230208-174148-marostegui.json [production]
17:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43850 and previous config saved to /var/cache/conftool/dbconfig/20230208-172021-marostegui.json [production]
17:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
17:20 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
17:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43849 and previous config saved to /var/cache/conftool/dbconfig/20230208-171634-marostegui.json [production]
17:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
17:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
17:14 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation [production]
17:13 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation [production]
17:11 <oblivian@cumin2002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in codfw: maintenance [production]
17:11 <oblivian@cumin2002> START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance [production]
17:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43848 and previous config saved to /var/cache/conftool/dbconfig/20230208-171028-marostegui.json [production]
17:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance [production]
17:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43847 and previous config saved to /var/cache/conftool/dbconfig/20230208-171006-marostegui.json [production]
17:09 <jynus> disable bacula job backup1002.eqiad.wmnet-Weekly-Thu-EsRwCodfw-mysql-srv-backups-dumps-latest [production]
17:08 <oblivian@cumin2002> END (FAIL) - Cookbook sre.discovery.datacenter (exit_code=93) status all services in codfw: maintenance [production]
17:08 <oblivian@cumin2002> START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance [production]
16:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43844 and previous config saved to /var/cache/conftool/dbconfig/20230208-165500-marostegui.json [production]
16:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
16:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
16:45 <ladsgroup@cumin1001> END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
16:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
16:44 <sukhe> [done] rolling restart of haproxy and trafficserver in A:cp [production]
16:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
16:43 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
16:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43842 and previous config saved to /var/cache/conftool/dbconfig/20230208-163954-marostegui.json [production]
16:27 <sukhe> rolling restart of haproxy and trafficserver in A:cp [production]
16:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43841 and previous config saved to /var/cache/conftool/dbconfig/20230208-162447-marostegui.json [production]
16:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43840 and previous config saved to /var/cache/conftool/dbconfig/20230208-161828-marostegui.json [production]
16:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43839 and previous config saved to /var/cache/conftool/dbconfig/20230208-161807-marostegui.json [production]
16:14 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on rpki1001.eqiad.wmnet with reason: Restarting to increase VM RAM allocation [production]
16:14 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 0:20:00 on rpki1001.eqiad.wmnet with reason: Restarting to increase VM RAM allocation [production]
16:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P43838 and previous config saved to /var/cache/conftool/dbconfig/20230208-160301-marostegui.json [production]
15:50 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2420.codfw.wmnet with OS buster [production]
15:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P43837 and previous config saved to /var/cache/conftool/dbconfig/20230208-154754-marostegui.json [production]
15:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T328817)', diff saved to https://phabricator.wikimedia.org/P43836 and previous config saved to /var/cache/conftool/dbconfig/20230208-154420-marostegui.json [production]
15:38 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new test VM in drmrs - jmm@cumin2002" [production]
15:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43835 and previous config saved to /var/cache/conftool/dbconfig/20230208-153248-marostegui.json [production]
15:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2126 (T328817)', diff saved to https://phabricator.wikimedia.org/P43834 and previous config saved to /var/cache/conftool/dbconfig/20230208-153022-marostegui.json [production]
15:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]
15:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]