2651-2700 of 10000 results (88ms)
2023-02-08 ยง
19:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T328817)', diff saved to https://phabricator.wikimedia.org/P43875 and previous config saved to /var/cache/conftool/dbconfig/20230208-191023-marostegui.json [production]
19:08 <milimetric@deploy1002> Finished deploy [analytics/refinery@9101b03] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9101b03] (duration: 01m 20s) [production]
19:06 <milimetric@deploy1002> Started deploy [analytics/refinery@9101b03] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9101b03] [production]
19:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P43874 and previous config saved to /var/cache/conftool/dbconfig/20230208-190608-ladsgroup.json [production]
19:06 <milimetric@deploy1002> Finished deploy [analytics/refinery@9101b03] (thin): Regular analytics weekly train THIN [analytics/refinery@9101b03] (duration: 00m 07s) [production]
19:05 <milimetric@deploy1002> Started deploy [analytics/refinery@9101b03] (thin): Regular analytics weekly train THIN [analytics/refinery@9101b03] [production]
19:05 <milimetric@deploy1002> Finished deploy [analytics/refinery@9101b03]: Regular analytics weekly train [analytics/refinery@9101b03] (duration: 06m 28s) [production]
19:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1129 (T328817)', diff saved to https://phabricator.wikimedia.org/P43873 and previous config saved to /var/cache/conftool/dbconfig/20230208-190410-marostegui.json [production]
19:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
19:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
19:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122 (T328817)', diff saved to https://phabricator.wikimedia.org/P43872 and previous config saved to /var/cache/conftool/dbconfig/20230208-190349-marostegui.json [production]
19:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175 (T328817)', diff saved to https://phabricator.wikimedia.org/P43871 and previous config saved to /var/cache/conftool/dbconfig/20230208-190226-marostegui.json [production]
18:59 <milimetric@deploy1002> Started deploy [analytics/refinery@9101b03]: Regular analytics weekly train [analytics/refinery@9101b03] [production]
18:57 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mw2420.codfw.wmnet with OS buster [production]
18:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2175 (T328817)', diff saved to https://phabricator.wikimedia.org/P43870 and previous config saved to /var/cache/conftool/dbconfig/20230208-185513-marostegui.json [production]
18:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
18:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
18:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43869 and previous config saved to /var/cache/conftool/dbconfig/20230208-185451-marostegui.json [production]
18:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P43868 and previous config saved to /var/cache/conftool/dbconfig/20230208-185102-ladsgroup.json [production]
18:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P43867 and previous config saved to /var/cache/conftool/dbconfig/20230208-184842-marostegui.json [production]
18:42 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host mw2420.codfw.wmnet with OS buster [production]
18:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P43866 and previous config saved to /var/cache/conftool/dbconfig/20230208-183945-marostegui.json [production]
18:39 <mutante> adding az.wikimedia.org to DNS - approved by affcom T306015 [production]
18:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T328255)', diff saved to https://phabricator.wikimedia.org/P43865 and previous config saved to /var/cache/conftool/dbconfig/20230208-183556-ladsgroup.json [production]
18:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P43864 and previous config saved to /var/cache/conftool/dbconfig/20230208-183336-marostegui.json [production]
18:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P43863 and previous config saved to /var/cache/conftool/dbconfig/20230208-182439-marostegui.json [production]
18:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122 (T328817)', diff saved to https://phabricator.wikimedia.org/P43862 and previous config saved to /var/cache/conftool/dbconfig/20230208-181829-marostegui.json [production]
18:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1122 (T328817)', diff saved to https://phabricator.wikimedia.org/P43861 and previous config saved to /var/cache/conftool/dbconfig/20230208-181222-marostegui.json [production]
18:12 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
18:12 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
18:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43860 and previous config saved to /var/cache/conftool/dbconfig/20230208-181200-marostegui.json [production]
18:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43859 and previous config saved to /var/cache/conftool/dbconfig/20230208-180933-marostegui.json [production]
18:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43858 and previous config saved to /var/cache/conftool/dbconfig/20230208-180216-marostegui.json [production]
18:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
18:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
18:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43857 and previous config saved to /var/cache/conftool/dbconfig/20230208-180153-marostegui.json [production]
17:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43856 and previous config saved to /var/cache/conftool/dbconfig/20230208-175654-marostegui.json [production]
17:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P43855 and previous config saved to /var/cache/conftool/dbconfig/20230208-174647-marostegui.json [production]
17:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43854 and previous config saved to /var/cache/conftool/dbconfig/20230208-174148-marostegui.json [production]
17:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43850 and previous config saved to /var/cache/conftool/dbconfig/20230208-172021-marostegui.json [production]
17:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
17:20 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
17:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43849 and previous config saved to /var/cache/conftool/dbconfig/20230208-171634-marostegui.json [production]
17:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
17:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
17:14 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation [production]
17:13 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation [production]
17:11 <oblivian@cumin2002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in codfw: maintenance [production]
17:11 <oblivian@cumin2002> START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance [production]
17:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43848 and previous config saved to /var/cache/conftool/dbconfig/20230208-171028-marostegui.json [production]