851-900 of 10000 results (110ms)
2024-04-17 ยง
08:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60738 and previous config saved to /var/cache/conftool/dbconfig/20240417-082901-root.json [production]
08:26 <aqu@deploy1002> Finished deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa] (duration: 02m 23s) [production]
08:24 <aqu@deploy1002> Started deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa] [production]
08:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60737 and previous config saved to /var/cache/conftool/dbconfig/20240417-081953-marostegui.json [production]
08:16 <aqu@deploy1002> Finished deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa] (duration: 03m 39s) [production]
08:13 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60736 and previous config saved to /var/cache/conftool/dbconfig/20240417-081356-root.json [production]
08:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60735 and previous config saved to /var/cache/conftool/dbconfig/20240417-081326-marostegui.json [production]
08:13 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:13 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:13 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
08:13 <aqu@deploy1002> Started deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa] [production]
08:13 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
08:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60734 and previous config saved to /var/cache/conftool/dbconfig/20240417-081256-marostegui.json [production]
08:10 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet [production]
08:07 <aqu@deploy1002> Finished deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa] (duration: 27m 57s) [production]
08:03 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2150.codfw.wmnet with OS bookworm [production]
08:00 <jayme@cumin1002> START - Cookbook sre.hosts.reboot-single for host kubestage2002.codfw.wmnet [production]
07:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60733 and previous config saved to /var/cache/conftool/dbconfig/20240417-075850-root.json [production]
07:57 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60732 and previous config saved to /var/cache/conftool/dbconfig/20240417-075748-marostegui.json [production]
07:49 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1173.eqiad.wmnet [production]
07:42 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2150.codfw.wmnet with reason: host reimage [production]
07:42 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60731 and previous config saved to /var/cache/conftool/dbconfig/20240417-074241-marostegui.json [production]
07:40 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db1173.eqiad.wmnet [production]
07:39 <aqu@deploy1002> Started deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa] [production]
07:39 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2150.codfw.wmnet with reason: host reimage [production]
07:39 <aqu> analytics/refinery deploy begin (added source jars 0.2.35) [production]
07:38 <jynus> restart db1216 database for mariadb upgrade [production]
07:37 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2214.codfw.wmnet [production]
07:27 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60730 and previous config saved to /var/cache/conftool/dbconfig/20240417-072733-marostegui.json [production]
07:27 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host db2214.codfw.wmnet [production]
07:26 <jynus> restart db1240 database for mariadb upgrade [production]
07:22 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db2150.codfw.wmnet with OS bookworm [production]
07:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60729 and previous config saved to /var/cache/conftool/dbconfig/20240417-072122-marostegui.json [production]
07:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2150', diff saved to https://phabricator.wikimedia.org/P60728 and previous config saved to /var/cache/conftool/dbconfig/20240417-072115-root.json [production]
07:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
07:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
07:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60727 and previous config saved to /var/cache/conftool/dbconfig/20240417-072059-marostegui.json [production]
07:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60726 and previous config saved to /var/cache/conftool/dbconfig/20240417-070552-marostegui.json [production]
07:02 <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60725 and previous config saved to /var/cache/conftool/dbconfig/20240417-070206-root.json [production]
06:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60724 and previous config saved to /var/cache/conftool/dbconfig/20240417-065044-marostegui.json [production]
06:47 <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60723 and previous config saved to /var/cache/conftool/dbconfig/20240417-064700-root.json [production]
06:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60722 and previous config saved to /var/cache/conftool/dbconfig/20240417-063537-marostegui.json [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60721 and previous config saved to /var/cache/conftool/dbconfig/20240417-063155-root.json [production]
06:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60720 and previous config saved to /var/cache/conftool/dbconfig/20240417-062918-marostegui.json [production]
06:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance [production]
06:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance [production]
06:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60719 and previous config saved to /var/cache/conftool/dbconfig/20240417-062856-marostegui.json [production]
06:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60718 and previous config saved to /var/cache/conftool/dbconfig/20240417-061649-root.json [production]
06:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60717 and previous config saved to /var/cache/conftool/dbconfig/20240417-061349-marostegui.json [production]
06:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60716 and previous config saved to /var/cache/conftool/dbconfig/20240417-060143-root.json [production]