2251-2300 of 10000 results (80ms)
2023-02-09 ยง
11:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T329203)', diff saved to https://phabricator.wikimedia.org/P43981 and previous config saved to /var/cache/conftool/dbconfig/20230209-112338-marostegui.json [production]
11:20 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:20 <jiji@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
11:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P43980 and previous config saved to /var/cache/conftool/dbconfig/20230209-111220-marostegui.json [production]
11:10 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage [production]
11:08 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2052.codfw.wmnet with OS bullseye [production]
11:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P43979 and previous config saved to /var/cache/conftool/dbconfig/20230209-110832-marostegui.json [production]
11:07 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage [production]
11:02 <effie> powercycle mc-gp1001 [production]
10:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on puppetdb2003.codfw.wmnet with reason: master is being reimaged [production]
10:59 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on puppetdb2003.codfw.wmnet with reason: master is being reimaged [production]
10:58 <joal@deploy1002> Finished deploy [airflow-dags/analytics@dff3f3b]: Fix analytics webrequest_actor_metrics_rollup sensor (duration: 00m 13s) [production]
10:58 <joal@deploy1002> Started deploy [airflow-dags/analytics@dff3f3b]: Fix analytics webrequest_actor_metrics_rollup sensor [production]
10:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P43978 and previous config saved to /var/cache/conftool/dbconfig/20230209-105714-marostegui.json [production]
10:55 <eoghan@cumin1001> START - Cookbook sre.hosts.reimage for host gitlab-runner1003.eqiad.wmnet with OS bullseye [production]
10:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P43977 and previous config saved to /var/cache/conftool/dbconfig/20230209-105325-marostegui.json [production]
10:52 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2052.codfw.wmnet with reason: host reimage [production]
10:50 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc2052.codfw.wmnet with reason: host reimage [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43976 and previous config saved to /var/cache/conftool/dbconfig/20230209-104218-root.json [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43975 and previous config saved to /var/cache/conftool/dbconfig/20230209-104214-root.json [production]
10:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2157 (T328817)', diff saved to https://phabricator.wikimedia.org/P43974 and previous config saved to /var/cache/conftool/dbconfig/20230209-104208-marostegui.json [production]
10:38 <moritzm> installing containerd security updates [production]
10:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T329203)', diff saved to https://phabricator.wikimedia.org/P43973 and previous config saved to /var/cache/conftool/dbconfig/20230209-103819-marostegui.json [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2157 (T328817)', diff saved to https://phabricator.wikimedia.org/P43972 and previous config saved to /var/cache/conftool/dbconfig/20230209-103733-marostegui.json [production]
10:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:37 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43971 and previous config saved to /var/cache/conftool/dbconfig/20230209-103712-marostegui.json [production]
10:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2114 (T329203)', diff saved to https://phabricator.wikimedia.org/P43970 and previous config saved to /var/cache/conftool/dbconfig/20230209-103604-marostegui.json [production]
10:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
10:35 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
10:34 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc-gp1001.eqiad.wmnet with OS bullseye [production]
10:34 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc2052.codfw.wmnet with OS bullseye [production]
10:31 <joal@deploy1002> Finished deploy [airflow-dags/analytics@2ab6564]: Analytics deploy for 3 druid jobs and webrequest_actor jobs (duration: 00m 17s) [production]
10:31 <joal@deploy1002> Started deploy [airflow-dags/analytics@2ab6564]: Analytics deploy for 3 druid jobs and webrequest_actor jobs [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43968 and previous config saved to /var/cache/conftool/dbconfig/20230209-102713-root.json [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43967 and previous config saved to /var/cache/conftool/dbconfig/20230209-102709-root.json [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P43966 and previous config saved to /var/cache/conftool/dbconfig/20230209-102206-marostegui.json [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43965 and previous config saved to /var/cache/conftool/dbconfig/20230209-101209-root.json [production]
10:12 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43964 and previous config saved to /var/cache/conftool/dbconfig/20230209-101204-root.json [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2137:3315', diff saved to https://phabricator.wikimedia.org/P43963 and previous config saved to /var/cache/conftool/dbconfig/20230209-100700-marostegui.json [production]
10:01 <kostajh> UTC morning deploys really done [production]
09:59 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:887848|ComputedUserImpactLookup: Reduce logspam for page view rate limiting (T328945)]] (duration: 09m 06s) [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43962 and previous config saved to /var/cache/conftool/dbconfig/20230209-095704-root.json [production]
09:57 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43961 and previous config saved to /var/cache/conftool/dbconfig/20230209-095659-root.json [production]
09:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2137:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43960 and previous config saved to /var/cache/conftool/dbconfig/20230209-095153-marostegui.json [production]
09:51 <kharlan@deploy1002> kharlan: Backport for [[gerrit:887848|ComputedUserImpactLookup: Reduce logspam for page view rate limiting (T328945)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
09:49 <kharlan@deploy1002> Started scap: Backport for [[gerrit:887848|ComputedUserImpactLookup: Reduce logspam for page view rate limiting (T328945)]] [production]
09:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2137:3315 (T328817)', diff saved to https://phabricator.wikimedia.org/P43959 and previous config saved to /var/cache/conftool/dbconfig/20230209-094816-marostegui.json [production]
09:48 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]
09:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]