51-100 of 10000 results (40ms)
2022-09-15 ยง
14:01 <sukhe> retarting bird.service on A:dns-auth for zlib update [production]
14:00 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 6b9784a0708cf1e7762034ccfba7e5604b2f6dc2: Enable the Vue version of the mentee overview in pilot wikis (T300532) (duration: 03m 45s) [production]
13:58 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@b9be20d]: Regular analytics weekly train [airflow-dags@b9be20d] (duration: 00m 09s) [production]
13:58 <aqu@deploy1002> Started deploy [airflow-dags/analytics@b9be20d]: Regular analytics weekly train [airflow-dags@b9be20d] [production]
13:57 <sukhe> retarting haproxy.service on A:dns-auth for zlib update [production]
13:57 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@b9be20d]: Regular analytics weekly train TEST [airflow-dags@b9be20d] (duration: 00m 10s) [production]
13:56 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@b9be20d]: Regular analytics weekly train TEST [airflow-dags@b9be20d] [production]
13:50 <jayme> updated rsyslog to 8.2208.0-1~bpo11+1 on all kubernetes masters and nodes - T289766 [production]
13:47 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:47 <aqu@deploy1002> Finished deploy [analytics/refinery@278c383] (hadoop-test): Regular analytics weekly train TEST (second try after freeing up some disk space) [analytics/refinery@278c383] (duration: 06m 01s) [production]
13:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:41 <aqu@deploy1002> Started deploy [analytics/refinery@278c383] (hadoop-test): Regular analytics weekly train TEST (second try after freeing up some disk space) [analytics/refinery@278c383] [production]
13:38 <sukhe> restarting bird.service on A:dns-rec for zlib update [production]
13:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:33 <sukhe> restarting pdns-recursor on A:dns-rec for zlib update [production]
13:33 <urbanecm@deploy1002> Synchronized php-1.39.0-wmf.28/extensions/GrowthExperiments/: f592e85858d17a2de99cde93627054ee4972c2bd: Mentee overview: avoid requiring the non-vue mentee overview script when loading the Vue one (T300532) (duration: 04m 05s) [production]
12:50 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
12:50 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
12:46 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1001.eqiad.wmnet with reason: temporarily disabled due to sessionstore issues [production]
12:46 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on sessionstore1001.eqiad.wmnet with reason: temporarily disabled due to sessionstore issues [production]
12:25 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1001.eqiad.wmnet with OS buster [production]
12:17 <jayme> fleet wide update of prometheus-rsyslog-exporter to 0.0.0+git20201008-4 - T289766 [production]
12:10 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1001.eqiad.wmnet with reason: host reimage [production]
12:06 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1001.eqiad.wmnet with reason: host reimage [production]
12:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 100%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34787 and previous config saved to /var/cache/conftool/dbconfig/20220915-120013-root.json [production]
11:51 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1001.eqiad.wmnet with OS buster [production]
11:50 <hnowlan@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1001.eqiad.wmnet with OS buster [production]
11:45 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1001.eqiad.wmnet with OS buster [production]
11:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 75%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34786 and previous config saved to /var/cache/conftool/dbconfig/20220915-114508-root.json [production]
11:44 <hnowlan@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1001.eqiad.wmnet with OS buster [production]
11:43 <moritzm> restart exim on lists1001 to pick up zlib security updates [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 50%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34785 and previous config saved to /var/cache/conftool/dbconfig/20220915-113003-root.json [production]
11:22 <jayme> importing prometheus-rsyslog-exporter 0.0.0+git20201008-4 to stretch-wikimedia, buster-wikimedia, bullseye-wikimedia - T289766 [production]
11:22 <hnowlan@cumin1001> START - Cookbook sre.hosts.reimage for host sessionstore1001.eqiad.wmnet with OS buster [production]
11:17 <jmm@cumin2002> END (PASS) - Cookbook sre.wdqs.restart-nginx (exit_code=0) rolling restart_daemons on A:wcqs-public [production]
11:15 <jmm@cumin2002> START - Cookbook sre.wdqs.restart-nginx rolling restart_daemons on A:wcqs-public [production]
11:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 25%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34784 and previous config saved to /var/cache/conftool/dbconfig/20220915-111458-root.json [production]
11:12 <hnowlan> sessionstore1001: c-foreach-nt drain [production]
11:10 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1001.eqiad.wmnet with reason: Testing reimage [production]
11:10 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on sessionstore1001.eqiad.wmnet with reason: Testing reimage [production]
11:04 <marostegui@cumin1001> dbctl commit (dc=all): 'pool db2129 into s6 API', diff saved to https://phabricator.wikimedia.org/P34783 and previous config saved to /var/cache/conftool/dbconfig/20220915-110453-root.json [production]
10:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 10%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34782 and previous config saved to /var/cache/conftool/dbconfig/20220915-105953-root.json [production]
10:44 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 5%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34781 and previous config saved to /var/cache/conftool/dbconfig/20220915-104448-root.json [production]
10:36 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
10:29 <marostegui@cumin1001> dbctl commit (dc=all): 'db2131 (re)pooling @ 3%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34780 and previous config saved to /var/cache/conftool/dbconfig/20220915-102943-root.json [production]