2251-2300 of 10000 results (131ms)
2024-09-10 §
12:24 <jmm@deploy1003> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
12:23 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2007.codfw.wmnet [production]
12:23 <jayme@cumin1002> START - Cookbook sre.hosts.remove-downtime for kafka-main2007.codfw.wmnet [production]
12:19 <jmm@deploy1003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
12:18 <jmm@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
12:13 <jmm@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
12:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1169 (T371742)', diff saved to https://phabricator.wikimedia.org/P68772 and previous config saved to /var/cache/conftool/dbconfig/20240910-121122-ladsgroup.json [production]
12:03 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68771 and previous config saved to /var/cache/conftool/dbconfig/20240910-120357-ladsgroup.json [production]
11:58 <jmm@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
11:58 <jmm@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
11:48 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P68770 and previous config saved to /var/cache/conftool/dbconfig/20240910-114850-ladsgroup.json [production]
11:33 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P68769 and previous config saved to /var/cache/conftool/dbconfig/20240910-113342-ladsgroup.json [production]
11:18 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68768 and previous config saved to /var/cache/conftool/dbconfig/20240910-111835-ladsgroup.json [production]
11:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1169 (T371742)', diff saved to https://phabricator.wikimedia.org/P68767 and previous config saved to /var/cache/conftool/dbconfig/20240910-110614-ladsgroup.json [production]
11:06 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance [production]
11:05 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance [production]
11:04 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68766 and previous config saved to /var/cache/conftool/dbconfig/20240910-110409-ladsgroup.json [production]
11:04 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
11:04 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
10:51 <vgutierrez> switching purged in codfw and ulsfo to use main-eqiad kafka cluster - T373189 [production]
10:49 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:49 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:44 <moritzm> installing bind9 security updates (client-side tools/libs only) [production]
10:34 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:33 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:14 <hnowlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1071659|Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241)]] (duration: 09m 39s) [production]
10:09 <hnowlan@deploy1003> hnowlan: Continuing with sync [production]
10:08 <hnowlan@deploy1003> hnowlan: Backport for [[gerrit:1071659|Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
10:05 <hnowlan@deploy1003> Started scap sync-world: Backport for [[gerrit:1071659|Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241)]] [production]
09:35 <cgoubert@cumin1002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2431.codfw.wmnet [production]
09:35 <cgoubert@cumin1002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2431.codfw.wmnet [production]
09:11 <arnaudb@cumin1002> dbctl commit (dc=all): 'T374421 → replag not catching up on exec, ^C to debug', diff saved to https://phabricator.wikimedia.org/P68763 and previous config saved to /var/cache/conftool/dbconfig/20240910-091114-arnaudb.json [production]
09:10 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:10 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:03 <jayme@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw [production]
09:02 <hashar> Restarting CI Jenkins [production]
09:02 <jayme@deploy1003> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:02 <jayme@deploy1003> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:02 <jayme@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:01 <jayme@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:01 <jayme@deploy1003> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:01 <jayme@deploy1003> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
09:01 <jayme@deploy1003> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
09:00 <jayme@deploy1003> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
09:00 <jayme@deploy1003> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:00 <jayme@deploy1003> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
09:00 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:59 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:59 <jayme@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:59 <jayme@deploy1003> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]