151-200 of 10000 results (100ms)
2024-08-28 ยง
17:47 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
17:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T371742)', diff saved to https://phabricator.wikimedia.org/P68075 and previous config saved to /var/cache/conftool/dbconfig/20240828-174749-ladsgroup.json [production]
17:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2149 (T370903)', diff saved to https://phabricator.wikimedia.org/P68074 and previous config saved to /var/cache/conftool/dbconfig/20240828-174514-ladsgroup.json [production]
17:45 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage [production]
17:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
17:44 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
17:42 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm [production]
17:42 <akosiaris@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage [production]
17:32 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P68073 and previous config saved to /var/cache/conftool/dbconfig/20240828-173242-ladsgroup.json [production]
17:30 <kcvelaga@deploy1003> Finished deploy [airflow-dags/analytics_product@cb0bc4d]: (no justification provided) (duration: 00m 18s) [production]
17:29 <kcvelaga@deploy1003> Started deploy [airflow-dags/analytics_product@cb0bc4d]: (no justification provided) [production]
17:27 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:27 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
17:26 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
17:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2127 (T370903)', diff saved to https://phabricator.wikimedia.org/P68072 and previous config saved to /var/cache/conftool/dbconfig/20240828-172653-ladsgroup.json [production]
17:24 <akosiaris@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker2045.codfw.wmnet with OS bullseye [production]
17:22 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:22 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:22 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:17 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P68071 and previous config saved to /var/cache/conftool/dbconfig/20240828-171735-ladsgroup.json [production]
17:14 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P68070 and previous config saved to /var/cache/conftool/dbconfig/20240828-171146-ladsgroup.json [production]
17:05 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:03 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:02 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:02 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T371742)', diff saved to https://phabricator.wikimedia.org/P68069 and previous config saved to /var/cache/conftool/dbconfig/20240828-170228-ladsgroup.json [production]
17:02 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:01 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
17:00 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:59 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:59 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P68068 and previous config saved to /var/cache/conftool/dbconfig/20240828-165638-ladsgroup.json [production]
16:51 <topranks> add qos config to management firewalls T339850 [production]
16:44 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm [production]
16:44 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm [production]
16:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2127 (T370903)', diff saved to https://phabricator.wikimedia.org/P68067 and previous config saved to /var/cache/conftool/dbconfig/20240828-164131-ladsgroup.json [production]
16:38 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-lab1001.eqiad.wmnet with OS bookworm [production]
16:36 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2009.codfw.wmnet [production]
16:35 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2009.codfw.wmnet [production]
16:35 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm [production]
16:32 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2009.codfw.wmnet [production]
16:32 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2009.codfw.wmnet [production]
16:30 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2009.codfw.wmnet [production]
16:30 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2009.codfw.wmnet [production]
16:26 <stevemunene@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:26 <stevemunene@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
16:24 <hnowlan@deploy1003> Finished scap sync-world: Backport for [[gerrit:1068025|timedmediahandler: revert using shellbox for commonswiki (T373517)]] (duration: 07m 13s) [production]
16:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2127 (T370903)', diff saved to https://phabricator.wikimedia.org/P68066 and previous config saved to /var/cache/conftool/dbconfig/20240828-162239-ladsgroup.json [production]
16:22 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
16:22 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]