4251-4300 of 10000 results (28ms)
2024-11-15 §
11:37 <mfossati@deploy2002> Started deploy [airflow-dags/platform_eng@2c533d6]: hotfix image suggestions weekly snapshots [production]
11:27 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
11:24 <cgoubert@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1305-1312].eqiad.wmnet [production]
11:24 <cgoubert@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1305-1312].eqiad.wmnet [production]
11:22 <claime> homer 'lsw1-f5-eqiad*' commit 'T377022' [production]
11:22 <claime> homer 'lsw1-f6-eqiad*' commit 'T377022' [production]
11:22 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host thanos-be1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
11:21 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
11:21 <claime> homer 'lsw1-f7-eqiad*' commit 'T377022' [production]
11:21 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host thanos-be1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
11:20 <claime> homer 'lsw1-e7-eqiad*' commit 'T377022' [production]
11:20 <claime> homer 'lsw1-e6-eqiad*' commit 'T377022' [production]
11:19 <claime> homer 'lsw1-e5-eqiad*' commit 'T377022' [production]
11:15 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:14 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:12 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:12 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:06 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:06 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
11:05 <claime> homer 'cr*eqiad*' commit 'T377022' [production]
10:56 <wmbot~taavi@tools-bastion-12> $ toolforge envvars create RUST_LOG debug # attempting to debug where the bot is [tools.ircservserv]
10:36 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
10:36 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:36 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:34 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on pc1013.eqiad.wmnet with reason: T373037, host is not pooled [production]
09:34 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on pc1013.eqiad.wmnet with reason: T373037, host is not pooled [production]
09:31 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:28 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:28 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:28 <elukey@cumin2002> END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:27 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:23 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:23 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:22 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:21 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host thanos-be2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:15 <aokoth@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Update [production]
08:48 <moritzm> installing Linux 6.1.115 kernel updates from Bookworm point release [production]
04:54 <rzl@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 12:00:00 on db1246.eqiad.wmnet with reason: depooled [production]
04:54 <rzl@cumin2002> START - Cookbook sre.hosts.downtime for 3 days, 12:00:00 on db1246.eqiad.wmnet with reason: depooled [production]
04:51 <rzl@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 12:00:00 on db1246.eqiad.wmnet with reason: depooled [production]
04:50 <rzl@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 12:00:00 on db1246.eqiad.wmnet with reason: depooled [production]
04:47 <rzl@cumin2002> dbctl commit (dc=all): 'db1246 depooled', diff saved to https://phabricator.wikimedia.org/P71052 and previous config saved to /var/cache/conftool/dbconfig/20241115-044705-rzl.json [production]
03:44 <ejegg> fundraising python tools upgraded from c6e2dbcc to b230f718 [production]
2024-11-14 §
23:17 <eileen> civicrm upgraded from 2a53f697 to d49a064d [production]
22:59 <eileen> civicrm upgraded from 2ab8334a to 2a53f697 [production]
22:37 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp4043.ulsfo.wmnet with reason: ATS upgrade 9.2.6 [production]
22:37 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp4043.ulsfo.wmnet with reason: ATS upgrade 9.2.6 [production]
22:30 <ryankemper> T376150 Depooled `wdqs20[18-20]` in preparation of merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1088185 [production]
21:49 <aqu@deploy2002> Finished deploy [airflow-dags/analytics@7a66849]: Stage Refine: fix Airflow skip (duration: 00m 59s) [production]
21:48 <aqu@deploy2002> Started deploy [airflow-dags/analytics@7a66849]: Stage Refine: fix Airflow skip [production]