3551-3600 of 10000 results (100ms)
2024-04-27 §
07:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P61271 and previous config saved to /var/cache/conftool/dbconfig/20240427-073703-ladsgroup.json [production]
07:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P61270 and previous config saved to /var/cache/conftool/dbconfig/20240427-072742-ladsgroup.json [production]
07:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P61269 and previous config saved to /var/cache/conftool/dbconfig/20240427-072155-ladsgroup.json [production]
07:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P61268 and previous config saved to /var/cache/conftool/dbconfig/20240427-071235-ladsgroup.json [production]
07:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T352010)', diff saved to https://phabricator.wikimedia.org/P61267 and previous config saved to /var/cache/conftool/dbconfig/20240427-070648-ladsgroup.json [production]
06:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P61266 and previous config saved to /var/cache/conftool/dbconfig/20240427-065728-ladsgroup.json [production]
00:51 <urandom> rebooting puppetserver1001.eqiad.wmnet via drac [production]
2024-04-26 §
23:13 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P61265 and previous config saved to /var/cache/conftool/dbconfig/20240426-231316-ladsgroup.json [production]
23:13 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance [production]
23:13 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance [production]
23:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176 (T352010)', diff saved to https://phabricator.wikimedia.org/P61264 and previous config saved to /var/cache/conftool/dbconfig/20240426-231252-ladsgroup.json [production]
22:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P61263 and previous config saved to /var/cache/conftool/dbconfig/20240426-225744-ladsgroup.json [production]
22:42 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P61262 and previous config saved to /var/cache/conftool/dbconfig/20240426-224235-ladsgroup.json [production]
22:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2176 (T352010)', diff saved to https://phabricator.wikimedia.org/P61261 and previous config saved to /var/cache/conftool/dbconfig/20240426-222728-ladsgroup.json [production]
22:24 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lists2001.wikimedia.org with OS bookworm [production]
22:07 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lists2001.wikimedia.org with reason: host reimage [production]
22:04 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lists2001.wikimedia.org with reason: host reimage [production]
21:43 <dzahn@cumin2002> START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bookworm [production]
21:38 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lists2001.wikimedia.org with OS bullseye [production]
21:38 <dzahn@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin2002" [production]
21:37 <dzahn@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin2002" [production]
21:25 <amastilovic@deploy1002> Finished deploy [airflow-dags/analytics@33b39d9]: (no justification provided) (duration: 00m 28s) [production]
21:24 <amastilovic@deploy1002> Started deploy [airflow-dags/analytics@33b39d9]: (no justification provided) [production]
21:21 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lists2001.wikimedia.org with reason: host reimage [production]
21:18 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lists2001.wikimedia.org with reason: host reimage [production]
21:01 <dzahn@cumin2002> START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bullseye [production]
19:11 <mutante> LDAP - added linafaridwmde to groups wmde and nda (T362959) [production]
19:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2164 (T352010)', diff saved to https://phabricator.wikimedia.org/P61260 and previous config saved to /var/cache/conftool/dbconfig/20240426-190909-ladsgroup.json [production]
19:10 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
19:09 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
19:09 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
19:09 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
19:08 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61259 and previous config saved to /var/cache/conftool/dbconfig/20240426-190842-ladsgroup.json [production]
18:53 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P61258 and previous config saved to /var/cache/conftool/dbconfig/20240426-185335-ladsgroup.json [production]
18:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P61257 and previous config saved to /var/cache/conftool/dbconfig/20240426-183827-ladsgroup.json [production]
18:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61256 and previous config saved to /var/cache/conftool/dbconfig/20240426-182320-ladsgroup.json [production]
17:57 <dancy@deploy1002> Finished scap: Testing T325530 (duration: 09m 14s) [production]
17:48 <dancy@deploy1002> Started scap: Testing T325530 [production]
17:47 <dancy@deploy1002> Installation of scap version "4.80.0" completed for 325 hosts [production]
17:47 <dancy@deploy1002> Installing scap version "4.80.0" for 325 hosts [production]
17:27 <bking@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet [production]
17:14 <eoghan@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lists2001.wikimedia.org with OS bookworm [production]
17:02 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
16:57 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1107\.eqiad\.wmnet [production]
16:36 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1107\.eqiad\.wmnet [production]
16:35 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1106\.eqiad\.wmnet [production]
16:33 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1106\.eqiad\.wmnet [production]
16:32 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1105\.eqiad\.wmnet [production]
16:30 <denisse> Delete the unused Prometheus PoP TLS certificates in the private repository as part of the cergen to CFSSL migration - T360414 [production]
16:22 <eoghan@cumin1002> START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bookworm [production]