7351-7400 of 10000 results (100ms)
2024-04-25 ยง
22:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P61228 and previous config saved to /var/cache/conftool/dbconfig/20240425-224146-ladsgroup.json [production]
22:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61227 and previous config saved to /var/cache/conftool/dbconfig/20240425-222638-ladsgroup.json [production]
22:23 <brett> Extend prometheus1005 and prometheus1006 logical volume by an extra 60G due to disk filling up [production]
19:33 <ebernhardson> T363516 started manual rebuild of enwiki titlesuggest indices in eqiad [production]
19:12 <dancy@deploy1002> Finished scap: Testing (duration: 08m 47s) [production]
19:03 <dancy@deploy1002> Started scap: Testing [production]
18:23 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.43.0-wmf.2 refs T361396 [production]
18:22 <logmsgbot> nshahquinn-wmf@deploy1002 Finished deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided) (duration: 00m 07s) [production]
18:22 <logmsgbot> nshahquinn-wmf@deploy1002 Started deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided) [production]
18:08 <brennen> train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group2 [production]
17:58 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2162 (T352010)', diff saved to https://phabricator.wikimedia.org/P61223 and previous config saved to /var/cache/conftool/dbconfig/20240425-175802-ladsgroup.json [production]
17:58 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
17:57 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
17:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61222 and previous config saved to /var/cache/conftool/dbconfig/20240425-175739-ladsgroup.json [production]
17:42 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P61221 and previous config saved to /var/cache/conftool/dbconfig/20240425-174233-ladsgroup.json [production]
17:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P61220 and previous config saved to /var/cache/conftool/dbconfig/20240425-172725-ladsgroup.json [production]
17:16 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1001.eqiad.wmnet with OS bullseye [production]
17:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61219 and previous config saved to /var/cache/conftool/dbconfig/20240425-171329-ladsgroup.json [production]
17:13 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
17:13 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
17:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61218 and previous config saved to /var/cache/conftool/dbconfig/20240425-171218-ladsgroup.json [production]
16:34 <mutante> releases1003 - docker and containerd restarted by manually starting wmf_auto_restart services [production]
15:38 <dancy@deploy1002> Finished scap: Testing (duration: 08m 44s) [production]
15:34 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@b17acd0]: (no justification provided) (duration: 00m 27s) [production]
15:33 <mforns@deploy1002> Started deploy [airflow-dags/analytics@b17acd0]: (no justification provided) [production]
15:29 <dancy@deploy1002> Started scap: Testing [production]
15:29 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2155.codfw.wmnet with OS bookworm [production]
15:27 <dancy@deploy1002> sync-world aborted: Testing (duration: 01m 33s) [production]
15:26 <klausman@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Java 11 security updates - klausman@cumin1002 [production]
15:25 <dancy@deploy1002> Started scap: Testing [production]
15:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2173 (T352010)', diff saved to https://phabricator.wikimedia.org/P61216 and previous config saved to /var/cache/conftool/dbconfig/20240425-151120-ladsgroup.json [production]
15:12 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
15:12 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
15:11 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
15:11 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance [production]
15:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170 (T352010)', diff saved to https://phabricator.wikimedia.org/P61215 and previous config saved to /var/cache/conftool/dbconfig/20240425-151041-ladsgroup.json [production]
15:07 <klausman@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Java 11 security updates - klausman@cumin1002 [production]
15:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2155.codfw.wmnet with reason: host reimage [production]
15:03 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2155.codfw.wmnet with reason: host reimage [production]
14:59 <klausman@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin1002 [production]
14:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P61214 and previous config saved to /var/cache/conftool/dbconfig/20240425-145534-ladsgroup.json [production]
14:53 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on db2187.codfw.wmnet with reason: Host has hardware issues [production]
14:53 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 20:00:00 on db2187.codfw.wmnet with reason: Host has hardware issues [production]
14:44 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: Host has hardware issues [production]
14:44 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: Host has hardware issues [production]
14:41 <klausman@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin1002 [production]
14:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P61213 and previous config saved to /var/cache/conftool/dbconfig/20240425-144027-ladsgroup.json [production]
14:29 <moritzm> installing Java 11 security updates [production]
14:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2170 (T352010)', diff saved to https://phabricator.wikimedia.org/P61212 and previous config saved to /var/cache/conftool/dbconfig/20240425-142520-ladsgroup.json [production]
14:21 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bookworm [production]