1-50 of 10000 results (10ms)
2026-05-14 §
09:33 <root@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS bullseye [production]
09:30 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS bullseye [production]
09:26 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS bullseye [production]
09:23 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage [production]
09:20 <Emperor> rebalance codfw swift rings T354872 [production]
09:18 <root@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage [production]
09:14 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage [production]
09:10 <jiji@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage [production]
09:06 <root@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage [production]
09:06 <jiji@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage [production]
09:06 <jiji@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage [production]
09:06 <jiji@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage [production]
08:55 <jiji@deploy1003> helmfile [codfw] DONE helmfile.d/services/ratelimit: apply [production]
08:55 <jiji@deploy1003> helmfile [codfw] START helmfile.d/services/ratelimit: apply [production]
08:54 <jiji@cumin1003> START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS bullseye [production]
08:54 <jiji@cumin1003> START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS bullseye [production]
08:54 <jiji@cumin1003> START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS bullseye [production]
08:54 <root@cumin1003> START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS bullseye [production]
08:39 <marostegui@cumin1003> dbctl commit (dc=all): 'Remove db2149 T424341', diff saved to https://phabricator.wikimedia.org/P92520 and previous config saved to /var/cache/conftool/dbconfig/20260514-083916-marostegui.json [production]
08:08 <aklapper@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.2 refs T423911 [production]
07:01 <kart_> Update cxserver to 2026-04-23-114216-production (T423002) [production]
07:00 <kartik@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
07:00 <kartik@deploy1003> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
06:50 <komla@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 176 gigabytes (T425905) [language]
06:50 <komla@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase by 176 gigabytes (T425905) [language]
06:41 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc[2013,2023].codfw.wmnet,pc1013.eqiad.wmnet with reason: Maintenance on pc3 [production]
06:40 <kartik@deploy1003> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
06:40 <kartik@deploy1003> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
06:39 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2013: Replacing HW T418973 [production]
06:39 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
06:39 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
06:39 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool pc2013: Replacing HW T418973 [production]
06:39 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: after reimage to trixie [production]
05:54 <marostegui@cumin1003> START - Cookbook sre.mysql.pool pool db1158: after reimage to trixie [production]
05:51 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1158.eqiad.wmnet with OS trixie [production]
05:29 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage [production]
05:25 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage [production]
05:12 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1158.eqiad.wmnet with OS trixie [production]
05:06 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1158: Reimage to Trixie [production]
05:05 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db1158: Reimage to Trixie [production]
05:05 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1158.eqiad.wmnet with reason: Reimage to Trixie [production]
05:04 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: Sanitarium s7 master: reimage to Debian Trixie [production]
05:04 <marostegui@cumin1003> DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 5:00:00 on 13 hosts with reason: Sanitarium s2 master: reimage to Debian Trixie [production]
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 49s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:07 <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 [production]
2026-05-13 §
21:12 <Amir1> remapping thumbsize of 0 to 2 in all group0 wikis (T376152) [production]
21:06 <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs: Restart for upgrade to JVM 11.0.31 - eevans@cumin1003 [production]
20:55 <jdlrobson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1287022|wgThumbLimits: Remove the exception for itwikiquote (T376152)]] (duration: 07m 48s) [production]
20:51 <jdlrobson@deploy1003> ladsgroup, jdlrobson: Continuing with deployment [production]