1-50 of 10000 results (86ms)
2025-05-20 §
06:52 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76332 and previous config saved to /var/cache/conftool/dbconfig/20250520-065249-root.json [production]
06:37 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76331 and previous config saved to /var/cache/conftool/dbconfig/20250520-063743-root.json [production]
06:22 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76330 and previous config saved to /var/cache/conftool/dbconfig/20250520-062237-root.json [production]
06:07 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P76329 and previous config saved to /var/cache/conftool/dbconfig/20250520-060731-root.json [production]
06:03 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P76328 and previous config saved to /var/cache/conftool/dbconfig/20250520-060313-root.json [production]
05:52 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76327 and previous config saved to /var/cache/conftool/dbconfig/20250520-055225-root.json [production]
05:48 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P76326 and previous config saved to /var/cache/conftool/dbconfig/20250520-054807-root.json [production]
05:37 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P76325 and previous config saved to /var/cache/conftool/dbconfig/20250520-053720-root.json [production]
05:33 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P76324 and previous config saved to /var/cache/conftool/dbconfig/20250520-053302-root.json [production]
05:22 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76323 and previous config saved to /var/cache/conftool/dbconfig/20250520-052215-root.json [production]
05:17 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76322 and previous config saved to /var/cache/conftool/dbconfig/20250520-051756-root.json [production]
05:07 <marostegui@cumin1002> dbctl commit (dc=all): 'db2236 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76321 and previous config saved to /var/cache/conftool/dbconfig/20250520-050710-root.json [production]
05:03 <marostegui> Install 10.11.13 on db2236 T394653 [production]
05:03 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2236.codfw.wmnet with reason: Maintenance [production]
05:02 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76320 and previous config saved to /var/cache/conftool/dbconfig/20250520-050250-root.json [production]
05:00 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2236 T394653', diff saved to https://phabricator.wikimedia.org/P76319 and previous config saved to /var/cache/conftool/dbconfig/20250520-050017-marostegui.json [production]
04:54 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Maintenance [production]
04:54 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1018.eqiad.wmnet with reason: Maintenance [production]
04:54 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1019.eqiad.wmnet with reason: Maintenance [production]
04:54 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1015.eqiad.wmnet with reason: Maintenance [production]
04:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1014.eqiad.wmnet with reason: Maintenance [production]
04:51 <marostegui> Stop mariadb on db1155, wiki replicas will show lag on: s2, s4, s6 and s7 T394624 [production]
04:50 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1155.eqiad.wmnet with reason: Maintenance [production]
04:47 <marostegui@cumin1002> dbctl commit (dc=all): 'db1183 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76318 and previous config saved to /var/cache/conftool/dbconfig/20250520-044744-root.json [production]
04:01 <mwpresync@deploy1003> Pruned MediaWiki: 1.44.0-wmf.27 (duration: 01m 33s) [production]
03:53 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.45.0-wmf.2 refs T392172 (duration: 50m 50s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.45.0-wmf.2 refs T392172 [production]
00:54 <rzl@deploy1003> Stopping before sync operations [production]
00:54 <rzl@deploy1003> Started scap sync-world: 1147901 [production]
2025-05-19 §
23:01 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1081.eqiad.wmnet with OS bullseye [production]
22:45 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host thanos-be2007.codfw.wmnet with OS bullseye [production]
22:30 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1072.eqiad.wmnet with OS bookworm [production]
22:29 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage [production]
22:28 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1080.eqiad.wmnet with OS bullseye [production]
22:26 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on relforge[1003-1004,1008-1010].eqiad.wmnet with reason: decom in progress [production]
22:25 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1081.eqiad.wmnet with reason: host reimage [production]
22:11 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1081 [production]
22:11 <bking@cumin2002> START - Cookbook sre.hosts.move-vlan for host cirrussearch1081 [production]
22:11 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host cirrussearch1081.eqiad.wmnet with OS bullseye [production]
22:08 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts relforge[1003-1004].eqiad.wmnet [production]
22:04 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Remove db1255 and db2241 from s8 (T351820)', diff saved to https://phabricator.wikimedia.org/P76317 and previous config saved to /var/cache/conftool/dbconfig/20250519-220432-ladsgroup.json [production]
22:03 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage [production]
22:02 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqiad - <bound method SREBatchRunnerBase._reason of <cookbooks.sre.cdn.roll-upgrade-varnish.RollUpgradeVarnishRunner object at 0x7f1ff9224b80>> [production]
22:02 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Remove db1209 and db2195 from x3 (T351820)', diff saved to https://phabricator.wikimedia.org/P76316 and previous config saved to /var/cache/conftool/dbconfig/20250519-220201-ladsgroup.json [production]
22:01 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts relforge[1003-1004].eqiad.wmnet [production]
22:01 <swfrench@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
22:00 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage [production]
22:00 <swfrench@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
21:58 <brett@cumin2002> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - <bound method SREBatchRunnerBase._reason of <cookbooks.sre.cdn.roll-upgrade-varnish.RollUpgradeVarnishRunner object at 0x7f9083afdf10>> [production]
21:58 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage [production]