1751-1800 of 10000 results (33ms)
2025-06-23 §
06:55 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P78577 and previous config saved to /var/cache/conftool/dbconfig/20250623-065503-marostegui.json [production]
06:48 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2235].codfw.wmnet,db[1164,1217,1228].eqiad.wmnet with reason: m5 master switch T397413 [production]
06:46 <marostegui@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P78576 and previous config saved to /var/cache/conftool/dbconfig/20250623-064628-root.json [production]
06:46 <marostegui@cumin1002> dbctl commit (dc=all): 'es2035 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78575 and previous config saved to /var/cache/conftool/dbconfig/20250623-064606-root.json [production]
06:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78574 and previous config saved to /var/cache/conftool/dbconfig/20250623-064358-root.json [production]
06:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T396130)', diff saved to https://phabricator.wikimedia.org/P78573 and previous config saved to /var/cache/conftool/dbconfig/20250623-063956-marostegui.json [production]
06:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1190 (T396130)', diff saved to https://phabricator.wikimedia.org/P78572 and previous config saved to /var/cache/conftool/dbconfig/20250623-063217-marostegui.json [production]
06:32 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T396130)', diff saved to https://phabricator.wikimedia.org/P78571 and previous config saved to /var/cache/conftool/dbconfig/20250623-063155-marostegui.json [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'es2039 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78570 and previous config saved to /var/cache/conftool/dbconfig/20250623-063123-root.json [production]
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'es2035 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78569 and previous config saved to /var/cache/conftool/dbconfig/20250623-063100-root.json [production]
06:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2039 T397599', diff saved to https://phabricator.wikimedia.org/P78568 and previous config saved to /var/cache/conftool/dbconfig/20250623-063050-marostegui.json [production]
06:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es2038 to es7 primary and set section read-write T397599', diff saved to https://phabricator.wikimedia.org/P78567 and previous config saved to /var/cache/conftool/dbconfig/20250623-062949-marostegui.json [production]
06:29 <marostegui> Starting es7 codfw failover from es2039 to es2038 - T397599 [production]
06:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78566 and previous config saved to /var/cache/conftool/dbconfig/20250623-062852-root.json [production]
06:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Set es2038 with weight 0 T397599', diff saved to https://phabricator.wikimedia.org/P78565 and previous config saved to /var/cache/conftool/dbconfig/20250623-062420-root.json [production]
06:18 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: Primary switchover es7 T397599 [production]
06:17 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: T397597 [production]
06:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P78564 and previous config saved to /var/cache/conftool/dbconfig/20250623-061648-marostegui.json [production]
06:15 <marostegui@cumin1002> dbctl commit (dc=all): 'es2035 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P78563 and previous config saved to /var/cache/conftool/dbconfig/20250623-061554-root.json [production]
06:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2035 T397597', diff saved to https://phabricator.wikimedia.org/P78562 and previous config saved to /var/cache/conftool/dbconfig/20250623-061511-marostegui.json [production]
06:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es2037 to es6 primary and set section read-write T397597', diff saved to https://phabricator.wikimedia.org/P78561 and previous config saved to /var/cache/conftool/dbconfig/20250623-061416-marostegui.json [production]
06:13 <marostegui> Starting es6 codfw failover from es2035 to es2037 - T397597 [production]
06:13 <marostegui@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78560 and previous config saved to /var/cache/conftool/dbconfig/20250623-061346-root.json [production]
06:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Set es2037 with weight 0 T397597', diff saved to https://phabricator.wikimedia.org/P78559 and previous config saved to /var/cache/conftool/dbconfig/20250623-061143-root.json [production]
06:09 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: T397597 [production]
06:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P78558 and previous config saved to /var/cache/conftool/dbconfig/20250623-060140-marostegui.json [production]
05:58 <marostegui@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78557 and previous config saved to /var/cache/conftool/dbconfig/20250623-055840-root.json [production]
05:55 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: T397597 [production]
05:48 <stevemunene@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [production]
05:47 <stevemunene@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [production]
05:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2215 T397419', diff saved to https://phabricator.wikimedia.org/P78556 and previous config saved to /var/cache/conftool/dbconfig/20250623-054725-marostegui.json [production]
05:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T396130)', diff saved to https://phabricator.wikimedia.org/P78555 and previous config saved to /var/cache/conftool/dbconfig/20250623-054633-marostegui.json [production]
05:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2196 to x1 primary T397419', diff saved to https://phabricator.wikimedia.org/P78554 and previous config saved to /var/cache/conftool/dbconfig/20250623-054616-marostegui.json [production]
05:45 <marostegui> Starting x1 codfw failover from db2215 to db2196 - T397419 [production]
05:42 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2196 with weight 0 T397419', diff saved to https://phabricator.wikimedia.org/P78553 and previous config saved to /var/cache/conftool/dbconfig/20250623-054206-root.json [production]
05:41 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x1 T397419 [production]
05:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1160 (T396130)', diff saved to https://phabricator.wikimedia.org/P78552 and previous config saved to /var/cache/conftool/dbconfig/20250623-053857-marostegui.json [production]
05:38 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
05:33 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
2025-06-22 §
21:42 <Krinkle> Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1162179 [releng]
00:09 <andrewbogott> rebooting tools-prometheus-8 [tools]
2025-06-21 §
20:07 <wmbot~anticomposite@tools-bastion-13> stewardbots/StewardBot/manage.sh restart # Disconnected [tools.stewardbots]
16:15 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [admin]
16:10 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [admin]
16:09 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 [tools]
15:58 <andrewbogott> rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of D state [tools]
15:57 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 [tools]
10:09 <wmbot~dcaro@acme> END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [tools]
09:27 <wmbot~dcaro@acme> START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]