1651-1700 of 10000 results (26ms)
2025-07-21 §
06:41 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
06:38 <marostegui@cumin1002> dbctl commit (dc=all): 'db1170 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P79451 and previous config saved to /var/cache/conftool/dbconfig/20250721-063859-root.json [production]
06:32 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P79450 and previous config saved to /var/cache/conftool/dbconfig/20250721-063232-root.json [production]
06:31 <elukey@cumin1003> START - Cookbook sre.hosts.provision for host ml-serve1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
06:23 <marostegui@cumin1002> dbctl commit (dc=all): 'db1170 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P79449 and previous config saved to /var/cache/conftool/dbconfig/20250721-062353-root.json [production]
06:17 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P79448 and previous config saved to /var/cache/conftool/dbconfig/20250721-061726-root.json [production]
06:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1170 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P79447 and previous config saved to /var/cache/conftool/dbconfig/20250721-061606-marostegui.json [production]
06:16 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
06:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2150 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P79446 and previous config saved to /var/cache/conftool/dbconfig/20250721-060923-marostegui.json [production]
06:09 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2150.codfw.wmnet with reason: Maintenance [production]
05:52 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance [production]
04:59 <kevinbazira@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
04:56 <kevinbazira@deploy1003> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
2025-07-20 §
22:50 <Reedy> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1170761 [releng]
17:59 <wmbot~bd808@tools-bastion-12> Restarted to deploy new image from 449e522e (T397571, T393928) [tools.gitlab-content]
17:58 <wmbot~bd808@tools-bastion-12> Built new image from 449e522e (T397571, T393928) [tools.gitlab-content]
10:44 <andrewbogott> rebooting cloudcephosd1006 to give us another few days before the memory runs out. T399858 [admin]
2025-07-19 §
19:09 <Ammar> Ran fixStuckGlobalRename.php for T399985 [production]
17:37 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 1f86d57dfe (add missing files) [tools.pagepile-visual-filter]
17:34 <wmbot~lucaswerkmeister@tools-bastion-13> deployed b737a66700 (Python 3.13 + Toolforge Build Service) [tools.pagepile-visual-filter]
17:34 <lucaswerkmeister> webservice stop && mv www{,-unused-tool-now-runs-on-buildservice} && wget https://gitlab.wikimedia.org/toolforge-repos/pagepile-visual-filter/-/raw/b737a66700/service.template && webservice start [tools.pagepile-visual-filter]
17:32 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 431359e7d9 (update URLs in README.md) [tools.speedpatrolling]
17:27 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 3f0897135f (read config from envvars) [tools.pagepile-visual-filter]
17:27 <wmbot~lucaswerkmeister@tools-bastion-13> commented out config.yaml, should use envvars instead [tools.pagepile-visual-filter]
17:27 <lucaswerkmeister> python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SECRET_KEY"]))' | toolforge envvars create TOOL_SECRET_KEY [tools.pagepile-visual-filter]
17:25 <wmbot~lucaswerkmeister@tools-bastion-13> deployed 13dd46ac60 (add health-check-path) [tools.pagepile-visual-filter]
17:18 <wmbot~lucaswerkmeister@tools-bastion-13> deployed cb694611a5 (split dev requirements from prod requirements) [tools.pagepile-visual-filter]
17:10 <wmbot~lucaswerkmeister@tools-bastion-13> deployed c8872b20e7 (upgrade dependencies) [tools.pagepile-visual-filter]
15:58 <wmbot~lucaswerkmeister@tools-bastion-13> deployed a11bdfe30c (Python 3.13 + Toolforge Build Service) [tools.speedpatrolling]
15:57 <lucaswerkmeister> webservice stop && mv www{,-unused-tool-now-runs-on-buildservice} && wget https://gitlab.wikimedia.org/toolforge-repos/speedpatrolling/-/raw/a11bdfe30c/service.template && webservice start [tools.speedpatrolling]
13:14 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 [tools]
13:08 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 [tools]
13:05 <andrewbogott> restarted neutron-metadata-agent on cloudnet100[56], again [admin]
11:22 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2239.codfw.wmnet with reason: Maintenance [production]
11:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2227 (T399249)', diff saved to https://phabricator.wikimedia.org/P79444 and previous config saved to /var/cache/conftool/dbconfig/20250719-112151-marostegui.json [production]
11:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P79443 and previous config saved to /var/cache/conftool/dbconfig/20250719-110644-marostegui.json [production]
10:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P79442 and previous config saved to /var/cache/conftool/dbconfig/20250719-105135-marostegui.json [production]
10:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2227 (T399249)', diff saved to https://phabricator.wikimedia.org/P79441 and previous config saved to /var/cache/conftool/dbconfig/20250719-103628-marostegui.json [production]
08:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2227 (T399249)', diff saved to https://phabricator.wikimedia.org/P79440 and previous config saved to /var/cache/conftool/dbconfig/20250719-080850-marostegui.json [production]
08:08 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2227.codfw.wmnet with reason: Maintenance [production]
08:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2205 (T399249)', diff saved to https://phabricator.wikimedia.org/P79439 and previous config saved to /var/cache/conftool/dbconfig/20250719-080837-marostegui.json [production]
07:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P79438 and previous config saved to /var/cache/conftool/dbconfig/20250719-075330-marostegui.json [production]
07:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P79437 and previous config saved to /var/cache/conftool/dbconfig/20250719-073822-marostegui.json [production]
07:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2205 (T399249)', diff saved to https://phabricator.wikimedia.org/P79436 and previous config saved to /var/cache/conftool/dbconfig/20250719-072315-marostegui.json [production]
06:59 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
06:59 <marostegui@cumin1002> START - Cookbook sre.mysql.parsercache [production]
06:55 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
06:55 <marostegui@cumin1002> START - Cookbook sre.mysql.parsercache [production]
05:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2205 (T399249)', diff saved to https://phabricator.wikimedia.org/P79433 and previous config saved to /var/cache/conftool/dbconfig/20250719-050137-marostegui.json [production]
05:01 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance [production]