|
2022-06-23
§
|
| 17:27 |
<wm-bot2> |
removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:22 |
<wm-bot2> |
removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:19 |
<wm-bot2> |
removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:17 |
<wm-bot2> |
removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:14 |
<wm-bot2> |
removing grid node tools-sgeexec-0933.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:11 |
<wm-bot2> |
removing grid node tools-sgeexec-0932.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:09 |
<wm-bot2> |
removing grid node tools-sgeexec-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 15:30 |
<wm-bot2> |
removing grid node tools-sgeexec-0947.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 13:59 |
<taavi> |
removing remaining continuous jobs from the stretch grid T277653 |
[tools] |
|
2022-06-03
§
|
| 20:07 |
<wm-bot2> |
created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:51 |
<balloons> |
Scaling webservice nodes to 20, using new 8G swap flavor T309821 |
[tools] |
| 19:35 |
<wm-bot2> |
created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:03 |
<wm-bot2> |
created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:01 |
<wm-bot2> |
created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 18:59 |
<balloons> |
depooled old nodes, bringing entirely new grid of nodes online T309821 |
[tools] |
| 18:22 |
<wm-bot2> |
created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 17:54 |
<wm-bot2> |
created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 17:52 |
<wm-bot2> |
created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 16:59 |
<andrewbogott> |
building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space |
[tools] |
| 16:56 |
<wm-bot2> |
created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 15:50 |
<balloons> |
fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821 |
[tools] |
| 15:50 |
<balloons> |
temp add 1.0G swap to sgeweblight hosts T309821 |
[tools] |
| 15:50 |
<balloons> |
fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821 |
[tools] |
| 15:49 |
<balloons> |
temp add 1.0G swap to sgeweblight hosts t309821 |
[tools] |
| 13:25 |
<bd808> |
Upgrading fleet to tools-webservice 0.86 (T309821) |
[tools] |
| 13:20 |
<bd808> |
publish tools-webservice 0.86 (T309821) |
[tools] |
| 12:46 |
<taavi> |
start webservicemonitor on tools-sgecron-01 T309821 |
[tools] |
| 10:36 |
<taavi> |
draining each sgeweblight node one by one, and removing the jobs stuck in 'deleting' too |
[tools] |
| 05:05 |
<taavi> |
removing duplicate (there should be only one per tool) web service jobs from the grid T309821 |
[tools] |
| 04:52 |
<taavi> |
revert bd808's changes to profile::toolforge::active_proxy_host |
[tools] |
| 03:21 |
<bd808> |
Cleared queue error states after deploying new toolforge-webservice package (T309821) |
[tools] |
| 03:10 |
<bd808> |
publish tools-webservice 0.85 with hack for T309821 |
[tools] |
|
2022-06-02
§
|
| 22:26 |
<bd808> |
Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler |
[tools] |
| 21:56 |
<bd808> |
Removed legacy "active_proxy_host" hiera setting |
[tools] |
| 21:55 |
<bd808> |
Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for profile::toolforge::active_proxy_host key |
[tools] |
| 21:41 |
<bd808> |
Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for active_redis key |
[tools] |
| 21:22 |
<wm-bot2> |
created node tools-sgeweblight-10-8.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko |
[tools] |
| 12:42 |
<wm-bot2> |
rebooting stretch exec grid workers - cookbook ran by taavi@runko |
[tools] |
| 12:13 |
<wm-bot2> |
created node tools-sgeweblight-10-7.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko |
[tools] |
| 12:03 |
<dcaro> |
refresh prometheus certs (T308402) |
[tools] |
| 11:47 |
<dcaro> |
refresh registry-admission-controller certs (T308402) |
[tools] |
| 11:42 |
<dcaro> |
refresh ingress-admission-controller certs (T308402) |
[tools] |