101-150 of 3589 results (20ms)
2022-06-21 §
15:23 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:20 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:18 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0913.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:07 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0912.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
2022-06-03 §
20:07 <wm-bot2> created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:51 <balloons> Scaling webservice nodes to 20, using new 8G swap flavor T309821 [tools]
19:35 <wm-bot2> created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:03 <wm-bot2> created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:01 <wm-bot2> created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
18:59 <balloons> depooled old nodes, bringing entirely new grid of nodes online T309821 [tools]
18:22 <wm-bot2> created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
17:54 <wm-bot2> created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
17:52 <wm-bot2> created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
16:59 <andrewbogott> building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space [tools]
16:56 <wm-bot2> created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
15:50 <balloons> fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821 [tools]
15:50 <balloons> temp add 1.0G swap to sgeweblight hosts T309821 [tools]
15:50 <balloons> fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821 [tools]
15:49 <balloons> temp add 1.0G swap to sgeweblight hosts t309821 [tools]
13:25 <bd808> Upgrading fleet to tools-webservice 0.86 (T309821) [tools]
13:20 <bd808> publish tools-webservice 0.86 (T309821) [tools]
12:46 <taavi> start webservicemonitor on tools-sgecron-01 T309821 [tools]
10:36 <taavi> draining each sgeweblight node one by one, and removing the jobs stuck in 'deleting' too [tools]
05:05 <taavi> removing duplicate (there should be only one per tool) web service jobs from the grid T309821 [tools]
04:52 <taavi> revert bd808's changes to profile::toolforge::active_proxy_host [tools]
03:21 <bd808> Cleared queue error states after deploying new toolforge-webservice package (T309821) [tools]
03:10 <bd808> publish tools-webservice 0.85 with hack for T309821 [tools]
2022-06-02 §
22:26 <bd808> Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler [tools]
21:56 <bd808> Removed legacy "active_proxy_host" hiera setting [tools]
21:55 <bd808> Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for profile::toolforge::active_proxy_host key [tools]
21:41 <bd808> Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for active_redis key [tools]
21:22 <wm-bot2> created node tools-sgeweblight-10-8.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
12:42 <wm-bot2> rebooting stretch exec grid workers - cookbook ran by taavi@runko [tools]
12:13 <wm-bot2> created node tools-sgeweblight-10-7.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
12:03 <dcaro> refresh prometheus certs (T308402) [tools]
11:47 <dcaro> refresh registry-admission-controller certs (T308402) [tools]
11:42 <dcaro> refresh ingress-admission-controller certs (T308402) [tools]
11:36 <dcaro> refresh volume-admission-controller certs (T308402) [tools]
11:24 <wm-bot2> created node tools-sgeweblight-10-6.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
11:17 <taavi> publish jobutils 1.44 that updates the grid default from stretch to buster T277653 [tools]
10:16 <taavi> publish tools-webservice 0.84 that updates the grid default from stretch to buster T277653 [tools]
09:54 <wm-bot2> created node tools-sgeexec-10-14.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
2022-06-01 §
11:18 <taavi> depool and remove tools-sgeexec-09[07-14] [tools]
2022-05-31 §
16:51 <taavi> delete tools-sgeexec-0904 for T309525 experimentation [tools]
2022-05-30 §
08:24 <taavi> depool tools-sgeexec-[0901-0909] (7 nodes total) T277653 [tools]
2022-05-26 §
15:39 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e6fa299) (T309146) - cookbook ran by taavi@runko [tools]
2022-05-22 §
17:04 <taavi> failover tools-redis to the updated cluster T278541 [tools]
16:42 <wm-bot2> removing grid node tools-sgeexec-0940.tools.eqiad1.wikimedia.cloud (T308982) - cookbook ran by taavi@runko [tools]
2022-05-16 §
14:02 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (7037eca) - cookbook ran by taavi@runko [tools]
2022-05-14 §
10:47 <taavi> hard reboot unresponsible tools-sgeexec-0940 [tools]