1601-1650 of 5111 results (30ms)
2022-06-27 §
17:49 <wm-bot2> removing grid node tools-sgeexec-0942.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:15 <taavi> T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen [tools]
14:58 <taavi> renew puppet ca cert and certificate for tools-puppetmaster-02 T311412 [tools]
14:50 <taavi> backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412 [tools]
2022-06-23 §
17:51 <wm-bot2> removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:49 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0916.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:46 <wm-bot2> removing grid node tools-sgewebgrid-generic-0901.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:32 <wm-bot2> removing grid node tools-sgeexec-0939.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:30 <wm-bot2> removing grid node tools-sgeexec-0938.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:27 <wm-bot2> removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:22 <wm-bot2> removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:19 <wm-bot2> removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:17 <wm-bot2> removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:14 <wm-bot2> removing grid node tools-sgeexec-0933.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:11 <wm-bot2> removing grid node tools-sgeexec-0932.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:09 <wm-bot2> removing grid node tools-sgeexec-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:30 <wm-bot2> removing grid node tools-sgeexec-0947.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
13:59 <taavi> removing remaining continuous jobs from the stretch grid T277653 [tools]
2022-06-22 §
15:54 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0917.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:51 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0918.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:47 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0919.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:45 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
2022-06-21 §
15:23 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:20 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:18 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0913.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:07 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0912.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
2022-06-03 §
20:07 <wm-bot2> created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:51 <balloons> Scaling webservice nodes to 20, using new 8G swap flavor T309821 [tools]
19:35 <wm-bot2> created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:03 <wm-bot2> created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
19:01 <wm-bot2> created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
18:59 <balloons> depooled old nodes, bringing entirely new grid of nodes online T309821 [tools]
18:22 <wm-bot2> created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
17:54 <wm-bot2> created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
17:52 <wm-bot2> created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
16:59 <andrewbogott> building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space [tools]
16:56 <wm-bot2> created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster [tools]
15:50 <balloons> fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821 [tools]
15:50 <balloons> temp add 1.0G swap to sgeweblight hosts T309821 [tools]
15:50 <balloons> fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821 [tools]
15:49 <balloons> temp add 1.0G swap to sgeweblight hosts t309821 [tools]
13:25 <bd808> Upgrading fleet to tools-webservice 0.86 (T309821) [tools]
13:20 <bd808> publish tools-webservice 0.86 (T309821) [tools]
12:46 <taavi> start webservicemonitor on tools-sgecron-01 T309821 [tools]
10:36 <taavi> draining each sgeweblight node one by one, and removing the jobs stuck in 'deleting' too [tools]
05:05 <taavi> removing duplicate (there should be only one per tool) web service jobs from the grid T309821 [tools]
04:52 <taavi> revert bd808's changes to profile::toolforge::active_proxy_host [tools]
03:21 <bd808> Cleared queue error states after deploying new toolforge-webservice package (T309821) [tools]
03:10 <bd808> publish tools-webservice 0.85 with hack for T309821 [tools]
2022-06-02 §
22:26 <bd808> Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler [tools]