|
2022-06-27
§
|
| 18:14 |
<taavi> |
restart calico, appears to have got stuck after the ca replacement operation |
[tools] |
| 18:02 |
<taavi> |
switchover active cron server to tools-sgecron-2 T284767 |
[tools] |
| 17:54 |
<wm-bot2> |
removing grid node tools-sgewebgrid-lighttpd-0915.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:52 |
<wm-bot2> |
removing grid node tools-sgewebgrid-generic-0902.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:49 |
<wm-bot2> |
removing grid node tools-sgeexec-0942.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:15 |
<taavi> |
T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen |
[tools] |
| 14:58 |
<taavi> |
renew puppet ca cert and certificate for tools-puppetmaster-02 T311412 |
[tools] |
| 14:50 |
<taavi> |
backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412 |
[tools] |
|
2022-06-23
§
|
| 17:51 |
<wm-bot2> |
removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:49 |
<wm-bot2> |
removing grid node tools-sgewebgrid-lighttpd-0916.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:46 |
<wm-bot2> |
removing grid node tools-sgewebgrid-generic-0901.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:32 |
<wm-bot2> |
removing grid node tools-sgeexec-0939.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:30 |
<wm-bot2> |
removing grid node tools-sgeexec-0938.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:27 |
<wm-bot2> |
removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:22 |
<wm-bot2> |
removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:19 |
<wm-bot2> |
removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:17 |
<wm-bot2> |
removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:14 |
<wm-bot2> |
removing grid node tools-sgeexec-0933.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:11 |
<wm-bot2> |
removing grid node tools-sgeexec-0932.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 17:09 |
<wm-bot2> |
removing grid node tools-sgeexec-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 15:30 |
<wm-bot2> |
removing grid node tools-sgeexec-0947.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko |
[tools] |
| 13:59 |
<taavi> |
removing remaining continuous jobs from the stretch grid T277653 |
[tools] |
|
2022-06-03
§
|
| 20:07 |
<wm-bot2> |
created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:51 |
<balloons> |
Scaling webservice nodes to 20, using new 8G swap flavor T309821 |
[tools] |
| 19:35 |
<wm-bot2> |
created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:03 |
<wm-bot2> |
created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 19:01 |
<wm-bot2> |
created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 18:59 |
<balloons> |
depooled old nodes, bringing entirely new grid of nodes online T309821 |
[tools] |
| 18:22 |
<wm-bot2> |
created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 17:54 |
<wm-bot2> |
created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 17:52 |
<wm-bot2> |
created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 16:59 |
<andrewbogott> |
building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space |
[tools] |
| 16:56 |
<wm-bot2> |
created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster |
[tools] |
| 15:50 |
<balloons> |
fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821 |
[tools] |
| 15:50 |
<balloons> |
temp add 1.0G swap to sgeweblight hosts T309821 |
[tools] |
| 15:50 |
<balloons> |
fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821 |
[tools] |
| 15:49 |
<balloons> |
temp add 1.0G swap to sgeweblight hosts t309821 |
[tools] |
| 13:25 |
<bd808> |
Upgrading fleet to tools-webservice 0.86 (T309821) |
[tools] |
| 13:20 |
<bd808> |
publish tools-webservice 0.86 (T309821) |
[tools] |
| 12:46 |
<taavi> |
start webservicemonitor on tools-sgecron-01 T309821 |
[tools] |