1-50 of 3548 results (12ms)
2022-08-24 §
12:20 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (eba66bc) - cookbook ran by taavi@runko [tools]
12:20 <taavi> upgrading ingress-nginx to v1.3 [tools]
2022-08-20 §
07:44 <dcaro_away> all k8s nodes ready now \o/ (T315718) [tools]
07:43 <dcaro_away> rebooted tools-k8s-control-2, seemed stuck trying to wait for tools home (nfs?), after reboot came back up (T315718) [tools]
07:41 <dcaro_away> cloudvirt1023 down took out 3 workers, 1 control, and a grid exec and a weblight, they are taking long to restart, looking (T315718) [tools]
2022-08-18 §
14:45 <andrewbogott> adding lucaswerkmeister as projectadmin (T314527) [tools]
14:43 <andrewbogott> removing some inactive projectadmins: rush, petrb, mdipietro, jeh, krenair [tools]
2022-08-17 §
16:34 <taavi> kubectl sudo delete cm -n tool-wdml maintain-kubeusers # T315459 [tools]
08:30 <taavi> failing the grid from the shadow back to the master, some disruption expected [tools]
2022-08-16 §
17:28 <taavi> fail over docker-registry, tools-docker-registry-06->docker-registry-05 [tools]
2022-08-11 §
16:57 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko [tools]
16:55 <taavi> restart puppetdb on tools-puppetdb-1, crashed during the ceph issues [tools]
2022-08-05 §
15:08 <wm-bot2> removing grid node tools-sgewebgen-10-1.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:05 <wm-bot2> removing grid node tools-sgeexec-10-12.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:00 <wm-bot2> created node tools-sgewebgen-10-3.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
2022-08-03 §
15:51 <dhinus> recreated jobs-api pods to pick up new ConfigMap [tools]
15:02 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c47ac41) - cookbook ran by fran@MacBook-Pro.station [tools]
2022-07-20 §
19:31 <taavi> reboot toolserver-proxy-01 to free up disk space probably held by stale file handles [tools]
08:06 <wm-bot2> removing grid node tools-sgeexec-10-6.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
2022-07-19 §
17:53 <wm-bot2> created node tools-sgeexec-10-21.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
17:00 <wm-bot2> removing grid node tools-sgeexec-10-3.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
16:58 <wm-bot2> removing grid node tools-sgeexec-10-4.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
16:24 <wm-bot2> created node tools-sgeexec-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
15:59 <taavi> tag current maintain-kubernetes :beta image as: :latest [tools]
2022-07-17 §
15:52 <wm-bot2> removing grid node tools-sgeexec-10-10.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:43 <wm-bot2> removing grid node tools-sgeexec-10-2.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
13:26 <wm-bot2> created node tools-sgeexec-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
2022-07-14 §
13:48 <taavi> rebooting tools-sgeexec-10-2 [tools]
2022-07-13 §
12:09 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus [tools]
2022-07-11 §
16:06 <wm-bot2> Increased quotas by {self.increases} (T312692) - cookbook ran by nskaggs@x1carbon [tools]
2022-07-07 §
07:34 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus [tools]
2022-06-28 §
17:34 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master (T311538) - cookbook ran by dcaro@vulcanus [tools]
15:51 <taavi> add 4096G cinder quota T311509 [tools]
2022-06-27 §
18:14 <taavi> restart calico, appears to have got stuck after the ca replacement operation [tools]
18:02 <taavi> switchover active cron server to tools-sgecron-2 T284767 [tools]
17:54 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0915.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:52 <wm-bot2> removing grid node tools-sgewebgrid-generic-0902.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:49 <wm-bot2> removing grid node tools-sgeexec-0942.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:15 <taavi> T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen [tools]
14:58 <taavi> renew puppet ca cert and certificate for tools-puppetmaster-02 T311412 [tools]
14:50 <taavi> backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412 [tools]
2022-06-23 §
17:51 <wm-bot2> removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:49 <wm-bot2> removing grid node tools-sgewebgrid-lighttpd-0916.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:46 <wm-bot2> removing grid node tools-sgewebgrid-generic-0901.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:32 <wm-bot2> removing grid node tools-sgeexec-0939.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:30 <wm-bot2> removing grid node tools-sgeexec-0938.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:27 <wm-bot2> removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:22 <wm-bot2> removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:19 <wm-bot2> removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
17:17 <wm-bot2> removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]