1-50 of 99 results (29ms)
2023-12-18 §
21:53 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) [devtools]
21:53 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase [devtools]
21:52 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T353671) [devtools]
21:51 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T353671) [devtools]
2023-12-13 §
23:49 <mutante> starting manual gitlab upgrade process on gitlab-prod-1002 [devtools]
2023-11-30 §
21:03 <mutante> - phabricator-bullseye - created user app_user and granted privileges in mysql for user from 127.0.0.1, ran ./phabricator/bin/storage upgrade --force; set 'phabricator_domain: phab-bull.wmcloud.org' in web Hiera [devtools]
20:33 <mutante> - phabricator-bullseye - running 'mariadb-secure-installation' interactive script - this fixed mysql shell which previously exited with "bash: /nonexistent: No such file or directory" [devtools]
20:29 <mutante> - phabricator-bullseye - running 'mariadb-install-db' [devtools]
20:27 <mutante> - phabricator-bullseye - attempting to fix mariadb/mysql server, apt-get remove mariadb-server, running puppet, debugging why it wont start [devtools]
2023-11-21 §
21:52 <mutante> - commit fake key for phabricator-bullseye host in git /var/lib/git/labs/private/modules/secret/secrets/ssl on puppetmaster-1001.devtools T327068 [devtools]
21:41 <mutante> - cert issue on new machine related to having local puppetmaster, like T349937#9288547 except "rm -rf /var/lib/puppet/ssl" was enough since puppetmaster did auto-sign new CSR - T327068 [devtools]
21:24 <mutante> - initial puppet run on newly created VM fails with "SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster-1001.devtools.eqiad.wmflabs" T327068 [devtools]
21:05 <mutante> - creating instance phabricator-bullseye g3.cores2.ram4.disk20 T327068 [devtools]
21:00 <mutante> - deleted instance phorge-1001 to get quota back and allow for creting new phabricator-on-bullseye instance T328595 T327068 [devtools]
2023-04-12 §
19:26 <mutante> - vrts-1001 - editing /etc/my.cnf to set mariadb datadir to /var/lib/mysql instead of /srv/sqldata and restart service, issue like T329571 [devtools]
2023-03-11 §
00:20 <mutante> - on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595 [devtools]
2023-03-07 §
23:34 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL max_allowed_packet=33554432; [devtools]
23:33 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL local_infile=0; [devtools]
23:31 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL sql_mode = "STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"; [devtools]
2023-02-13 §
23:09 <mutante> - shutting down gerrit-prod-1001 [devtools]
22:22 <mutante> certbot renew --apache fixed cert issue - https://ldapauth-gitldap.wmflabs.org/ does not exist unrelatedly - T329444 [devtools]
22:18 <mutante> install package python3-certbot-apache on gerrit-prod-1001 - T329444 [devtools]
22:03 <mutante> - re-activating disabled puppet on gerrit-prod-1001 (reason given was 'gerrit deploy' but it was about 17 days ago) [devtools]
21:58 <mutante> rebooting instance gerrit-prod-1001 which can't be reached T329444 [devtools]
2023-01-31 §
22:39 <mutante> remove role::gitlab from gitlab-prod-1001. to be replaced with gitlab-prod-1002. T318521 [devtools]
2023-01-28 §
16:26 <taavi> adjust gitlab-prod-1002 network port settings to allow adding the secondary IP, requested in T318521 [devtools]
2023-01-24 §
09:47 <wm-bot2> Increased quotas by 1 instances (T327750) - cookbook ran by arturo@nostromo [devtools]
2022-11-30 §
15:57 <wm-bot2> Increased quotas by 1 floating-ips (T323986) - cookbook ran by dcaro@vulcanus [devtools]
2022-11-17 §
18:18 <andrewbogott> committed a local puppet change on puppetprimary to fix upstream syncs [devtools]
2022-10-28 §
20:24 <mutante> - removing from Horizon / project-wide hiera: profile::phabricator::main::manage_scap_user: true (set in the repo) [devtools]
20:23 <mutante> - removing from Horizon / project-wide hiera: profile::keyholder::server::require_encrypted_keys: 'no' (set in the repo) [devtools]
20:19 <mutante> - removing from Horizon / project-wide hiera: profile::gerrit::daemon_user: gerrit2, profile::gerrit::manage_scap_user: true, profile::gerrit::scap_user: gerrit-deploy (all of these are set in the repo) [devtools]
2022-10-19 §
19:21 <mutante> - on puppetmaster-1001.devtools created /var/lib/puppet/volatile/GeoIP directory - to fix puppet error on deploy-1004.devtools - reacting to puppet-broken-nagging-emails [devtools]
2022-06-24 §
12:43 <taavi> `os quota set devtools --ram 45056 --cores 22 --instances 9` # T311302 [devtools]
2022-06-15 §
20:30 <mutante> - created gitlab-runner-1002 - applied puppet role - attached cinder volume "docker" - running puppet again [devtools]
17:06 <mutante> deleting instance gitlab-runner-1001 which just disconnects people. gut feeling is it has to do with the fact that a previous instance name was used again [devtools]
2022-06-14 §
23:29 <mutante> - creating instance gitlab-runner-1001 since we did not have a test machine for gitlab-runners but need one to test things like gerrit:791655 before hitting prod T308271 [devtools]
2022-04-29 §
20:59 <mutante> - restarting instance gitlab-prod-1001 - No route to host [devtools]
20:55 <mutante> - attempting to soft reboot instance deploy1004 (got the puppet fail mail and wasnt reachable by ssh), this happened lately as well to gitlab-prod-1001, same project, different instance, but this time it doesn't just come back yet [devtools]
2022-04-20 §
17:03 <mutante> soft rebooting gitlab-prod-1001 which was sending "failed puppet" reports and was unreachable, just like the other day. [devtools]
2022-04-18 §
19:08 <mutante> - gitlab-prod-1001 is indeed back after soft rebooting the instance. uptime 1 min T297411 [devtools]
19:07 <mutante> - gitlab-prod-1001 randomly stopped working. we got the "puppet failed" mails without having made changes and can't ssh to the instance anymore when trying to check out why. trying soft reboot via Horizon T297411 [devtools]
2022-04-15 §
18:00 <mutante> - deleting deploy-1002 - use deploy-1004 instead - T306069 [devtools]
17:03 <mutante> - not sure if possible (for me) to create a bullseye deployment server in cloud, using scap: failed: Execution of '/usr/bin/scap deploy --init', missing PHP packages, missing prometheus-mcrouter-exporter and more T306069 [devtools]
17:02 <mutante> - not sure if possible (for me) to create ad deployment server in cloud, using scap: failed: Execution of '/usr/bin/scap deploy --init' [devtools]
16:40 <mutante> : creating deploy1003 to replace deploy1002 T306069 [devtools]
16:36 <mutante> : deleting instance gitlab-runner-1001 - was just for testing, real runners are upgrade in their own project [devtools]
2022-03-02 §
22:22 <mutante> - creating gitlab-runner-1001 on bullseye - purely test for T297659 [devtools]
2022-03-01 §
18:16 <taavi> allocated secondary IP for gitlab-prod-1001 per request on T302803 [devtools]
2022-02-15 §
16:08 <taavi> created devtools.wmcloud.org dns zone for the devtools project T301793 [devtools]