1-50 of 125 results (31ms)
2024-04-11 §
20:08 <mutante> zuul-1001 - switching to new puppetmaster-1003 in puppet.conf manually, switched project defaults in repo too [devtools]
19:58 <mutante> manually editing puppet.conf to use puppetmaster-1003 instead of -1001 because you can't switch the puppetmaster via puppet if puppet is already broken :) [devtools]
19:41 <mutante> switching gitlab-runner-1005 from puppetmaster-1001 to puppetmaster-1003 via web Hiera [devtools]
19:03 <mutante> - deleting instance contint-bullseye which was only used by me for a test before we created contint1003 in prod T334517 T361224 [devtools]
18:38 <mutante> - attempting to fix puppet run on vrts-1001 related to switching prod to cfssl for SSL cers [devtools]
18:23 <mutante> - shutting down puppetmaster-1001 on buster - should now be replaced by puppetmaster-1003 on bookworm (thanks brennen) T360964 T360470 [devtools]
18:01 <mutante> - shutting down instance devtools-puppetdb1001 - which is on buster - basically to see what breaks of complains, if anything [devtools]
2024-04-09 §
19:46 <mutante> - soft rebooting gerrit-prod-1001 buster instance (to be removed ) [devtools]
2024-04-01 §
17:27 <mutante> - added profile::pki::client::ensure: present to instance hiera for etherpad-bookworm - fixing broken puppet run [devtools]
17:25 <mutante> - attempting to fix puppet on instance etherpad-bookworm but SSL provider cfssl doesn't appear to work in cloud [devtools]
2024-02-09 §
19:08 <mutante> deleting instance phabricator-prod-1001 (shut down a couple days ago, buster instance replaced by phabricator-bullseye instance) T356530 [devtools]
2024-02-07 §
21:51 <mutante> rebooting gitlab-prod-1002 T356906 [devtools]
2024-02-02 §
23:37 <mutante> phabricator-bullseye configured auth provider for simple passwords, letting users register users, locked auth config option again [devtools]
23:34 <mutante> phabricator-bullseye:/srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./auth unlock [devtools]
22:53 <mutante> - phabricator-bullseye sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' | sudo ./config set security.alternate-file-domain 'https://phab-usercontent.wmcloud.org' | delete proxy phab-prod-usercontent, create proxy phab-usercontent.wmcloud.org, restart apache T356530 [devtools]
22:37 <mutante> - phabricator-bullseye - /srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' T356530 [devtools]
22:31 <mutante> - phabricator-bullseye, instance hiera: setting phabricator_domain to phabricator.wmcloud.org and phabricator_altdomain to phab-usercontent.wmcloud.org T356530 [devtools]
22:26 <mutante> - deleted proxies phab.wmflabs.org, phab-prod-usercontent.wmflabs.org, phabricator.wmflabs.org - created proxies phabricator.wmcloud.org, phab-usercontent.wmcloud.org - wmflabs names are legacy and should migrate T356530 [devtools]
20:59 <mutante> - changing phabricator domain in instance Hiera of phabricator-bullseye to phab.wmflabs.org and running puppet to update apache config/rewrite rules T356530 [devtools]
20:41 <mutante> shutting down instance phabricator-prod-1001 (buster), replaced by phab-bullseye (bullseye) T356530 [devtools]
20:38 <mutante> deleting proxy phab-bull.wmcloud.org after previous proxy names are switched to bullseye backend T356530 [devtools]
20:17 <mutante> editing proxies phab.wmflabs.org and phab-prod-usercontent.wmflabs.org to point to bullseye instance instead of buster T356530 [devtools]
20:13 <mutante> running "scap deploy" in /srv/deployment/phabricator/deployment on deploy-1004 which deploys to phabricator-bullseye and phabricator-prod-1001 T356530 [devtools]
19:29 <mutante> deleting web proxy phabricator-prod.wmflabs.org which pointed to port 443 on the buster instance and timed out T356530 [devtools]
19:12 <mutante> deleting web proxy phorge.wmcloud.org which pointed to 172.16.7.98 which doesn't exist anymore T356530 [devtools]
2024-01-05 §
21:30 <mutante> contint-bullseye - sudo /usr/sbin/a2dismod mpm_event ; sudo /usr/sbin/a2endmod php74 - the usual issue we have had for years [devtools]
2023-12-18 §
21:53 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) [devtools]
21:53 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase [devtools]
21:52 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T353671) [devtools]
21:51 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T353671) [devtools]
2023-12-13 §
23:49 <mutante> starting manual gitlab upgrade process on gitlab-prod-1002 [devtools]
2023-11-30 §
21:03 <mutante> - phabricator-bullseye - created user app_user and granted privileges in mysql for user from 127.0.0.1, ran ./phabricator/bin/storage upgrade --force; set 'phabricator_domain: phab-bull.wmcloud.org' in web Hiera [devtools]
20:33 <mutante> - phabricator-bullseye - running 'mariadb-secure-installation' interactive script - this fixed mysql shell which previously exited with "bash: /nonexistent: No such file or directory" [devtools]
20:29 <mutante> - phabricator-bullseye - running 'mariadb-install-db' [devtools]
20:27 <mutante> - phabricator-bullseye - attempting to fix mariadb/mysql server, apt-get remove mariadb-server, running puppet, debugging why it wont start [devtools]
2023-11-21 §
21:52 <mutante> - commit fake key for phabricator-bullseye host in git /var/lib/git/labs/private/modules/secret/secrets/ssl on puppetmaster-1001.devtools T327068 [devtools]
21:41 <mutante> - cert issue on new machine related to having local puppetmaster, like T349937#9288547 except "rm -rf /var/lib/puppet/ssl" was enough since puppetmaster did auto-sign new CSR - T327068 [devtools]
21:24 <mutante> - initial puppet run on newly created VM fails with "SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster-1001.devtools.eqiad.wmflabs" T327068 [devtools]
21:05 <mutante> - creating instance phabricator-bullseye g3.cores2.ram4.disk20 T327068 [devtools]
21:00 <mutante> - deleted instance phorge-1001 to get quota back and allow for creting new phabricator-on-bullseye instance T328595 T327068 [devtools]
2023-04-12 §
19:26 <mutante> - vrts-1001 - editing /etc/my.cnf to set mariadb datadir to /var/lib/mysql instead of /srv/sqldata and restart service, issue like T329571 [devtools]
2023-03-11 §
00:20 <mutante> - on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595 [devtools]
2023-03-07 §
23:34 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL max_allowed_packet=33554432; [devtools]
23:33 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL local_infile=0; [devtools]
23:31 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL sql_mode = "STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"; [devtools]
2023-02-13 §
23:09 <mutante> - shutting down gerrit-prod-1001 [devtools]
22:22 <mutante> certbot renew --apache fixed cert issue - https://ldapauth-gitldap.wmflabs.org/ does not exist unrelatedly - T329444 [devtools]
22:18 <mutante> install package python3-certbot-apache on gerrit-prod-1001 - T329444 [devtools]
22:03 <mutante> - re-activating disabled puppet on gerrit-prod-1001 (reason given was 'gerrit deploy' but it was about 17 days ago) [devtools]
21:58 <mutante> rebooting instance gerrit-prod-1001 which can't be reached T329444 [devtools]