51-100 of 158 results (30ms)
2024-02-02 §
22:26 <mutante> - deleted proxies phab.wmflabs.org, phab-prod-usercontent.wmflabs.org, phabricator.wmflabs.org - created proxies phabricator.wmcloud.org, phab-usercontent.wmcloud.org - wmflabs names are legacy and should migrate T356530 [devtools]
20:59 <mutante> - changing phabricator domain in instance Hiera of phabricator-bullseye to phab.wmflabs.org and running puppet to update apache config/rewrite rules T356530 [devtools]
20:41 <mutante> shutting down instance phabricator-prod-1001 (buster), replaced by phab-bullseye (bullseye) T356530 [devtools]
20:38 <mutante> deleting proxy phab-bull.wmcloud.org after previous proxy names are switched to bullseye backend T356530 [devtools]
20:17 <mutante> editing proxies phab.wmflabs.org and phab-prod-usercontent.wmflabs.org to point to bullseye instance instead of buster T356530 [devtools]
20:13 <mutante> running "scap deploy" in /srv/deployment/phabricator/deployment on deploy-1004 which deploys to phabricator-bullseye and phabricator-prod-1001 T356530 [devtools]
19:29 <mutante> deleting web proxy phabricator-prod.wmflabs.org which pointed to port 443 on the buster instance and timed out T356530 [devtools]
19:12 <mutante> deleting web proxy phorge.wmcloud.org which pointed to 172.16.7.98 which doesn't exist anymore T356530 [devtools]
2024-01-05 §
21:30 <mutante> contint-bullseye - sudo /usr/sbin/a2dismod mpm_event ; sudo /usr/sbin/a2endmod php74 - the usual issue we have had for years [devtools]
2023-12-18 §
21:53 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) [devtools]
21:53 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase [devtools]
21:52 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T353671) [devtools]
21:51 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T353671) [devtools]
2023-12-13 §
23:49 <mutante> starting manual gitlab upgrade process on gitlab-prod-1002 [devtools]
2023-11-30 §
21:03 <mutante> - phabricator-bullseye - created user app_user and granted privileges in mysql for user from 127.0.0.1, ran ./phabricator/bin/storage upgrade --force; set 'phabricator_domain: phab-bull.wmcloud.org' in web Hiera [devtools]
20:33 <mutante> - phabricator-bullseye - running 'mariadb-secure-installation' interactive script - this fixed mysql shell which previously exited with "bash: /nonexistent: No such file or directory" [devtools]
20:29 <mutante> - phabricator-bullseye - running 'mariadb-install-db' [devtools]
20:27 <mutante> - phabricator-bullseye - attempting to fix mariadb/mysql server, apt-get remove mariadb-server, running puppet, debugging why it wont start [devtools]
2023-11-21 §
21:52 <mutante> - commit fake key for phabricator-bullseye host in git /var/lib/git/labs/private/modules/secret/secrets/ssl on puppetmaster-1001.devtools T327068 [devtools]
21:41 <mutante> - cert issue on new machine related to having local puppetmaster, like T349937#9288547 except "rm -rf /var/lib/puppet/ssl" was enough since puppetmaster did auto-sign new CSR - T327068 [devtools]
21:24 <mutante> - initial puppet run on newly created VM fails with "SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster-1001.devtools.eqiad.wmflabs" T327068 [devtools]
21:05 <mutante> - creating instance phabricator-bullseye g3.cores2.ram4.disk20 T327068 [devtools]
21:00 <mutante> - deleted instance phorge-1001 to get quota back and allow for creting new phabricator-on-bullseye instance T328595 T327068 [devtools]
2023-04-12 §
19:26 <mutante> - vrts-1001 - editing /etc/my.cnf to set mariadb datadir to /var/lib/mysql instead of /srv/sqldata and restart service, issue like T329571 [devtools]
2023-03-11 §
00:20 <mutante> - on phorge1001, enable general query log in mysql (mariadb), to learn about database scheme, don't forget to turn that off so VM doesn't run out of disk (SET GLOBAL general_log=1;) T328595 [devtools]
2023-03-07 §
23:34 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL max_allowed_packet=33554432; [devtools]
23:33 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL local_infile=0; [devtools]
23:31 <mutante> - phorge-1001 - MariaDB [(none)]> SET GLOBAL sql_mode = "STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"; [devtools]
2023-02-13 §
23:09 <mutante> - shutting down gerrit-prod-1001 [devtools]
22:22 <mutante> certbot renew --apache fixed cert issue - https://ldapauth-gitldap.wmflabs.org/ does not exist unrelatedly - T329444 [devtools]
22:18 <mutante> install package python3-certbot-apache on gerrit-prod-1001 - T329444 [devtools]
22:03 <mutante> - re-activating disabled puppet on gerrit-prod-1001 (reason given was 'gerrit deploy' but it was about 17 days ago) [devtools]
21:58 <mutante> rebooting instance gerrit-prod-1001 which can't be reached T329444 [devtools]
2023-01-31 §
22:39 <mutante> remove role::gitlab from gitlab-prod-1001. to be replaced with gitlab-prod-1002. T318521 [devtools]
2023-01-28 §
16:26 <taavi> adjust gitlab-prod-1002 network port settings to allow adding the secondary IP, requested in T318521 [devtools]
2023-01-24 §
09:47 <wm-bot2> Increased quotas by 1 instances (T327750) - cookbook ran by arturo@nostromo [devtools]
2022-11-30 §
15:57 <wm-bot2> Increased quotas by 1 floating-ips (T323986) - cookbook ran by dcaro@vulcanus [devtools]
2022-11-17 §
18:18 <andrewbogott> committed a local puppet change on puppetprimary to fix upstream syncs [devtools]
2022-10-28 §
20:24 <mutante> - removing from Horizon / project-wide hiera: profile::phabricator::main::manage_scap_user: true (set in the repo) [devtools]
20:23 <mutante> - removing from Horizon / project-wide hiera: profile::keyholder::server::require_encrypted_keys: 'no' (set in the repo) [devtools]
20:19 <mutante> - removing from Horizon / project-wide hiera: profile::gerrit::daemon_user: gerrit2, profile::gerrit::manage_scap_user: true, profile::gerrit::scap_user: gerrit-deploy (all of these are set in the repo) [devtools]
2022-10-19 §
19:21 <mutante> - on puppetmaster-1001.devtools created /var/lib/puppet/volatile/GeoIP directory - to fix puppet error on deploy-1004.devtools - reacting to puppet-broken-nagging-emails [devtools]
2022-06-24 §
12:43 <taavi> `os quota set devtools --ram 45056 --cores 22 --instances 9` # T311302 [devtools]
2022-06-15 §
20:30 <mutante> - created gitlab-runner-1002 - applied puppet role - attached cinder volume "docker" - running puppet again [devtools]
17:06 <mutante> deleting instance gitlab-runner-1001 which just disconnects people. gut feeling is it has to do with the fact that a previous instance name was used again [devtools]
2022-06-14 §
23:29 <mutante> - creating instance gitlab-runner-1001 since we did not have a test machine for gitlab-runners but need one to test things like gerrit:791655 before hitting prod T308271 [devtools]
2022-04-29 §
20:59 <mutante> - restarting instance gitlab-prod-1001 - No route to host [devtools]
20:55 <mutante> - attempting to soft reboot instance deploy1004 (got the puppet fail mail and wasnt reachable by ssh), this happened lately as well to gitlab-prod-1001, same project, different instance, but this time it doesn't just come back yet [devtools]
2022-04-20 §
17:03 <mutante> soft rebooting gitlab-prod-1001 which was sending "failed puppet" reports and was unreachable, just like the other day. [devtools]
2022-04-18 §
19:08 <mutante> - gitlab-prod-1001 is indeed back after soft rebooting the instance. uptime 1 min T297411 [devtools]