1-50 of 158 results (29ms)
2024-08-01 §
19:18 <mutante> gitlab-prod-1002 - moved contents of /var/log/gitlab out of the way temp, emptied it, letting wmcs-prepare-cinder-volume mount new volume into /var/log/gitlab and format it with ext4, moved old existing logs back into it T371066 [devtools]
19:11 <mutante> attaching volume 'gitlab-prod-logs' to instance 'gitlab-prod-1002' T371066 | running 'sudo wmcs-prepare-cinder-volume' manually and answering interactive questions to mount/format it (once T371573 is resolved will do this with puppet) [devtools]
19:07 <mutante> creating new 20GB volume 'gitlab-prod-logs' T371066 [devtools]
09:21 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [devtools]
09:21 <sstefanova@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase [devtools]
2024-07-24 §
22:42 <mutante> gitlab-prod-1002 out of disk again when attempting package upgrade - apt-get clean; rm /var/log/syslog.2*.gz rm /var/log/messages.2*.gz to get space [devtools]
2024-07-10 §
21:46 <mutante> gitlab-runner* instances: apt-get upgrade (upgrading gitlab-runner, exim*. downgrading lshw, pythong3-jinja2) [devtools]
21:34 <mutante> gitlab-prod-1002 apt-get upgrade (upgrading gitlab-ce, exim-*. downgrading lshw, python3-jinja2) [devtools]
2024-06-18 §
10:31 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [devtools]
10:18 <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.migrate_project_to_ovs [devtools]
2024-05-06 §
20:37 <mutante> deleting buster deployment server deploy-1004, replaced by deploy-1006 - T360964 [devtools]
2024-05-02 §
23:52 <mutante> switching puppetmaster for deploy-1006 back to local project puppetmaster; rm -rf /var/lib/puppet/ssl that still referred to puppetmaster-1001, signing new request on puppetmaster-1003 T360470 T363415 [devtools]
23:10 <mutante> replacing deploy-1004 (buster) with deploy-1006 (bullseye) as new deployment server in both repo and Horizon hiera T360964 T363415 [devtools]
2024-05-01 §
23:59 <mutante> creating instance gerrit-bullseye (T363196) [devtools]
22:45 <mutante> rebooting deploy-1006 to see if issue with starting mw-cgroup goes away - .. and it did! - it must have been the grub config from https://gerrit.wikimedia.org/r/c/operations/puppet/+/991347 but needed one reboot - T363957 [devtools]
2024-04-24 §
20:28 <mutante> changing puppetmaster of deploy-1006 to puppetmaster.cloudinfra.wmflabs.org instead of the local project one [devtools]
20:12 <mutante> - puppet run on deploy-1006 deployment_server - scap init duplicate declaration fixed after adding "profile::mediawiki::scap_client::is_master: true" to new deploy-prefix Hiera - thanks to Jaime Nuche per comment on gerrit [devtools]
19:46 <mutante> creating new prefix 'deploy' to apply needed Hiera keys to deployment hosts based on host name prefix (both deploy1004, deploy1006 and future deploy*) [devtools]
19:41 <mutante> deleting instance puppetmaster-1001 that was > 4 years old, on buster and I had shutdown a couple days ago. replaced by puppetmaster-1003 (bookworm, puppetserver) T360964 T360470 [devtools]
19:21 <mutante> gitlab-prod-1002; gitlab-runner-1003; gitlab-runner-1002 - apt-get update && apt-get upgrade [devtools]
2024-04-17 §
21:18 <mutante> - resizing puppetmaster-1003 from g3.cores1.ram2.disk20 to g3.cores2.ram4.disk20 - T360470 [devtools]
2024-04-16 §
17:09 <mutante> - deleting devtools-puppetdb1001 instance (T360964) [devtools]
16:53 <mutante> - soft rebooting puppetmaster-1003, shutting down puppetmaster-1001 [devtools]
16:38 <mutante> - can't ssh to new puppetmaster again [devtools]
16:33 <mutante> - deleting deploy-1005 - don't try deployment server in bookworm, first bullseye [devtools]
16:29 <mutante> - shutting down puppetdb instance again [devtools]
2024-04-15 §
17:52 <mutante> - added profile::labs::cindermount::srv to puppetmaster-1003 in horizon to get missing cinder volume - T360470 [devtools]
17:51 <mutante> - added Notice: /Stage[main]/Profile::Labs::Cindermount::Srv/Cinderutils::Ensure[cinder_on_srv]/Exec[prepare_cinder_volume_/srv]/returns: executed successfully [devtools]
16:57 <mutante> - puppetmaster-1003 reachable again but service fails to start and puppetserver-deploy-code fails [devtools]
16:50 <mutante> - rebooted unreachable puppetmaster-1003 - was "no route to host" - but is back now, log had a " /dev/sdb: Can't open blockdev" as well [devtools]
2024-04-12 §
17:55 <mutante> - changed both puppetmaster and puppetdb hiera setting back to puppetmaster-1001 for instance deploy-1004 [devtools]
17:49 <mutante> - deploy-1004 itself is on buster and buster and puppet 7 don't mix well - testing deployment role on bookworm, creating deploy-1005 [devtools]
17:48 <mutante> - deploy-1004 has a puppet problem when talking to new puppetmaster-1003 that goes away when switching back to puppetmaster-1001 [devtools]
2024-04-11 §
20:08 <mutante> zuul-1001 - switching to new puppetmaster-1003 in puppet.conf manually, switched project defaults in repo too [devtools]
19:58 <mutante> manually editing puppet.conf to use puppetmaster-1003 instead of -1001 because you can't switch the puppetmaster via puppet if puppet is already broken :) [devtools]
19:41 <mutante> switching gitlab-runner-1005 from puppetmaster-1001 to puppetmaster-1003 via web Hiera [devtools]
19:03 <mutante> - deleting instance contint-bullseye which was only used by me for a test before we created contint1003 in prod T334517 T361224 [devtools]
18:38 <mutante> - attempting to fix puppet run on vrts-1001 related to switching prod to cfssl for SSL cers [devtools]
18:23 <mutante> - shutting down puppetmaster-1001 on buster - should now be replaced by puppetmaster-1003 on bookworm (thanks brennen) T360964 T360470 [devtools]
18:01 <mutante> - shutting down instance devtools-puppetdb1001 - which is on buster - basically to see what breaks of complains, if anything [devtools]
2024-04-09 §
19:46 <mutante> - soft rebooting gerrit-prod-1001 buster instance (to be removed ) [devtools]
2024-04-01 §
17:27 <mutante> - added profile::pki::client::ensure: present to instance hiera for etherpad-bookworm - fixing broken puppet run [devtools]
17:25 <mutante> - attempting to fix puppet on instance etherpad-bookworm but SSL provider cfssl doesn't appear to work in cloud [devtools]
2024-02-09 §
19:08 <mutante> deleting instance phabricator-prod-1001 (shut down a couple days ago, buster instance replaced by phabricator-bullseye instance) T356530 [devtools]
2024-02-07 §
21:51 <mutante> rebooting gitlab-prod-1002 T356906 [devtools]
2024-02-02 §
23:37 <mutante> phabricator-bullseye configured auth provider for simple passwords, letting users register users, locked auth config option again [devtools]
23:34 <mutante> phabricator-bullseye:/srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./auth unlock [devtools]
22:53 <mutante> - phabricator-bullseye sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' | sudo ./config set security.alternate-file-domain 'https://phab-usercontent.wmcloud.org' | delete proxy phab-prod-usercontent, create proxy phab-usercontent.wmcloud.org, restart apache T356530 [devtools]
22:37 <mutante> - phabricator-bullseye - /srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' T356530 [devtools]
22:31 <mutante> - phabricator-bullseye, instance hiera: setting phabricator_domain to phabricator.wmcloud.org and phabricator_altdomain to phab-usercontent.wmcloud.org T356530 [devtools]