42 results (16ms)
2021-07-28 §
16:39 <andrewbogott> rebooting gerrit-prod-1001; seemingly unreachable [devtools]
2021-03-10 §
10:58 <arturo> briefly stopped VM 'doc' to disable VMX cpu flag and live-migrate it [devtools]
2021-02-22 §
20:58 <mutante> fixed puppet run on deploy-1002 by adding empty array of wikimedia-sites to hiera [devtools]
20:01 <mutante> deploy-1002 is broken because mediawiki::sites is not in Hiera (yet) [devtools]
2020-10-28 §
17:01 <andrewbogott> fixed puppet runs on phabricator-stage-1001 (previously puppetmaster name mismatch) [devtools]
2020-09-01 §
00:16 <mutante> - unbreaking puppet run on the local deployment after it was broken since July due to changes in prod deployment_server role [devtools]
2020-06-30 §
20:22 <mutante> managed to let certbot get LE certs for gerrit.devtools.wmflabs.org and the floating IP [devtools]
2020-06-17 §
19:55 <paladox> ran `iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT` on phabricator-prod-1001 [devtools]
2020-05-08 §
07:01 <mutante> phabricator-prod-1001 - removing cron for public task dump (though puppet should have removed it) [devtools]
2020-05-07 §
09:24 <mutante> - cloud puppetmasters still affected by https://phabricator.wikimedia.org/T83447#5807825 [devtools]
09:07 <mutante> - puppetmaster-1001 - Permission denied @ rb_sysopen - /var/lib/puppet/volatile/GeoIP/.geoipupdate.lock [devtools]
09:06 <mutante> - avoiding the need for a second role for deployment_servers in cloud with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594903 [devtools]
09:05 <mutante> - puppet fixed on deploy-1002 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594900 [devtools]
08:04 <mutante> - broken puppet again from prod changes. this time: deploy-1002 - []' is not applicable to an Undef Value. mediawiki/mcrouter_wancache.pp, line: 19 [devtools]
2020-04-13 §
10:00 <mutante> - phabricator-stage-1001: replace deployment-tin.deployment-rep with deploy-1002.devtools in deployment-cache/.config [devtools]
09:40 <mutante> set missing (and new) profile::tlsproxy::envoy::capitalize_headers: true to fix puppet errors [devtools]
09:35 <mutante> set phabricator::vcs::address::v6 to fe80 local address to fix puppet error on phabricator-stage-1001 [devtools]
2020-01-16 §
00:53 <mutante> deploy-1002 - become 'trebuchet' user and ssh to phabricator scap targets. to fix ssh host key verification issue on first deploy [devtools]
00:30 <mutante> deploy-1002 live hack /srv/deployment/phabricator/deployment/scap/phabricator-targets and replace prod server with cloud instances; scap deploy in phabricator repo [devtools]
2020-01-15 §
23:51 <paladox> deploy-1002 rm -rf /srv/deployment [devtools]
23:44 <mutante> deploy-1002 sudo git init in /srv/deployment ; scap deploy --init (now fails with 'fatal: Not a valid object name HEAD') [devtools]
23:42 <mutante> deploy-1002 mkdir /srv/deployments/.git ; chown trebuchet:wikidev .git ; manually run "scap deploy --init" as trebuchet user in an attempt to fix initial puppet run on deployment_server [devtools]
2020-01-14 §
00:59 <mutante> deleting instance codesearch-buster [devtools]
00:54 <mutante> - deleting instance codesearch-stretch, creating codesearch-buster [devtools]
2020-01-11 §
00:35 <mutante> deleting instance codesearch-buster, creating codesearch-stretch [devtools]
00:05 <mutante> s/cloudsearch/codesearch/g [devtools]
00:04 <mutante> creating throwaway instance "cloudsearch" [devtools]
00:04 <mutante> deleting instance deploy1001 (buster), creating deploy-1002 (stretch) instead [devtools]
2020-01-04 §
16:01 <bstorm_> moving vm puppetmaster-1001 from cloudvirt1024 to cloudvirt1009 due to hardware error T241884 [devtools]
2020-01-03 §
22:37 <mutante> - sudo vi /srv/deployment/phabricator/deployment-cache/.config on both phabricator instances to fix deployment server (remove deployment-tin (!)) [devtools]
21:50 <mutante> assigned 172.16.0.198/32 on eth0 on phabricator-prod-1001 [devtools]
21:50 <jeh> add secondary interface to phabricator-prod-1001 [devtools]
21:32 <mutante> configure 172.16.0.189 as "vcs" address v4 for phabricator-stage-1001 [devtools]
21:24 <jeh> add secondary interface to phabricator-stage-1001 [devtools]
00:47 <paladox> puppet cert generate puppetmaster-1001.devtools.eqiad.wmflabs [devtools]
00:30 <paladox> set puppetmaster: puppetmaster-1001.devtools.eqiad.wmflabs in hiera [devtools]
2020-01-02 §
23:42 <paladox> puppetmaster-1001: ln -s /var/lib/puppet/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem [devtools]
23:40 <paladox> puppetmaster-1001: ln -s /var/lib/puppet/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem [devtools]
23:35 <mutante> attempting to create local puppetmaster (formerly puppetmaster::self) [devtools]
2019-12-23 §
04:14 <mutante> - turns out instance creation doesn't work using Brave browser but does work using Firefox (T241345) - created phabricator-prod-1001 [devtools]
03:45 <mutante> - creating new instance seems to fail - nothing shows up at all [devtools]
03:42 <mutante> - launching medium sized buster instance gerrit-prod-01 (T236309) [devtools]