|
2016-09-21
§
|
| 16:01 |
<hashar> |
deployment-tin02 applied puppet classes beta::autoupdater, beta::deployaccess, role::deployment::server, role::labs::lvm::srv |
[releng] |
| 15:32 |
<hashar> |
spawned deployment-tin02 |
[releng] |
| 14:55 |
<hashar> |
removed the CI puppet class from deployment-sca01 and deployment-sca02 . Stopped services using /srv , unmounted /srv, removed it from /etc/fstab |
[releng] |
| 14:27 |
<hashar> |
deployment-sca01 and deployment-sca02 are now broken. The CI puppet class mount /srv which ends up being only 500 MBytes |
[releng] |
| 14:08 |
<hashar> |
deployment-mira adding puppet class beta::autoupdater |
[releng] |
| 14:06 |
<hashar> |
Enabling Jenkins slave deployment-mira |
[releng] |
| 14:05 |
<hashar> |
deployment-mira seems ready for action and is the primary deployment server. Enabling jenkins to it |
[releng] |
| 11:24 |
<hashar> |
removing Jenkins slave deployment-tin , deployment-mira is the new deployment master T144578 |
[releng] |
| 10:58 |
<hashar> |
Changing Jenkins slaves home dir for deployment-sca01 and deployment-sca02 from /mnt/home/jenkins-deploy to /srv/jenkins/home/jenkins-deploy |
[releng] |
| 10:57 |
<hashar> |
Changing Jenkins slaves home dir for deployment-tin and deployment-mira from /mnt/home/jenkins-deploy to /srv/jenkins/home/jenkins-deploy |
[releng] |
| 10:10 |
<hashar> |
deployment-mira removing "role::labs::lvm::srv" duplicate with role::ci::slave::labs::common |
[releng] |
| 10:07 |
<hashar> |
Making deployment-mira a Jenkins slave by applying puppet class role::ci::slave::labs::common T144578 |
[releng] |
| 10:05 |
<hashar> |
Arming keyholder on deployment-mira |
[releng] |
| 09:43 |
<hashar> |
beta: switching master deployment server from deployment-tin to deployment-mira |
[releng] |
| 09:34 |
<hashar> |
From [[Hiera:deployment-prep]] remove bit already in puppet: "scap::deployment_server": deployment-tin.deployment-prep.eqiad.wmflabs |
[releng] |
| 08:55 |
<moritzm> |
remove mira from deployment-prep (replaced by deployment-mira) |
[releng] |
| 08:37 |
<hashar> |
beta: manually rebased puppetmaster |
[releng] |
| 08:11 |
<elukey> |
terminated jobrunner01 and removed from deployment-prep's sacp dsh list |
[releng] |
| 07:19 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/311927 |
[releng] |
|
2016-09-20
§
|
| 21:49 |
<hashar> |
Deleting deployment-mira02 /srv was too small. Replaced by deployment-mira |
[releng] |
| 20:54 |
<hashar> |
from deployment-tin for T144578, accept ssh host key of deployment-mira : sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mira.deployment-prep.eqiad.wmflabs |
[releng] |
| 20:47 |
<hashar> |
Creating deployment-mira instance with flavor c8.m8.s60 (8 cpu, 8G RAM and 60G disk) T144578 |
[releng] |
| 20:16 |
<Krenair> |
enabled trusty-backports on deployment-puppetmaster |
[releng] |
| 19:00 |
<thcipriani> |
cherry-picked https://gerrit.wikimedia.org/r/#/c/311760/ to deployment-puppetmaster to fix failing beta-scap-eqiad job, had to manually start rsync, puppet failed to start |
[releng] |
| 18:38 |
<hashar> |
on tin: `sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mira02.deployment-prep.eqiad.wmflabs` - T144006 |
[releng] |
| 18:33 |
<hashar> |
on deployment-mira02 ran `sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs` per T144006 |
[releng] |
| 18:01 |
<marxarelli> |
deployed mediawiki-config changes on beta cluster. back in read/write mode using new database instances |
[releng] |
| 17:37 |
<marxarelli> |
deployment-db04 restored from backup and replication started |
[releng] |
| 16:54 |
<marxarelli> |
upgraded package and data to mariadb 10 on deployment-db03 |
[releng] |
| 16:31 |
<marxarelli> |
cherry picking operations/puppet patches (T138778) to deployment-puppetmaster |
[releng] |
| 16:30 |
<moritzm> |
rebooting deployment-mira02 |
[releng] |
| 16:23 |
<marxarelli> |
applied innodb transaction logs to deployment-db1 backup and successfully restored on deployment-db03 |
[releng] |
| 15:47 |
<marxarelli> |
completed innobackupex on deployment-db1. copying backup to deployment-db03 for restoration |
[releng] |
| 14:54 |
<hashar> |
beta: cherry picking fix up for the jobrunner logging https://gerrit.wikimedia.org/r/#/c/311702/ and https://gerrit.wikimedia.org/r/311719 T146040 |
[releng] |
| 14:44 |
<marxarelli> |
entering read-only mode on beta cluster |
[releng] |
| 14:27 |
<elukey> |
stopped puppet, jobrunner and jobchron on deployment-jobrunner01 |
[releng] |
| 14:20 |
<marxarelli> |
disabling beta cluster jenkins jobs in preparation for data migration (T138778) |
[releng] |
| 13:07 |
<godog> |
add deployment-prometheus01 instance T53497 |
[releng] |
| 11:20 |
<elukey> |
applied beta::deployaccess, role::labs::lvm::srv, role::mediawiki::jobrunner to jobrunner02 |
[releng] |
| 10:45 |
<elukey> |
created deployment-jobrunner02 in deployment-prep |
[releng] |
|
2016-09-19
§
|
| 22:01 |
<legoktm> |
shutdown integration-puppetmaster |
[releng] |
| 21:29 |
<yuvipanda> |
regenerated client certs only on integration-puppetmaster01, seems ok now |
[releng] |
| 20:46 |
<yuvipanda> |
re-enable puppet everywhere |
[releng] |
| 20:43 |
<yuvipanda> |
enable puppet and run on integration-slave-trusty-1003.eqiad.wmflabs |
[releng] |
| 20:42 |
<yuvipanda> |
accidentally deleted /var/lib/puppet/ssl on integration-puppetmaster01 as well, causing it to lose keys. Reprovision by pointing to labs puppetmaster |
[releng] |
| 20:34 |
<yuvipanda> |
rm -rf /var/lib/puppet/ssl on all integration nodes |
[releng] |
| 20:34 |
<yuvipanda> |
copied /etc/puppet/puppet.conf from integration-trusty-slave-1001 to all integration |
[releng] |
| 20:25 |
<yuvipanda> |
delete /etc/puppet/puppet.conf.d/10-self.conf and /var/lib/puppet/ssl on integration-slave-trusty-1001 |
[releng] |
| 20:20 |
<yuvipanda> |
re-enabled puppet on integration-slave-trusty-1001 |
[releng] |
| 20:08 |
<yuvipanda> |
reset puppetmaster of integration-puppetmaster01 to be labs puppetmaster |
[releng] |