| 2021-03-11
      
      § | 
    
  | 22:11 | <marxarelli> | reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job(T277236) | [releng] | 
            
  | 22:10 | <marxarelli> | running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236) | [releng] | 
            
  | 22:03 | <marxarelli> | running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236) | [releng] | 
            
  | 16:57 | <Majavah> | copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 T276419 | [releng] | 
            
  | 16:49 | <Majavah> | delete deployment-etcd-01 T276462 | [releng] | 
            
  | 13:51 | <Majavah> | shut down deployment-db06, now unused T277070 | [releng] | 
            
  | 13:48 | <Majavah> | set deployment-db07 as r/w T277070 | [releng] | 
            
  | 13:48 | <Majavah> | stop mariadb to ensure reads have stopped on deployment-db06 T277070 | [releng] | 
            
  | 13:41 | <Majavah> | stop slave on deployment-db06 T276968 | [releng] | 
            
  | 13:37 | <Majavah> | make deployment-db06 and deployment-db08 be replicas of deployment-db07 T277070 | [releng] | 
            
  | 13:34 | <Majavah> | stop and reset slave on deployment-db07 T277070 | [releng] | 
            
  | 13:32 | <Majavah> | set deployment-db06 as read only T277070 | [releng] | 
            
  | 09:12 | <hashar> | Updated plugins on https://releases-jenkins.wikimedia.org/ | [releng] | 
            
  
    | 2021-03-10
      
      § | 
    
  | 19:36 | <Majavah> | shutdown deployment-ircd T277081 | [releng] | 
            
  | 18:46 | <Majavah> | switch floating ip 185.15.56.34 to deployment-ircd02 T277081 | [releng] | 
            
  | 18:05 | <Majavah> | create deployment-ircd02 for T277081 | [releng] | 
            
  | 17:26 | <marxarelli> | `rm -rf /srv/dump` on deployment-db06 and reenabling puppet | [releng] | 
            
  | 17:25 | <marxarelli> | `rm -rf /srv/restore` on deployment-db08 and reenabling puppet | [releng] | 
            
  | 17:24 | <marxarelli> | `rm -rf /srv/backup /srv/restore` on deployment-db07 and reenabling puppet | [releng] | 
            
  | 17:09 | <Majavah> | set beta cluster mediawiki as read write on mw config (T276968) | [releng] | 
            
  | 17:03 | <Majavah> | make deployment-db06 read-write T276968 | [releng] | 
            
  | 16:50 | <Majavah> | `reset slave;` on new master deployment-db06 T276968 | [releng] | 
            
  | 16:49 | <Majavah> | add deployment-db07 as a replica of db06 for T276968 | [releng] | 
            
  | 16:45 | <Urbanecm> | root@deployment-db07:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1 # T276968 | [releng] | 
            
  | 16:12 | <Majavah> | deployment-db08 CHANGE MASTER to MASTER_USER='repl', MASTER_PASSWORD='redacted', MASTER_PORT=3306, MASTER_HOST='deployment-db06.deployment-prep.eqiad1.wikimedia.cloud', MASTER_LOG_FILE='deployment-db06-bin.000059', MASTER_LOG_POS=522469730; (T276968) | [releng] | 
            
  | 16:06 | <Urbanecm> | start root@deployment-db07:/srv/sqldata.db06# rsync --progress -r deployment-db06:/srv/sqldata/ . (T276968) | [releng] | 
            
  | 15:57 | <Majavah> | set deployment-db06 as readonly from mysql side T276968 | [releng] | 
            
  | 15:54 | <Urbanecm> | Start `root@deployment-db08:/opt/wmf-mariadb104/bin# ./mysql_upgrade -h 127.0.0.1` (T276968) | [releng] | 
            
  | 15:54 | <Urbanecm> | Start mariadb on db08 (T276968) | [releng] | 
            
  | 15:22 | <Urbanecm> | rsync deployment-db06:/srv/sqldata to deployment-db08:/srv/sqldata in a tmux session on deploymdeployment-db08 (T276968) | [releng] | 
            
  | 14:52 | <Majavah> | delete deployment-db08 /srv/sqldata to attempt procedure in https://phabricator.wikimedia.org/T276968#6900199 | [releng] | 
            
  | 10:16 | <arturo> | briefly stopping deployment-puppetdb03 to disable VMX CPU flag | [releng] | 
            
  | 00:28 | <marxarelli> | mariadb successfully started on db07 following transfer/extraction using mariabackup and following mysql_upgrade (T276968) | [releng] | 
            
  | 00:10 | <marxarelli> | restore of db06 failed yet again. trying mariabackup db06 -> db07 instead of mysqldump (after fixing docs/usage of the former) (T276968) | [releng] |