| 
      
        2019-04-01
      
      §
     | 
  
    
  | 10:32 | 
  <_joe_> | 
  pruning old images on boron | 
  [production] | 
            
  | 10:31 | 
  <oblivian@deploy1001> | 
  Finished deploy [docker-pkg/deploy@7ef5ca3]: Upgrade to 1.1.2 (duration: 00m 26s) | 
  [production] | 
            
  | 10:31 | 
  <oblivian@deploy1001> | 
  Started deploy [docker-pkg/deploy@7ef5ca3]: Upgrade to 1.1.2 | 
  [production] | 
            
  | 10:27 | 
  <arturo> | 
  T219626 reimaging cloudcontrol2001-dev | 
  [production] | 
            
  | 09:09 | 
  <moritzm> | 
  installing Chromium security updates on proton* (tested the new release in deployment-prep) | 
  [production] | 
            
  | 08:40 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Repool db2033 (duration: 00m 51s) | 
  [production] | 
            
  | 08:09 | 
  <marostegui> | 
  Deploy testing schema change on enwiki.echo_event on db2033 and upgrade mysql - T143961 | 
  [production] | 
            
  | 07:54 | 
  <ariel@deploy1001> | 
  Finished deploy [dumps/dumps@7abb6c8]: get db user/passwd va mw maint script (duration: 00m 03s) | 
  [production] | 
            
  | 07:54 | 
  <ariel@deploy1001> | 
  Started deploy [dumps/dumps@7abb6c8]: get db user/passwd va mw maint script | 
  [production] | 
            
  | 07:30 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Depool db2033 (duration: 00m 51s) | 
  [production] | 
            
  | 06:28 | 
  <_joe_> | 
  pushing wikimedia-jessie:{20190401,latest} to docker-registry.w.o T219580 | 
  [production] | 
            
  | 06:27 | 
  <_joe_> | 
  installing new bootstrap-vz on boron T219580 | 
  [production] | 
            
  | 05:38 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1077 (duration: 00m 50s) | 
  [production] | 
            
  | 05:08 | 
  <marostegui> | 
  Deploy schema change on db1077, this will generate lag on s3 on labs | 
  [production] | 
            
  | 05:08 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1077 (duration: 00m 53s) | 
  [production] | 
            
  
    | 
      
        2019-03-29
      
      §
     | 
  
    
  | 22:06 | 
  <bstorm_> | 
  stopped database services on labsdb1004 and labsdb1005 | 
  [production] | 
            
  | 21:01 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 3) (duration: 05m 14s) | 
  [production] | 
            
  | 20:55 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 3) | 
  [production] | 
            
  | 20:49 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 3) (duration: 03m 13s) | 
  [production] | 
            
  | 20:46 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 3) | 
  [production] | 
            
  | 20:35 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 2) (duration: 03m 30s) | 
  [production] | 
            
  | 20:31 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (part 2) | 
  [production] | 
            
  | 20:30 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers (duration: 00m 30s) | 
  [production] | 
            
  | 20:29 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@ff9d424]: Elasticsearch 6 fixes for content-type headers | 
  [production] | 
            
  | 18:41 | 
  <ejegg> | 
  updated payments-wiki from 4b49bb7333 to 793bce1a5f | 
  [production] | 
            
  | 15:51 | 
  <XioNoX> | 
  repool ulsfo - T219591 | 
  [production] | 
            
  | 15:48 | 
  <XioNoX> | 
  bump ulsfo-codfw ospf link cost to 1000 - T219591 | 
  [production] | 
            
  | 15:14 | 
  <_joe_> | 
  pruning old images and containers on boron | 
  [production] | 
            
  | 15:00 | 
  <mutante> | 
  ldap-eqiad-replica02 - running out of disk - apt-get clean - gzipping /var/log/debug | 
  [production] | 
            
  | 13:06 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=varnish-fe | 
  [production] | 
            
  | 13:06 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=cp2005.codfw.wmnet,service=nginx | 
  [production] | 
            
  | 13:06 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=cp2002.codfw.wmnet,service=varnish-fe | 
  [production] | 
            
  | 13:06 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=cp2002.codfw.wmnet,service=nginx | 
  [production] | 
            
  | 13:05 | 
  <ema> | 
  cp2002/cp2005: repool varnish-fe for user traffic T213263 | 
  [production] | 
            
  | 12:55 | 
  <thcipriani> | 
  gerrit running on 2.15.11 | 
  [production] | 
            
  | 12:53 | 
  <thcipriani> | 
  restarting gerrit to finish rollback to 2.15.11 | 
  [production] | 
            
  | 12:52 | 
  <thcipriani@deploy1001> | 
  Finished deploy [gerrit/gerrit@670ddb8]: Gerrit (back) to version 2.15.11 on cobalt -- restart of gerrit incoming (duration: 00m 11s) | 
  [production] | 
            
  | 12:52 | 
  <thcipriani@deploy1001> | 
  Started deploy [gerrit/gerrit@670ddb8]: Gerrit (back) to version 2.15.11 on cobalt -- restart of gerrit incoming | 
  [production] | 
            
  | 12:51 | 
  <moritzm> | 
  removing php 7.0 packages from snapshot1008, dumps are only using 7.2 (T218193) | 
  [production] | 
            
  | 12:50 | 
  <thcipriani@deploy1001> | 
  Finished deploy [gerrit/gerrit@670ddb8]: Gerrit (back) to version 2.15.11 (gerrit2001 only) (duration: 00m 10s) | 
  [production] | 
            
  | 12:50 | 
  <thcipriani@deploy1001> | 
  Started deploy [gerrit/gerrit@670ddb8]: Gerrit (back) to version 2.15.11 (gerrit2001 only) | 
  [production] | 
            
  | 12:46 | 
  <moritzm> | 
  upgrading snapshot1008 to component/php72 (T218193) | 
  [production] | 
            
  | 12:46 | 
  <moritzm> | 
  upgrading snapshot1005-1007/1009 to component/php72 (T218193) | 
  [production] | 
            
  | 12:23 | 
  <ema> | 
  rolling ATS restarts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500011/ T213263 | 
  [production] | 
            
  | 11:45 | 
  <mutante> | 
  cobalt - systemctl restart gerrit | 
  [production] | 
            
  | 10:36 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: name=cp2002.codfw.wmnet,service=varnish-fe | 
  [production] | 
            
  | 10:36 | 
  <ema@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: name=cp2002.codfw.wmnet,service=nginx | 
  [production] |