| 2019-03-15
      
      § | 
    
  | 09:53 | <akosiaris@deploy1001> | scap-helm zotero finished | [production] | 
            
  | 09:53 | <akosiaris@deploy1001> | scap-helm zotero cluster codfw completed | [production] | 
            
  | 09:52 | <akosiaris@deploy1001> | scap-helm zotero upgrade -f zotero-values-codfw.yaml production stable/zotero [namespace: zotero, clusters: codfw] | [production] | 
            
  | 09:42 | <godog> | bounce grafana-server on grafana1001 | [production] | 
            
  | 09:32 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1103 (duration: 00m 50s) | [production] | 
            
  | 09:28 | <godog> | correction, prometheus2004 | [production] | 
            
  | 09:27 | <godog> | temporarily disable read queries to prometheus@k8s on prometheus2003 | [production] | 
            
  | 09:19 | <jiji@cumin1001> | conftool action : set/weight=12; selector: dc=eqiad,service=cxserver,cluster=scb,name=kubernetes.* | [production] | 
            
  | 09:18 | <jiji@cumin1001> | conftool action : set/weight=15; selector: dc=codfw,service=cxserver,cluster=scb,name=kubernetes.* | [production] | 
            
  | 09:17 | <jijiki> | Ramp up  cxserver k8s traffic to 50% - T213195 | [production] | 
            
  | 08:57 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1103 (duration: 00m 50s) | [production] | 
            
  | 08:52 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 47s) | [production] | 
            
  | 08:22 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s) | [production] | 
            
  | 07:28 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Fully repool db1091 (duration: 00m 49s) | [production] | 
            
  | 07:17 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) | [production] | 
            
  | 07:05 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) | [production] | 
            
  | 07:01 | <kart_> | Finished manual run of unpublished ContentTranslation draft purge script (T217818) | [production] | 
            
  | 06:55 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) | [production] | 
            
  | 06:44 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 (duration: 00m 48s) | [production] | 
            
  | 06:04 | <marostegui> | Upgrade db1091 | [production] | 
            
  | 06:04 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 50s) | [production] | 
            
  | 04:01 | <kart_> | Started manual run of unpublished ContentTranslation draft purge script (T217818) | [production] | 
            
  | 01:25 | <ejegg> | re-enabled ingenico audit parser | [production] | 
            
  | 01:25 | <ejegg> | updated fundraising CiviCRM from 41efa14fb0 to a2316be94f | [production] | 
            
  
    | 2019-03-14
      
      § | 
    
  | 22:54 | <ejegg> | temporarily disabled Ingenico WX audit parsing | [production] | 
            
  | 22:05 | <cdanis> | cdanis@icinga2001.wikimedia.org ~ % sudo systemctl restart icinga.service | [production] | 
            
  | 21:58 | <cdanis> | cdanis@icinga2001.wikimedia.org ~ % sudo systemctl restart nsca.service | [production] | 
            
  | 21:01 | <crusnov@deploy1001> | Finished deploy [netbox/deploy@090a0c3]: Another minor bugfix releaes for ganeti-netbox script (duration: 00m 56s) | [production] | 
            
  | 21:00 | <crusnov@deploy1001> | Started deploy [netbox/deploy@090a0c3]: Another minor bugfix releaes for ganeti-netbox script | [production] | 
            
  | 20:26 | <thcipriani> | gerrit live on 2.15.11 | [production] | 
            
  | 20:24 | <thcipriani> | restarting gerrit for 2.15.11 | [production] | 
            
  | 20:23 | <thcipriani@deploy1001> | Finished deploy [gerrit/gerrit@2bc8af0]: Gerrit to 2.15.11 on cobalt (duration: 00m 02s) | [production] | 
            
  | 20:23 | <thcipriani@deploy1001> | Started deploy [gerrit/gerrit@2bc8af0]: Gerrit to 2.15.11 on cobalt | [production] | 
            
  | 20:22 | <thcipriani@deploy1001> | Finished deploy [gerrit/gerrit@2bc8af0]: Gerrit to 2.15.11 on gerrit2001 only (duration: 00m 04s) | [production] | 
            
  | 20:22 | <thcipriani@deploy1001> | Started deploy [gerrit/gerrit@2bc8af0]: Gerrit to 2.15.11 on gerrit2001 only | [production] | 
            
  | 20:17 | <ejegg> | updated CiviCRM from b4e3cf16cc to 41efa14fb0 | [production] | 
            
  | 20:17 | <thcipriani> | gerrit back to 2.15.8 | [production] | 
            
  | 20:15 | <thcipriani> | restart gerrit on cobalt | [production] | 
            
  | 20:14 | <thcipriani@deploy1001> | Finished deploy [gerrit/gerrit@2bc8af0]: Revert Gerrit to 2.15.11 on cobalt (duration: 00m 07s) | [production] | 
            
  | 20:14 | <thcipriani@deploy1001> | Started deploy [gerrit/gerrit@2bc8af0]: Revert Gerrit to 2.15.11 on cobalt | [production] | 
            
  | 20:14 | <thcipriani@deploy1001> | Finished deploy [gerrit/gerrit@2bc8af0]: Revert Gerrit to 2.15.11 on gerrit2001 only (duration: 00m 10s) | [production] | 
            
  | 20:13 | <thcipriani@deploy1001> | Started deploy [gerrit/gerrit@2bc8af0]: Revert Gerrit to 2.15.11 on gerrit2001 only | [production] | 
            
  | 20:13 | <bstorm_> | Placed labstore1006 back in rotation for NFS and rsync | [production] | 
            
  | 20:11 | <crusnov@deploy1001> | Finished deploy [netbox/deploy@c6cf7d6]: Minor bugfix releaes for ganeti-netbox script (duration: 00m 54s) | [production] | 
            
  | 20:10 | <crusnov@deploy1001> | Started deploy [netbox/deploy@c6cf7d6]: Minor bugfix releaes for ganeti-netbox script | [production] | 
            
  | 20:03 | <jforrester@deploy1001> | Synchronized php-1.33.0-wmf.21/extensions/GrowthExperiments/extension.json: Hot-deploy I19414dc31 to fix dependencies on mw.Uri (duration: 00m 49s) | [production] | 
            
  | 19:37 | <XioNoX> | set protocols bgp group Anycast4 multihop ttl 193 on cr1/2-esams - T209989 | [production] | 
            
  | 19:25 | <XioNoX> | merged Juniper BFD Icinga check | [production] | 
            
  | 19:12 | <thcipriani> | gerrit back up | [production] | 
            
  | 19:08 | <thcipriani> | restarting gerrit on cobalt for 2.15.11 upgrade | [production] |