| 
      
        2019-06-17
      
      ยง
     | 
  
    
  | 15:30 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main finished | 
  [production] | 
            
  | 15:30 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main cluster staging completed | 
  [production] | 
            
  | 15:30 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: staging] | 
  [production] | 
            
  | 15:17 | 
  <thcipriani> | 
  gerrit back | 
  [production] | 
            
  | 15:16 | 
  <thcipriani> | 
  gerrit restart to pick up new config changes. | 
  [production] | 
            
  | 14:45 | 
  <elukey> | 
  stop eventlogging on eventlog1002 and reboot for kernel upgrades | 
  [production] | 
            
  | 14:32 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main finished | 
  [production] | 
            
  | 14:32 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main cluster codfw completed | 
  [production] | 
            
  | 14:32 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/codfw-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: codfw] | 
  [production] | 
            
  | 14:26 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main finished | 
  [production] | 
            
  | 14:26 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main cluster staging completed | 
  [production] | 
            
  | 14:26 | 
  <otto@deploy1001> | 
  scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: staging] | 
  [production] | 
            
  | 14:15 | 
  <moritzm> | 
  installing poppler security updates on jessie | 
  [production] | 
            
  | 14:03 | 
  <cdanis> | 
  cdanis@cobalt.wikimedia.org ~ % sudo systemctl start gerrit.service | 
  [production] | 
            
  | 13:53 | 
  <ema@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) | 
  [production] | 
            
  | 13:49 | 
  <moritzm> | 
  installing libav security updates | 
  [production] | 
            
  | 13:45 | 
  <ema@cumin1001> | 
  START - Cookbook sre.hosts.upgrade-and-reboot | 
  [production] | 
            
  | 13:45 | 
  <ema> | 
  reboot cp4027 for dist and Varnish upgrade T224694 | 
  [production] | 
            
  | 13:34 | 
  <elukey> | 
  reboot of an-worker* (Hadoop worker nodes) for kernel + openjdk upgrades | 
  [production] | 
            
  | 13:25 | 
  <ema> | 
  cp4027: upgrade Varnish packages to 5.1.3-1wm10 T224694 | 
  [production] | 
            
  | 12:37 | 
  <jbond42> | 
  upgrade mtail on lithium - T225604 | 
  [production] | 
            
  | 12:35 | 
  <jbond42> | 
  add mtail_3.0.0~rc24.1-1+wmf1_amd64.deb to jessie-wikimedia backports | 
  [production] | 
            
  | 12:13 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/CommonSettings-labs.php: SWAT: [[gerrit:514715|FileImporter configuration to fetch sitelinks from Wikidata (T225609 T224007)]] - finishing partial deployment (duration: 00m 47s) | 
  [production] | 
            
  | 12:06 | 
  <awight> | 
  EU SWAT complete | 
  [production] | 
            
  | 12:05 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|517391 Enable AMC mode for Persian, Japanese, Thai and Italian wikis (T225123)]] (duration: 00m 47s) | 
  [production] | 
            
  | 12:02 | 
  <Urbanecm> | 
  EU SWAT is going a few minutes beyond its window | 
  [production] | 
            
  | 11:55 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|516608 Enable feature flag for breaking Wikibase API change (T223303)]] (duration: 00m 47s) | 
  [production] | 
            
  | 11:49 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|516478 Set EntityUsageTable addUsage batch size to 200 (T225500)]] (duration: 00m 47s) | 
  [production] | 
            
  | 11:46 | 
  <awight@deploy1001> | 
  Synchronized php-1.34.0-wmf.8/extensions/ContentTranslation: SWAT: [[gerrit:514734|Fix undefined index notices (T225198)]] (duration: 00m 49s) | 
  [production] | 
            
  | 11:33 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:517371|Add autoreview protection level on ar.wikipedia (T225896)]] (duration: 00m 47s) | 
  [production] | 
            
  | 11:28 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:512220|Enable VisualEditor in draft namespace on sr.wiki (T223024)]] (duration: 00m 47s) | 
  [production] | 
            
  | 11:23 | 
  <awight> | 
  ran mwscript namespaceDupes.php nds_nlwiki, no dupes found | 
  [production] | 
            
  | 11:22 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:517372|Set nds_nlwiki's sitename and metanamespace back to defaults (T224349)]] (duration: 00m 47s) | 
  [production] | 
            
  | 11:11 | 
  <awight@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: wmf-config/CommonSettings-labs.php SWAT: [[gerrit:514715|FileImporter configuration to fetch sitelinks from Wikidata (T225609 T224007)]] (duration: 00m 47s) | 
  [production] | 
            
  | 10:51 | 
  <jdrewniak@deploy1001> | 
  Synchronized portals: Wikimedia Portals Update: [[gerrit:517404| Bumping portals to master (T128546)]] (duration: 00m 47s) | 
  [production] | 
            
  | 10:51 | 
  <jdrewniak@deploy1001> | 
  Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:517404| Bumping portals to master (T128546)]] (duration: 00m 49s) | 
  [production] | 
            
  | 09:39 | 
  <moritzm> | 
  rebooting mw2184, mw1265 for some tests | 
  [production] | 
            
  | 09:36 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 09:36 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 09:36 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 09:36 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 09:31 | 
  <elukey> | 
  set cpu governor to performance (was powersave) on analytics1070 (hadoop worker node) | 
  [production] | 
            
  | 09:17 | 
  <moritzm> | 
  rebooting sulfur for some tests | 
  [production] | 
            
  | 09:15 | 
  <_joe_> | 
  The governor was set to "powersave", not "ondemand" | 
  [production] | 
            
  | 09:13 | 
  <_joe_> | 
  setting cpufreq governor to "ondemand" on mw1348, T225713 | 
  [production] | 
            
  | 08:52 | 
  <onimisionipe> | 
  remove maps1001 from cassandra cluster - T224395 | 
  [production] | 
            
  | 07:25 | 
  <XioNoX> | 
  restart snmp daemon on mr1-eqsin | 
  [production] | 
            
  | 07:10 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Repool db2107 (duration: 00m 47s) | 
  [production] | 
            
  | 06:22 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Repool db2084 (duration: 00m 47s) | 
  [production] | 
            
  | 06:12 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-codfw.php: Depool db2084 for a reboot (duration: 00m 48s) | 
  [production] |