| 
      
        2015-12-09
      
      §
     | 
  
    
  | 17:31 | 
  <akosiaris> | 
  restarted hhvm on mw1125 | 
  [production] | 
            
  | 17:17 | 
  <thcipriani@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/Wikidata: SWAT: Update Wikibase [[gerrit:257901]] (duration: 00m 49s) | 
  [production] | 
            
  | 17:05 | 
  <krenair@tin> | 
  Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 28s) | 
  [production] | 
            
  | 16:44 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Add rights to CU+OS groups on en.wikipedia.org [[gerrit:255129]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:32 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off language detection user test [[gerrit:254071]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:25 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Set initial titlesuggest shard sizes [[gerrit:256974]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:18 | 
  <thcipriani@tin> | 
  Synchronized dblists/group1.dblist: SWAT: Move Catalan and Hebrew Wikipedias to group1 [[gerrit:253371]] (duration: 00m 29s) | 
  [production] | 
            
  | 16:12 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable importupload and some import sources for officewiki [[gerrit:256881]] (duration: 00m 42s) | 
  [production] | 
            
  | 16:06 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part II [[gerrit:257800]] (duration: 00m 28s) | 
  [production] | 
            
  | 16:05 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: Add config for wgVisualEditorUseSingleEditTab and set true for enwiki Labs Part I [[gerrit:257800]] (duration: 00m 29s) | 
  [production] | 
            
  | 15:48 | 
  <akosiaris> | 
  uploaded to apt.wikimedia.org jessie-wikimedia: etherpad-lite_1.5.7-2 | 
  [production] | 
            
  | 15:01 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool es1019 after maintenance (duration: 00m 33s) | 
  [production] | 
            
  | 14:09 | 
  <godog> | 
  decommission restbase1007 | 
  [production] | 
            
  | 13:06 | 
  <hashar> | 
  restarting Jenkins (upgrading Gearman plugin) | 
  [production] | 
            
  | 11:51 | 
  <akosiaris> | 
  reissue certificates for labs-ldap.{eqiad,codfw}.wikimedia.org with a 2048bit key size | 
  [production] | 
            
  | 11:29 | 
  <kart_> | 
  Update cxserver to c88ef57 | 
  [production] | 
            
  | 10:45 | 
  <akosiaris> | 
  sudo gnt-instance modify -H cpu_type=SandyBridge seaborgium to get aesni and sse cpu flags. rebooting seaborgium to apply those | 
  [production] | 
            
  | 10:25 | 
  <akosiaris> | 
  reboot serpens to add 3 more vCPUs | 
  [production] | 
            
  | 10:21 | 
  <akosiaris> | 
  reboot seaborgium to add 3 more vCPUs | 
  [production] | 
            
  | 10:04 | 
  <moritzm> | 
  restarting slapd on serpens/seaborgium to apply updated indices | 
  [production] | 
            
  | 10:03 | 
  <ori> | 
  running hhvm-collect-heaps in screen on mw1001 with a 600s interval | 
  [production] | 
            
  | 10:00 | 
  <akosiaris> | 
  stopped slapd on seaborgium, run sudo -u openldap slapindex to add the new indices, started slapd | 
  [production] | 
            
  | 10:00 | 
  <paravoid> | 
  manually ran update-ubuntu-mirror twice on carbon to fix mirror errors | 
  [production] | 
            
  | 09:58 | 
  <akosiaris> | 
  stopped slapd on serpens, run sudo -u openldap slapindex to add the new indices, started slapd | 
  [production] | 
            
  | 08:15 | 
  <ori> | 
  Memory utilization on the job runners began to climb around 2015-12-08T10:00:00+00:00 UTC. To investigate, enabled jemalloc profiling and restarted HHVM on mw1001. | 
  [production] | 
            
  | 07:40 | 
  <YuviPanda> | 
  restarted create-dbusers on labstore1001 | 
  [production] | 
            
  | 04:44 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Wed Dec  9 04:44:57 UTC 2015 (duration 1h 50m 43s) | 
  [production] | 
            
  | 02:54 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.8) (duration: 04m 34s) | 
  [production] | 
            
  | 02:34 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.7) (duration: 10m 27s) | 
  [production] | 
            
  | 02:23 | 
  <ori@tin> | 
  Synchronized wmf-config/wikitech.php: I4ef826af47: Increase LDAP log verbosity (duration: 00m 44s) | 
  [production] | 
            
  | 02:16 | 
  <catrope@tin> | 
  Finished scap: Graph cherry-picks for wmf.8, including i18n changes (duration: 26m 38s) | 
  [production] | 
            
  | 01:49 | 
  <catrope@tin> | 
  Started scap: Graph cherry-picks for wmf.8, including i18n changes | 
  [production] | 
            
  | 01:45 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/Gadgets/: SWAT: bump MediaWikiGadgetsDefinitionRepo cache version (duration: 00m 29s) | 
  [production] | 
            
  
    | 
      
        2015-12-08
      
      §
     | 
  
    
  | 23:47 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: Train: group0 to 1.27.0-wmf.8 | 
  [production] | 
            
  | 23:43 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.8/extensions/MobileFrontend: I7b86a521: Ensure the parser cache varies on images disabled and 'light' images (duration: 00m 32s) | 
  [production] | 
            
  | 23:41 | 
  <thcipriani@tin> | 
  Finished scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II (duration: 56m 13s) | 
  [production] | 
            
  | 22:45 | 
  <thcipriani@tin> | 
  Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache take II | 
  [production] | 
            
  | 22:24 | 
  <thcipriani@tin> | 
  scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.d9wYd5Axk0" ' returned non-zero exit status 1 (duration: 01m 01s) | 
  [production] | 
            
  | 22:23 | 
  <thcipriani@tin> | 
  Started scap: testwiki to 1.27.0-wmf.8 and rebuild l10n cache | 
  [production] | 
            
  | 22:16 | 
  <yurik> | 
  deployed latest kartotherian & tilerator | 
  [production] | 
            
  | 21:09 | 
  <thcipriani> | 
  checking out wmf/1.27.0-wmf.8 on tin for train deploy | 
  [production] | 
            
  | 20:51 | 
  <moritzm> | 
  moved dns aliases ldap-eqiad.wikimedia.org and ldap-codfw.wikimedia.org to seaborgium/serpens (these are for compat reasons, using ldap-labs.[eqiad|codfw].wikimedia.org is preferred | 
  [production] | 
            
  | 20:50 | 
  <gwicke> | 
  restbase: finished deploy of 49fdc615b to all nodes | 
  [production] | 
            
  | 20:37 | 
  <gwicke> | 
  restbase: start deploy of 49fdc615b to all nodes | 
  [production] | 
            
  | 20:35 | 
  <hashar> | 
  Jenkins: changing LDAP config from ldaps://ldap-eqiad.wikimedia.org:636 to ldaps://ldap-labs.eqiad.wikimedia.org:636 | 
  [production] | 
            
  | 20:34 | 
  <gwicke> | 
  restbase: canary deploy of 49fdc615b to restbase1001 | 
  [production] | 
            
  | 20:31 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.7/extensions/MobileFrontend: I8a1fb69724b: Vary HTML output by NetSpeed designation (duration: 00m 28s) | 
  [production] | 
            
  | 19:52 | 
  <akosiaris> | 
  fixed ldap logging on seaborgium/serpens | 
  [production] | 
            
  | 19:51 | 
  <ori@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable ldap log channel (duration: 00m 28s) | 
  [production] | 
            
  | 19:50 | 
  <cmjohnson1> | 
  swapping failed disk db1019 | 
  [production] |