| 2015-11-18
      
      § | 
    
  | 14:47 | <mobrovac> | mobileapps deploying 151c312 | [production] | 
            
  | 13:55 | <bblack> | disabling pybal on lvs100[123] - fallback to lvs100[456] | [production] | 
            
  | 12:00 | <hashar> | reenabled puppet on gallium. Havent pooled zuul-merger yet | [production] | 
            
  | 11:57 | <hashar> | disabled puppet on gallium | [production] | 
            
  | 11:56 | <hashar> | pooling new zuul-merger on scandium | [production] | 
            
  | 11:44 | <mark> | Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1 | [production] | 
            
  | 10:49 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s) | [production] | 
            
  | 10:31 | <hashar> | gallium: apt-get upgrade | [production] | 
            
  | 10:30 | <hashar> | upgrading Zuul on gallium (CI interruption for a minute) | [production] | 
            
  | 09:48 | <jynus> | several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled) | [production] | 
            
  | 09:25 | <godog> | reimage restbase2002 | [production] | 
            
  | 09:11 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s) | [production] | 
            
  | 09:09 | <_joe_> | restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock | [production] | 
            
  | 06:31 | <_joe_> | rsyncing terbium homes from rutherfordium | [production] | 
            
  | 02:39 | <l10nupdate@tin> | Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s) | [production] | 
            
  | 01:46 | <andrewbogott> | repooling mw1156 | [production] | 
            
  | 01:43 | <yurik_> | synced graphoid service | [production] | 
            
  | 01:40 | <andrewbogott> | updated kernel on mw1156, now rebooting. | [production] | 
            
  | 01:35 | <andrewbogott> | repooling mw1158, depooling mw1156 | [production] | 
            
  | 01:22 | <andrewbogott> | rebooting mw1158 | [production] | 
            
  | 01:19 | <andrewbogott> | upgrading mw1158 to 3.13.0-62 | [production] | 
            
  | 00:26 | <legoktm@tin> | Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s) | [production] | 
            
  | 00:04 | <maxsem@tin> | Synchronized portals/: SWAT (duration: 00m 32s) | [production] | 
            
  
    | 2015-11-17
      
      § | 
    
  | 23:43 | <mutante> | deactivated wekipedia.om | [production] | 
            
  | 23:40 | <YuviPanda> | remove realm variable from all instances in ldap | [production] | 
            
  | 23:18 | <mutante> | deactivated wikiartpedia.[biz|co|info|me|mobi|net|org] | [production] | 
            
  | 23:13 | <mutante> | deactivated border-wikipedia.de | [production] | 
            
  | 23:01 | <mutante> | deactivated wikimedia.xyz | [production] | 
            
  | 22:57 | <mutante> | deactivated visualwikipedia.[com|net] | [production] | 
            
  | 22:56 | <mutante> | deactivated softwarewikipedia.[com|net|org] | [production] | 
            
  | 22:54 | <mutante> | deactivated indiawikipedia.com | [production] | 
            
  | 22:52 | <mutante> | deactivated vikipedia.com.tr, vikipedia.com.tr | [production] | 
            
  | 22:23 | <mutante> | deactivated wikimaps.com, wikimaps.net, wikimaps.org | [production] | 
            
  | 22:20 | <mutante> | deactivated wikimedia.biz, webhostingwikipedia.com | [production] | 
            
  | 22:18 | <twentyafterfour@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.7 | [production] | 
            
  | 21:07 | <twentyafterfour@tin> | Finished scap: sync new branch 1.27.0-wmf.7 and enable for testwiki (duration: 29m 40s) | [production] | 
            
  | 20:39 | <mutante> | deleted sitemap.wikimedia.org (T101486) | [production] | 
            
  | 20:37 | <twentyafterfour@tin> | Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki | [production] | 
            
  | 20:31 | <twentyafterfour@tin> | scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2994655917"/* "/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/l10n"' returned non-zero exit status 1 (duration: 00m 42s) | [production] | 
            
  | 20:30 | <twentyafterfour@tin> | Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki | [production] | 
            
  | 20:26 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4014066321" --threads=4 --lang en  --quiet' returned non-zero exit status 1 (duration: 01m 30s) | [production] | 
            
  | 20:25 | <twentyafterfour@tin> | Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki | [production] | 
            
  | 18:54 | <andrewbogott> | depooled mw1158 due to kernel errors.  https://phabricator.wikimedia.org/T118888 | [production] | 
            
  | 18:10 | <cmjohnson1> | swapping failing disk 32:8 on db1027..will cause icinga alert (jynus) | [production] | 
            
  | 18:02 | <bblack> | upgrading and re-enabling pybal on lvs300[12].esams.wmnet (1.10 -> 1.12) | [production] | 
            
  | 17:54 | <yurik> | deployed latest graphoid service | [production] | 
            
  | 16:45 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation for bs.wikipedia.org [[gerrit:250862]] (duration: 00m 26s) | [production] | 
            
  | 16:42 | <bblack> | upgrading / re-enabling pybal on lvs200[123].codfw.wmnet (1.10 -> 1.12) | [production] | 
            
  | 16:30 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Set import sources on en.wikiversity [[gerrit:250709]] (duration: 00m 27s) | [production] | 
            
  | 16:25 | <thcipriani@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ru.wikipedia.org [[gerrit:252911]] (duration: 00m 26s) | [production] |