| 2015-03-17
      
      § | 
    
  | 15:20 | <demon> | Synchronized php-1.25wmf21/includes/specials/SpecialUploadStash.php: (no message) (duration: 00m 07s) | [production] | 
            
  | 15:18 | <demon> | Synchronized wmf-config/InitialiseSettings.php: restbase off for wikipedias (duration: 00m 06s) | [production] | 
            
  | 15:17 | <gwicke> | restarted cassandra on restbase1006 after clearing the data & removing it from its own seeds | [production] | 
            
  | 15:09 | <demon> | Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 07s) | [production] | 
            
  | 15:04 | <demon> | Synchronized wmf-config/InitialiseSettings.php: draft namespace for kowiki (duration: 00m 05s) | [production] | 
            
  | 14:59 | <bblack> | rebooting cp1072-4, cp3030-49 (none in production) | [production] | 
            
  | 12:57 | <_joe_> | updating sudo across all production | [production] | 
            
  | 12:05 | <_joe_> | upgraded libicu48 and mediawiki-math-texvc across the cluster | [production] | 
            
  | 11:16 | <YuviPanda> | ran chown -R gitpuppet:gitpuppet /var/lib/git/operations/puppet on palladium, fix permission issues | [production] | 
            
  | 11:14 | <YuviPanda> | ran chown -R gitpuppet:gitpuppet /var/lib/git/operations/puppet on strontium, fix permission issues | [production] | 
            
  | 11:10 | <akosiaris> | chown gitpuppet:gitpuppet /var/lib/git/operations/puppet/.git/logs/refs/remotes/origin/production on strontium, palladium. Somehow it was owned by root | [production] | 
            
  | 10:59 | <godog> | depool restbase1006, provisioning | [production] | 
            
  | 07:40 | <_joe_> | powercycled mw2027, went down with an unresponsive console | [production] | 
            
  | 07:07 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Mar 17 07:06:36 UTC 2015 (duration 6m 35s) | [production] | 
            
  | 04:46 | <ejegg> | updated tools from 84442d51a841af4265ff103827cda83d5dd9dc54 to 9fd0a885e84074f215082aad689649a0684660f9 | [production] | 
            
  | 02:36 | <LocalisationUpdate> | completed (1.25wmf21) at 2015-03-17 02:35:00+00:00 | [production] | 
            
  | 02:34 | <l10nupdate> | Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 00m 04s) | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.25wmf20) at 2015-03-17 02:20:58+00:00 | [production] | 
            
  | 02:20 | <l10nupdate> | Synchronized php-1.25wmf20/cache/l10n: (no message) (duration: 00m 03s) | [production] | 
            
  | 02:03 | <ori> | applied I98d383a629 locally on mw1017 | [production] | 
            
  | 01:45 | <ori> | scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_0.713383032704/*" "/srv/mediawiki-staging/php-1.25wmf20/cache/l10n"' returned non-zero exit status 1 (duration: 00m 10s) | [production] | 
            
  | 01:45 | <ori> | Started scap: (no message) | [production] | 
            
  | 01:44 | <ori> | scap failed: TypeError temp_dir() takes exactly 2 arguments (1 given) (duration: 00m 10s) | [production] | 
            
  | 01:44 | <ori> | Started scap: (no message) | [production] | 
            
  | 01:31 | <tstarling> | Finished scap: (no message) (duration: 30m 19s) | [production] | 
            
  | 01:21 | <ori> | restarted HHVM on mw1139; bt in https://phabricator.wikimedia.org/P406 | [production] | 
            
  | 01:04 | <mutante> | disabled contacts.wikimedia.org - if you are an (unexpected)  user please contact me or T90679 | [production] | 
            
  | 01:01 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:58 | <ori> | Increased memory limit on HHVM app servers from 300M to 500M in an attempt to reduce the rate at which T89918 occurs | [production] | 
            
  | 00:57 | <tstarling> | scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2149279197/*" "/srv/mediawiki-staging/php-1.25wmf20/cache/l10n"' returned non-zero exit status 1 (duration: 04m 27s) | [production] | 
            
  | 00:53 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:48 | <tstarling> | scap failed: ValueError unsupported format character '/' (0x2f) at index 18 (duration: 04m 29s) | [production] | 
            
  | 00:44 | <tgr> | running extensions/GlobalUsage/refreshGlobalimagelinks.php --wiki=plwiki --pages=nonexisting | [production] | 
            
  | 00:44 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:17 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:11 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:10 | <tstarling> | scap failed: TypeError cannot concatenate 'str' and 'int' objects (duration: 00m 07s) | [production] | 
            
  | 00:10 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:08 | <tstarling> | scap failed: NameError global name 'random' is not defined (duration: 00m 08s) | [production] | 
            
  | 00:08 | <tstarling> | Started scap: (no message) | [production] | 
            
  | 00:01 | <Tim> | on tin: disabling puppet for scap test. Patching scap locally | [production] |