| 2017-06-30
      
      § | 
    
  | 06:45 | <_joe_> | started manually burrow on krypton, could not start due to a stale pidfile | [production] | 
            
  | 06:38 | <marostegui> | Deploy alter table on s6 all eqiad hosts (primary master not included) - T168661 | [production] | 
            
  | 06:12 | <marostegui> | Deploy alter table on db1018 on s2 - T168661 | [production] | 
            
  | 06:12 | <marostegui> | Deploy alter table on db1090 on s2 - T168661 | [production] | 
            
  | 06:11 | <marostegui> | Deploy alter table on db1076 on s2 - T168661 | [production] | 
            
  | 06:10 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1060 - T168661 (duration: 00m 42s) | [production] | 
            
  | 06:10 | <marostegui> | Deploy alter table on db1074 on s2 - T168661 | [production] | 
            
  | 06:07 | <marostegui> | Deploy alter table on db1060 on s2 - T168661 | [production] | 
            
  | 06:06 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1060 - T168661 (duration: 00m 42s) | [production] | 
            
  | 05:59 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1036 - T168661 (duration: 00m 43s) | [production] | 
            
  | 05:56 | <marostegui> | Deploy alter table on db1047 on s2 - T168661 | [production] | 
            
  | 05:56 | <marostegui> | Deploy alter table on db1036 on s2 - T168661 | [production] | 
            
  | 05:55 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1036 - T168661 (duration: 00m 42s) | [production] | 
            
  | 05:47 | <marostegui> | Deploy alter table on db1021 on s2 - T168661 | [production] | 
            
  | 05:44 | <marostegui> | Deploy alter table on dbstore1002 on s2 - T168661 | [production] | 
            
  | 05:43 | <marostegui> | Deploy alter table on dbstore1001 on s2 - T168661 | [production] | 
            
  | 05:37 | <marostegui> | Deploy alter table on db1069 (and let it replicate) on s2 - T168661 | [production] | 
            
  | 02:29 | <chasemp> | labstore1005 start drbd | [production] | 
            
  | 02:14 | <chasemp> | reboot labstore1005 (5m ago) | [production] | 
            
  | 01:25 | <chasemp> | reboot labstoer1005 | [production] | 
            
  | 01:23 | <chasemp> | fail nfs from labstore1005 to labstore1004 (I failed to log a previous failover to 1004 and back) | [production] | 
            
  
    | 2017-06-29
      
      § | 
    
  | 23:32 | <RoanKattouw> | Sorry I meant T169163 | [production] | 
            
  | 23:31 | <catrope@tin> | Synchronized php-1.30.0-wmf.7/resources/src/mediawiki.rcfilters/: RCFilters fixes (T169169, T169107, T169042) (duration: 00m 42s) | [production] | 
            
  | 23:28 | <catrope@tin> | Synchronized php-1.30.0-wmf.7/extensions/WikimediaEvents/: Add event logging for explode-similar on SRP (T149809) (duration: 00m 42s) | [production] | 
            
  | 23:26 | <catrope@tin> | Synchronized php-1.30.0-wmf.7/extensions/CirrusSearch/: "Explore similar" widget for CirrusSearch (T149809) (duration: 00m 54s) | [production] | 
            
  | 23:19 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Stop reader surveys (T131949) (duration: 00m 43s) | [production] | 
            
  | 22:16 | <chasemp> | set cfq scheduler on labstore1005 | [production] | 
            
  | 21:51 | <mutante> | APT - uploading python-django-south from jessie to wikimedia-stretch for librenms on stretch (T159756) | [production] | 
            
  | 21:40 | <chasemp> | reboot labstore1004 with grub set to gnulinux-advanced-1773f282-5a1b-441e-865c-8b70a0ebc925>gnulinux-4.4.0-3-amd64-advanced-1773f282-5a1b-441e-865c-8b70a0ebc925 | [production] | 
            
  | 21:10 | <mobrovac@tin> | Finished deploy [restbase/deploy@bcb83f4]: (no justification provided) (duration: 04m 21s) | [production] | 
            
  | 21:06 | <mobrovac@tin> | Started deploy [restbase/deploy@bcb83f4]: (no justification provided) | [production] | 
            
  | 21:06 | <mobrovac@tin> | Finished deploy [restbase/deploy@bcb83f4]: (no justification provided) (duration: 01m 02s) | [production] | 
            
  | 21:05 | <mobrovac@tin> | Started deploy [restbase/deploy@bcb83f4]: (no justification provided) | [production] | 
            
  | 21:04 | <mobrovac@tin> | Finished deploy [restbase/deploy@bcb83f4]: Fix special char handling in PDF back-end requests - T169223 (duration: 03m 14s) | [production] | 
            
  | 21:00 | <mobrovac@tin> | Started deploy [restbase/deploy@bcb83f4]: Fix special char handling in PDF back-end requests - T169223 | [production] | 
            
  | 20:46 | <mutante> | APT - reprepro copy stretch-wikimedia jessie-wikimedia prometheus-snmp-exporter (to make it available on stretch for netmon1002) (T159756) | [production] | 
            
  | 20:39 | <ppchelko@tin> | Finished deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 (duration: 01m 01s) | [production] | 
            
  | 20:38 | <ppchelko@tin> | Started deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 | [production] | 
            
  | 19:23 | <demon@tin> | Synchronized scap/plugins/clean.py: Because I need to learn basic python syntax before trying stuff (duration: 00m 42s) | [production] | 
            
  | 19:21 | <twentyafterfour@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.7 refs T167536 | [production] | 
            
  | 19:16 | <twentyafterfour> | deploying wmf/1.30.0-wmf.7 to all wikis refs T167536 | [production] | 
            
  | 19:06 | <demon@tin> | Pruned MediaWiki: 1.30.0-wmf.5 [keeping static files] (duration: 01m 16s) | [production] | 
            
  | 18:30 | <chasemp> | restart nfs on labstore1004 (primary) | [production] | 
            
  | 18:10 | <demon@tin> | Synchronized php-1.30.0-wmf.7/extensions/TextExtracts/extension.json: T107206 (duration: 00m 47s) | [production] | 
            
  | 17:28 | <arlolra> | Updated Parsoid to b4187f18 (T168900, T168675, T168404, T153203) | [production] | 
            
  | 17:21 | <arlolra@tin> | Finished deploy [parsoid/deploy@717df08]: Updating Parsoid to b4187f18 (duration: 09m 41s) | [production] | 
            
  | 17:12 | <arlolra@tin> | Started deploy [parsoid/deploy@717df08]: Updating Parsoid to b4187f18 | [production] | 
            
  | 17:08 | <mobrovac> | scb2005 repooling back the services - T167763 | [production] | 
            
  | 16:21 | <godog> | temporarily stop ircecho, puppet spam | [production] | 
            
  | 16:05 | <akosiaris@tin> | Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 46s) | [production] |