| 2018-09-24
      
      § | 
    
  | 06:57 | <godog> | repair sdn on ms-be1041 - T199198 | [production] | 
            
  | 06:54 | <volans@deploy1001> | Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s) | [production] | 
            
  | 06:54 | <volans@deploy1001> | Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix | [production] | 
            
  | 05:48 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) | [production] | 
            
  | 05:28 | <marostegui> | Deploy schema change on db2088:3311 - T203709 | [production] | 
            
  | 05:27 | <marostegui@deploy1001> | Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) | [production] | 
            
  | 04:15 | <kartik@deploy1001> | Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) | [production] | 
            
  | 04:10 | <kartik@deploy1001> | Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) | [production] | 
            
  | 02:48 | <l10nupdate@deploy1001> | ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) | [production] | 
            
  | 02:37 | <l10nupdate@deploy1001> | scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) | [production] | 
            
  
    | 2018-09-21
      
      § | 
    
  | 23:47 | <mutante> | mwmaint1002 - rsyncing home dirs over from mwmaint1001 which synced them from terbium. still large old files from terbium here, like a NASA video :) (T201343) | [production] | 
            
  | 22:40 | <mutante> | mwmaint1002 - scap pull | [production] | 
            
  | 22:13 | <mutante> | mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (T201343) | [production] | 
            
  | 22:13 | <mutante> | mwmaint1002 - re-enabling puppet, now that it has mcrouter certs, that works (201343) | [production] | 
            
  | 22:12 | <mutante> | how to generate mcrouter certs, needed when you add any new mw server: https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host | [production] | 
            
  | 19:10 | <XioNoX> | shutdown cr3/4-ulsfo (not in prod) | [production] | 
            
  | 18:19 | <bawolff> | deploy patch related to T197279 T204825 | [production] | 
            
  | 18:08 | <bblack> | upgrade multatuli to gdnsd-2.99.1729-beta | [production] | 
            
  | 18:06 | <bblack> | uploaded 2.99.1729-beta-1+wmf1 to stretch-wikimedia | [production] | 
            
  | 17:59 | <mutante> | *.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses | [production] | 
            
  | 17:58 | <mutante> | toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty | [production] | 
            
  | 16:37 | <XioNoX> | route ns2-v6 to multatuli.wikimedia.org on cr1/2-esams | [production] | 
            
  | 16:33 | <mutante> | puppetmaster1001: cergen --base-path /srv/private/modules/secret/secrets/mcrouter/ --generate /etc/cergen/mcrouter.manifests.d | [production] | 
            
  | 16:32 | <mutante> | puppetmaster1001: mcrouter_generate_certs --generate | [production] | 
            
  | 16:29 | <mutante> | puppetmaster: running mcrouter_generate_certs to add an mcrouter cert for mwmaint1002 (T201343) https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host | [production] | 
            
  | 15:39 | <bblack> | upgrading gdnsd to 2.99.42 on authdns2001 | [production] | 
            
  | 15:16 | <XioNoX> | route ns2 to multatuli.wikimedia.org on cr1/2-eqiad | [production] | 
            
  | 13:35 | <ladsgroup@deploy1001> | Finished deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) (duration: 24m 10s) | [production] | 
            
  | 13:11 | <ladsgroup@deploy1001> | Started deploy [ores/deploy@7b987a7]: Pass the real IP (T205087) | [production] | 
            
  | 13:01 | <volans> | wiping netbox DB to re-import it cleanly from racktables - T199083 | [production] | 
            
  | 12:43 | <banyek> | adding wmf-pt-kill_2.2.20-1+wmf1 package for jessie | [production] | 
            
  | 12:31 | <banyek> | adding wmf-pt-kill_2.2.20-1+wmf1 package for strech | [production] | 
            
  | 12:24 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 49s) | [production] | 
            
  | 12:13 | <marostegui> | Stop MySQL on pc1006 for kernel upgrade | [production] | 
            
  | 12:13 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool pc1006 (duration: 00m 50s) | [production] | 
            
  | 07:17 | <moritzm> | uploaded debdeploy 0.0.99.6 to apt.wikimedia.org | [production] | 
            
  | 07:00 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 50s) | [production] | 
            
  | 06:51 | <marostegui> | Stop MySQL on pc1005 for kernel upgrade | [production] | 
            
  | 06:51 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 51s) | [production] |