| 2018-11-12
      
      § | 
    
  | 13:46 | <moritzm> | updating libfastjson on stretch to 0.99.8-1~bpo9+1wmf1 | [production] | 
            
  | 13:41 | <gehel> | starting rolling restart of elasticsearch codfw for JVM upgrade | [production] | 
            
  | 13:32 | <phuedx> | phuedx@mwmaint1002 running restPageRandom.php maintenance script for mediawikiwiki | [production] | 
            
  | 13:23 | <phuedx> | phuedx@mwmaint1002 running resetPageRandom.php maintenance script for testwiki | [production] | 
            
  | 13:17 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 13:16 | <phuedx@deploy1001> | Synchronized php-1.33.0-wmf.3/maintenance/resetPageRandom.php: SWAT: [[gerrit:472958|Provide a script to reset the page_random column (T208909)]] (duration: 00m 53s) | [production] | 
            
  | 13:16 | <moritzm> | updating liblognorm on stretch to 2.0.3-1~bpo9+1wmf1 | [production] | 
            
  | 13:14 | <phuedx@deploy1001> | Synchronized php-1.33.0-wmf.3/autoload.php: SWAT: [[gerrit:472958|Provide a script to reset the page_random column (T208909)]] (duration: 00m 55s) | [production] | 
            
  | 13:12 | <elukey> | upgrade the Hadoop Analytics cluster to CDH 5.15 (downtime required) | [production] | 
            
  | 12:54 | <zfilipin@deploy1001> | Synchronized wmf-config/throttle.php: SWAT: [[gerrit:472691|Add new throttle rule for Wikipedia event in Ireland on 2018-11-13 (T209037)]] (duration: 00m 53s) | [production] | 
            
  | 12:15 | <jiji> | Restarting nutcracker on scb200[1-6] - T206450 | [production] | 
            
  | 12:00 | <moritzm> | uploaded jenkins 2.138.3 to apt.wikimedia.org (jessie and stretch) | [production] | 
            
  | 11:49 | <hashar> | updating puppet CI job for mtail upgrade https://gerrit.wikimedia.org/r/#/c/integration/config/+/472962/ | [production] | 
            
  | 11:37 | <hashar> | contint1001 : cleaning disk | T209123 ? | [production] | 
            
  | 11:26 | <moritzm> | installing Java security updates on elastic* | [production] | 
            
  | 10:51 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 55s) | [production] | 
            
  | 10:48 | <godog> | upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia | [production] | 
            
  | 10:45 | <marostegui> | Deploy schema change on db2048 (s1 codfw master), this will generate lag on s1 codfw - T51191 | [production] | 
            
  | 10:43 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 53s) | [production] | 
            
  | 10:32 | <elukey> | upload mcrouter exporter 0.0.0+git20181106 to stretch-wikimedia | [production] | 
            
  | 09:57 | <elukey> | upgraded cdh packages (cdh 5.10 -> 5.15) for thirdparty/cloudera in jessie/stretch-wikimedia | [production] | 
            
  | 09:12 | <marostegui> | Deploy schema change on db2048 (s1 codfw master) (replication will be stopped) - T67448 | [production] | 
            
  | 08:53 | <marostegui> | Deploy schema change on db1099:3318 - T203709 | [production] | 
            
  | 08:52 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 01m 01s) | [production] | 
            
  | 08:41 | <marostegui> | Change sync_binlog to 0 and trx_commit to 2 on dbstore2002:3313 to let it catch up | [production] | 
            
  | 08:26 | <_joe_> | uploading new php-{luasandbox,wikidiff2} to stretch main component, rebuild php-{luasandbox,wikidiff2,geoip,msgpack} for php 7.2, upload to stretch component php72, T208433 | [production] | 
            
  | 08:23 | <godog> | temporarily disable puppet in codfw before enabling rsyslog_exporter | [production] | 
            
  
    | 2018-11-09
      
      § | 
    
  | 21:46 | <SMalyshev> | repooled wdqs1004 - looks like other servers feel worse so probably makes sense to share the load equally | [production] | 
            
  | 21:16 | <jiji> | Reimaging rdb2003, rdb2004 - T206450 | [production] | 
            
  | 20:46 | <SMalyshev> | depooled wdqs1004 to let it catch up | [production] | 
            
  | 20:40 | <legoktm@deploy1001> | Synchronized docroot/mediawiki/keys/: Add my (20after4) PGP key to mediawiki.org/keys/keys.(txt|html) (duration: 00m 55s) | [production] | 
            
  | 20:08 | <andrewbogott> | restarted neutron-linuxbridge-agent on cloudvirt1018 and cloudvirt1023 | [production] | 
            
  | 19:39 | <banyek|away> | repooling labsdb1011 (T189158) | [production] | 
            
  | 15:23 | <banyek> | depooling labsdb1011 (T189158) | [production] | 
            
  | 15:23 | <banyek> | depooling labsdb1011 | [production] | 
            
  | 15:08 | <banyek> | repooling labsdb1009 (T189158) | [production] | 
            
  | 15:06 | <bblack> | cp1008/pinkunicorn: puppet disabled, public-facing testing of new globalsign 2018 certs | [production] | 
            
  | 15:04 | <ladsgroup@deploy1001> | Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s) | [production] | 
            
  | 14:50 | <andrewbogott> | rebooting cloudvirt1024 to (I hope) cause a page | [production] | 
            
  | 14:49 | <ladsgroup@deploy1001> | Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II | [production] | 
            
  | 14:48 | <ladsgroup@deploy1001> | deploy aborted: T191842 T209060 (duration: 09m 32s) | [production] | 
            
  | 14:39 | <ladsgroup@deploy1001> | Started deploy [ores/deploy@0728805]: T191842 T209060 | [production] | 
            
  | 14:18 | <addshore@deploy1001> | Synchronized wmf-config: BETA ONLY: Enable SSR termbox for wikibase on beta - T209143 (duration: 00m 56s) | [production] | 
            
  | 13:32 | <moritzm> | rebooting acrab for some qemu tests | [production] | 
            
  | 13:21 | <godog> | upload graphite-web_1.0.2+debian-2.1wmf1 to stretch-wikimedia - T208782 | [production] | 
            
  | 13:10 | <moritzm> | upgrading qemu on ganeti2001 (packages supporting SSBD passthrough) | [production] |