| 2018-05-25
      
      § | 
    
  | 22:57 | <mutante> | apt.wikimedia.org - import jenkins-debian-glue_0.18.4-wmf3 for jessie-wikimedia (T193910) | [production] | 
            
  | 21:26 | <legoktm@tin> | Synchronized php-1.32.0-wmf.5/skins/MonoBook/: Temporarily remove responsive support (T195625) (duration: 01m 21s) | [production] | 
            
  | 20:33 | <mutante> | LDAP: added user wmde-leszek to group 'nda' (T195358) | [production] | 
            
  | 17:46 | <XenoRyet> | updated civicrm from 4d797fc592 to 0b97f1f5b2 | [production] | 
            
  | 15:23 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 20s) | [production] | 
            
  | 13:47 | <akosiaris> | repool ulsfo, links have been stable for quite a few hours | [production] | 
            
  | 13:27 | <marostegui> | Deploy schema change on db1119 - https://phabricator.wikimedia.org/T190148 https://phabricator.wikimedia.org/T191519 https://phabricator.wikimedia.org/T188299 | [production] | 
            
  | 13:26 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 01m 20s) | [production] | 
            
  | 13:15 | <marostegui> | Add indexes back on s8 codfw primary master (db2045) this will generate lag on codfw - T194273 | [production] | 
            
  | 13:14 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 20s) | [production] | 
            
  | 12:28 | <moritzm> | fixed dpkg installation state on mx2001 | [production] | 
            
  | 11:32 | <akosiaris> | switch to SSH RSA 2048 bit keys for eqiad ganeti intracluster communication | [production] | 
            
  | 11:22 | <akosiaris> | upgrade eqiad ganeti cluster to ganeti 2.15.2-7+deb9u1~bpo8+1 | [production] | 
            
  | 11:21 | <akosiaris> | rebalance row_B codfw ganeti nodegroup. Cluster is now fully upgraded to stretch | [production] | 
            
  | 11:18 | <akosiaris> | powercycling ms-be1034, box is unresposive, tons of logs "sd 0:1:0:1: rejecting I/O to offline device" | [production] | 
            
  | 10:37 | <XioNoX> | test force mtu 1400 between cp1074 and cp3039 - T195365 | [production] | 
            
  | 09:20 | <marostegui> | Deploy schema change on db1105:3311 - T190148 T191519 T188299 | [production] | 
            
  | 09:20 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 01m 20s) | [production] | 
            
  | 09:16 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 19s) | [production] | 
            
  | 09:10 | <marostegui@tin> | scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) | [production] | 
            
  | 09:03 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 20s) | [production] | 
            
  | 08:44 | <marostegui> | Stop MySQL on db1120 to transfer its content to db1125 - T190704 | [production] | 
            
  | 08:36 | <marostegui> | Add tmp1 back on db1092 - https://phabricator.wikimedia.org/T194273 | [production] | 
            
  | 08:36 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s) | [production] | 
            
  | 08:30 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 20s) | [production] | 
            
  | 08:26 | <gilles@tin> | Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on frwiki (duration: 01m 22s) | [production] | 
            
  | 07:05 | <jynus> | stop db1117:m2 to clone it to db1065 | [production] | 
            
  | 06:57 | <marostegui> | Deploy schema change on db1114 - T190148 T191519 T188299 | [production] | 
            
  | 06:57 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 01m 20s) | [production] | 
            
  | 06:53 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 01m 20s) | [production] | 
            
  | 06:37 | <jynus> | reimage db1065 after raid rebuild | [production] | 
            
  | 05:57 | <marostegui> | Add tmp1 index back on dbstore1002 - T194273 | [production] | 
            
  | 05:25 | <marostegui> | Stop MySQL on db1116 to copy its content to db1124 - T190704 | [production] | 
            
  | 05:23 | <marostegui> | Deploy schema change on db1089 - T190148 T191519 T188299 | [production] | 
            
  | 05:23 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 01m 20s) | [production] | 
            
  | 05:17 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 01m 21s) | [production] | 
            
  | 05:05 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 20s) | [production] | 
            
  | 05:05 | <marostegui> | Add tmp1 index back to db1109 - T194273 | [production] | 
            
  | 04:59 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 21s) | [production] | 
            
  | 03:10 | <papaul> | OS install on db209[4-5] | [production] | 
            
  
    | 2018-05-24
      
      § | 
    
  | 23:13 | <ladsgroup@tin> | Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Dumps: Allow several --entity-type arguments (T195420) (duration: 02m 25s) | [production] | 
            
  | 22:21 | <ladsgroup@tin> | Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 20s) | [production] | 
            
  | 22:11 | <ladsgroup@tin> | Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 21s) | [production] | 
            
  | 20:58 | <marostegui> | Add tmp1 index back on db1104 | [production] | 
            
  | 20:57 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 20s) | [production] | 
            
  | 20:56 | <legoktm@tin> | Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/: no-op, sync with git state (duration: 01m 21s) | [production] | 
            
  | 20:54 | <legoktm@tin> | Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s) | [production] | 
            
  | 20:41 | <legoktm@tin> | Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s) | [production] | 
            
  | 20:17 | <legoktm@tin> | Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: wmf.5 this time (duration: 01m 19s) | [production] | 
            
  | 20:14 | <legoktm@tin> | Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Add debug logging (duration: 01m 19s) | [production] |