| 
      
        2018-03-14
      
      §
     | 
  
    
  | 10:16 | 
  <jynus> | 
  archiving and dropping bugzilla_testing from m2 | 
  [production] | 
            
  | 10:10 | 
  <marostegui> | 
  Stop mariadb on pc2006 for kernel and mariadb upgrade + change socket location | 
  [production] | 
            
  | 10:09 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Depool pc2006 for kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 10:07 | 
  <jynus> | 
  archiving and dropping testblog from m2 | 
  [production] | 
            
  | 10:03 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Repool pc2005 after kernel and mariadb upgrade (duration: 01m 15s) | 
  [production] | 
            
  | 09:50 | 
  <marostegui> | 
  Stop mariadb on pc2005 for kernel and mariadb upgrade + change socket location | 
  [production] | 
            
  | 09:50 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Depool pc2005 for kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 09:44 | 
  <moritzm> | 
  installing samba security update (just the client side libraries) | 
  [production] | 
            
  | 09:40 | 
  <marostegui> | 
  Stop mysql on es2015 to upgrade socket path | 
  [production] | 
            
  | 09:37 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Repool pc2004 after kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 09:34 | 
  <marostegui> | 
  Stop mysql on es2014 to upgrade socket path | 
  [production] | 
            
  | 09:23 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-codfw.php: Depool pc2004 for kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 09:23 | 
  <marostegui> | 
  Stop mariadb on pc2004 for kernel upgrade | 
  [production] | 
            
  | 09:13 | 
  <marostegui> | 
  Stop mysql on es2013 to upgrade socket path | 
  [production] | 
            
  | 09:08 | 
  <marostegui> | 
  Stop mysql on es2012 to upgrade socket path | 
  [production] | 
            
  | 08:57 | 
  <ema> | 
  cp3041: restart varnish-be | 
  [production] | 
            
  | 08:55 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Fully repool es1013 after kernel and mariadb upgrade (duration: 01m 15s) | 
  [production] | 
            
  | 08:28 | 
  <ema> | 
  cp3040: restart varnish-be | 
  [production] | 
            
  | 08:21 | 
  <hashar> | 
  Restarting the CI Jenkins | 
  [production] | 
            
  | 07:45 | 
  <marostegui> | 
  Reboot es2004 for kernel upgrade | 
  [production] | 
            
  | 07:45 | 
  <marostegui> | 
  Reboot es2003 for kernel upgrade | 
  [production] | 
            
  | 07:34 | 
  <marostegui> | 
  Reboot es2002 for kernel upgrade | 
  [production] | 
            
  | 07:21 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Slowly repool es1013 after kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 07:03 | 
  <marostegui> | 
  Stop mariadb on es1013 for mariadb and kernel upgrade | 
  [production] | 
            
  | 06:59 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool es1013 for kernel and mariadb upgrade (duration: 01m 14s) | 
  [production] | 
            
  | 06:45 | 
  <marostegui> | 
  Deploy schema change on db1064 with replication (this will generate lag on s4 on labs hosts) - T187089 T185128 T153182 | 
  [production] | 
            
  | 06:39 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1064 for alter table (duration: 01m 14s) | 
  [production] | 
            
  | 06:30 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 01m 15s) | 
  [production] | 
            
  | 03:13 | 
  <mutante> | 
  bacula is working again - restored missing file set (https://gerrit.wikimedia.org/r/419341 ) | 
  [production] | 
            
  | 02:49 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.31.0-wmf.24) (duration: 05m 40s) | 
  [production] | 
            
  | 02:44 | 
  <Jamesofur> | 
  deleted 46 archived files | 
  [production] | 
            
  | 02:18 | 
  <mutante> | 
  helium - running bacula-dir with -f in foreground revealed: ERROR TERMINATION at parse_conf.c:485 - Config error: Could not find config Resource mysql-srv-backups - line 7, col 33 of file /etc/bacula/jobs.d/bohrium.eqiad.wmnet-mysql-predump-piwik-Weekly-Wed-production.conf | 
  [production] | 
            
  | 02:17 | 
  <mutante> | 
  helium - bacula director process failed (Bacula interrupted by signal 11: Segmentation violation), icinga alerted. attempted to restart it. then: bacula-dir - the configtest failed! | 
  [production] | 
            
  | 00:01 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: crwiki logo (duration: 01m 15s) | 
  [production] | 
            
  | 00:00 | 
  <reedy@tin> | 
  Synchronized static/images/project-logos/crwiki.png: (no justification provided) (duration: 01m 14s) | 
  [production] | 
            
  
    | 
      
        2018-03-13
      
      §
     | 
  
    
  | 23:46 | 
  <reedy@tin> | 
  Synchronized php-1.31.0-wmf.24/extensions/MobileFrontend/: T188825 (duration: 01m 18s) | 
  [production] | 
            
  | 23:43 | 
  <mutante> | 
  tin: chmod -R g+w /srv/mediawiki-staging/.git/objects/* ;  chmod -R g+w /srv/mediawiki-staging/php-1.31.0-wmf.24/.git/objects/* | 
  [production] | 
            
  | 23:35 | 
  <Reedy> | 
  that was Enable VirtualPageViews on Hungarian Wikipedia T184793 | 
  [production] | 
            
  | 23:35 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 01m 15s) | 
  [production] | 
            
  | 23:26 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: moar logos (duration: 01m 15s) | 
  [production] | 
            
  | 23:24 | 
  <reedy@tin> | 
  Synchronized static/images/project-logos/: YOU GET A LOGO, YOU GET A LOGO. YOU ALL GET LOGOS (duration: 01m 16s) | 
  [production] | 
            
  | 23:11 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable RemexHTML on 96 wikis T188010 (duration: 01m 16s) | 
  [production] | 
            
  | 23:10 | 
  <mutante> | 
  restbase-dev1006 - reinstalling, manually skipping " Volume group name already in use" (T185494) | 
  [production] | 
            
  | 22:52 | 
  <eileen> | 
  civicrm revision changed from c8458c4a2f to 9e79d63426, config revision is 08b7e6216e (Benevity comma fix) | 
  [production] | 
            
  | 20:40 | 
  <demon@tin> | 
  rebuilt and synchronized wikiversions files: group0 to wmf.25 | 
  [production] | 
            
  | 20:09 | 
  <demon@tin> | 
  Finished scap: bootstrap wmf.25 (duration: 67m 17s) | 
  [production] | 
            
  | 19:02 | 
  <demon@tin> | 
  Started scap: bootstrap wmf.25 | 
  [production] | 
            
  | 18:47 | 
  <demon@tin> | 
  scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.operations_mediawiki-config.lock"; owner is "awight"; reason is "Beta: Fix ORES thresholds and enable JADE, T181159, T176333" (duration: 00m 00s) | 
  [production] | 
            
  | 18:46 | 
  <demon@tin> | 
  scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.operations_mediawiki-config.lock"; owner is "awight"; reason is "Beta: Fix ORES thresholds and enable JADE, T181159, T176333" (duration: 00m 00s) | 
  [production] | 
            
  | 18:42 | 
  <gehel> | 
  repool wdqs1004 & wdqs2001 now that data reload is completed T189548 | 
  [production] |