| 
      
        2020-07-02
      
      §
     | 
  
    
  | 23:22 | 
  <jhuneidi@deploy1001> | 
  helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' . | 
  [production] | 
            
  | 23:16 | 
  <jhuneidi@deploy1001> | 
  helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' . | 
  [production] | 
            
  | 22:03 | 
  <jhuneidi@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . | 
  [production] | 
            
  | 21:56 | 
  <mutante> | 
  gerrit1001 (prod gerrit) - restarting gerrit service | 
  [production] | 
            
  | 21:52 | 
  <maryum> | 
  frwikibooks reindex sucessful, continuing on with remainder of french wikis | 
  [production] | 
            
  | 21:32 | 
  <mutante> | 
  gerrit - deleted gerrit db_pass from prod private repo, running puppet | 
  [production] | 
            
  | 21:25 | 
  <mutante> | 
  gerrit2001 - restarted gerrit | 
  [production] | 
            
  | 21:14 | 
  <mutante> | 
  gerrit1002 restarted gerrit | 
  [production] | 
            
  | 20:20 | 
  <maryum> | 
  reindexing frwikibooks to test https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/604221 | 
  [production] | 
            
  | 19:52 | 
  <mutante> | 
  gerrit2001 - restarting gerrit after removing db_pass from config | 
  [production] | 
            
  | 16:05 | 
  <cmjohnson@cumin1001> | 
  END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) | 
  [production] | 
            
  | 16:01 | 
  <cmjohnson@cumin1001> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 15:40 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) | 
  [production] | 
            
  | 15:37 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.reboot-single | 
  [production] | 
            
  | 15:23 | 
  <cmjohnson@cumin1001> | 
  END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) | 
  [production] | 
            
  | 15:19 | 
  <cmjohnson@cumin1001> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 15:07 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 15:07 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 14:42 | 
  <moritzm> | 
  rebooting mw1370-mw1389 for kernel security updates | 
  [production] | 
            
  | 14:33 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 14:33 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 14:03 | 
  <kormat> | 
  stopped mariadb@s8 on dbstore1005 for data restoration T256966 | 
  [production] | 
            
  | 12:43 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:43 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 12:36 | 
  <jmm@cumin2001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:36 | 
  <jmm@cumin2001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 12:32 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:32 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 12:31 | 
  <moritzm> | 
  rebooting mw1349-mw1369 for kernel security updates | 
  [production] | 
            
  | 12:28 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:27 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 12:27 | 
  <vgutierrez> | 
  rolling restart of esams load balancers to catch up on kernel upgrades | 
  [production] | 
            
  | 12:12 | 
  <XioNoX> | 
  pre-configure asw2-b-eqiad<->cloudsw1-c8-eqiad - T251632 | 
  [production] | 
            
  | 12:10 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 12:10 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 11:53 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 11:53 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 11:40 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 11:40 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 11:35 | 
  <vgutierrez@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 11:35 | 
  <vgutierrez@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 11:33 | 
  <vgutierrez> | 
  rolling restart of codfw load balancers to catch up on kernel upgrades | 
  [production] | 
            
  | 11:18 | 
  <akosiaris> | 
  preactively restart docker-registry on registry1001, registry1002 to force CA refresh | 
  [production] | 
            
  | 11:16 | 
  <akosiaris> | 
  restart docker-registry on registry2002 for CA refresh | 
  [production] | 
            
  | 11:14 | 
  <_joe_> | 
  restarting docker-registry on registry2001 | 
  [production] |