| 
      
        2021-05-14
      
      §
     | 
  
    
  | 20:42 | 
  <dzahn@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet | 
  [production] | 
            
  | 20:32 | 
  <mutante> | 
  people1002 - decom'ing - please use people1003 and see list mail | 
  [production] | 
            
  | 20:31 | 
  <dzahn@cumin1001> | 
  START - Cookbook sre.hosts.decommission for hosts people1002.eqiad.wmnet | 
  [production] | 
            
  | 18:58 | 
  <cdanis@cumin1001> | 
  END (PASS) - Cookbook sre.network.cf (exit_code=0) | 
  [production] | 
            
  | 18:58 | 
  <cdanis@cumin1001> | 
  START - Cookbook sre.network.cf | 
  [production] | 
            
  | 18:39 | 
  <cdanis> | 
  ✔️ cdanis@install1003.wikimedia.org ~ 🕝☕ sudo systemctl restart squid.service | 
  [production] | 
            
  | 18:14 | 
  <mutante> | 
  people1003/people2002: awk -F: '$6 ~ "^\/home" {print $1,$6}' /etc/passwd  | while read line ; do user=${line% *}; dir=${line#* }; sudo mkdir -p ${dir}/public_html; sudo chown $user ${dir}/public_html; done (courtesy of Jbond) | 
  [production] | 
            
  | 17:49 | 
  <bblack> | 
  install1003 - restored normal resolv.conf + re-enabled+ran puppet | 
  [production] | 
            
  | 17:41 | 
  <bblack> | 
  install1003 - restart squid | 
  [production] | 
            
  | 17:35 | 
  <bblack> | 
  install1003 - puppet disabled and /etc/resolv.conf manually patched over to deal with a current issue | 
  [production] | 
            
  | 17:25 | 
  <cdanis> | 
  rolled back cr1-eqiad/cr2-eqiad interface disables T282881 | 
  [production] | 
            
  | 17:10 | 
  <cdanis> | 
  cdanis@re0.cr1-eqiad# set interfaces gr-3/3/0.1 disable   # T282881 | 
  [production] | 
            
  | 17:03 | 
  <cdanis> | 
  cdanis@re0.cr2-eqiad# set interfaces gr-4/3/0.2 disable   # T282881 | 
  [production] | 
            
  | 15:22 | 
  <cdanis@cumin2002> | 
  END (PASS) - Cookbook sre.network.cf (exit_code=0) | 
  [production] | 
            
  | 15:22 | 
  <cdanis@cumin2002> | 
  START - Cookbook sre.network.cf | 
  [production] | 
            
  | 15:05 | 
  <Urbanecm> | 
  Start server-side upload for 1 video file (T282874) | 
  [production] | 
            
  | 14:09 | 
  <andrew@deploy1002> | 
  Finished deploy [horizon/deploy@5d0a683]: removing 'locality' from trove dashboard (duration: 04m 15s) | 
  [production] | 
            
  | 14:04 | 
  <andrew@deploy1002> | 
  Started deploy [horizon/deploy@5d0a683]: removing 'locality' from trove dashboard | 
  [production] | 
            
  | 12:54 | 
  <bblack> | 
  re-running puppet agent on cp5* | 
  [production] | 
            
  | 12:19 | 
  <jbond42> | 
  run puppet on CP servers | 
  [production] | 
            
  | 04:20 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/revisionlist/RevisionItem.php: fix deprecation warning T282825 (duration: 01m 07s) | 
  [production] | 
            
  | 04:19 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/revisiondelete/RevDelRevisionItem.php: fix deprecation warning T282825 (duration: 01m 07s) | 
  [production] | 
            
  | 04:18 | 
  <ariel@deploy1002> | 
  Finished deploy [dumps/dumps@b97a2a9]: eliminate double slash in construction of api path (duration: 00m 03s) | 
  [production] | 
            
  | 04:18 | 
  <ariel@deploy1002> | 
  Started deploy [dumps/dumps@b97a2a9]: eliminate double slash in construction of api path | 
  [production] | 
            
  | 03:25 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/extensions/MapSources/includes/specials/MapSourcesPage.php: fix PHP notice T282833 (duration: 01m 07s) | 
  [production] | 
            
  | 03:20 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/page/WikiPage.php: T282844 (duration: 01m 06s) | 
  [production] | 
            
  | 03:18 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/page/PageArchive.php: T282844 (duration: 01m 07s) | 
  [production] | 
            
  | 03:16 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/Revision/RevisionArchiveRecord.php: fix DeletedContributions breakage T282844 (duration: 01m 07s) | 
  [production] | 
            
  | 03:13 | 
  <tstarling@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/includes/logging/LogEventsList.php: fix PHP notice T282834 (duration: 01m 08s) | 
  [production] | 
            
  | 00:39 | 
  <ryankemper> | 
  T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage` | 
  [production] | 
            
  
    | 
      
        2021-05-13
      
      §
     | 
  
    
  | 23:53 | 
  <mutante> | 
  [sodium:~] $ sudo systemctl start update-ubuntu-mirror.service | 
  [production] | 
            
  | 23:50 | 
  <mutante> | 
  [sodium:~] $ sudo -u mirror /usr/local/sbin/update-ubuntu-mirror | 
  [production] | 
            
  | 23:22 | 
  <thcipriani@deploy1002> | 
  Synchronized php-1.37.0-wmf.5/extensions/WikimediaEvents: Backport: [[gerrit:690081|Fix "final_state: vector" bug in VectorPrefDiffInstrumentation (T261842)]] (duration: 01m 07s) | 
  [production] | 
            
  | 23:11 | 
  <thcipriani@deploy1002> | 
  Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:686700|Enable WikiLove extension on tawiki (T280326)]] (duration: 01m 07s) | 
  [production] | 
            
  | 23:10 | 
  <ryankemper> | 
  T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage` | 
  [production] | 
            
  | 23:09 | 
  <ryankemper> | 
  T280382 `sudo -i wmf-auto-reimage-host -p T280382 wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage` | 
  [production] | 
            
  | 23:09 | 
  <ryankemper> | 
  T280382 `sudo -i wmf-auto-reimage-host -p T280382 wdqs1003.eqiad.wmnet` on `ryankemper@cumin1001` tmux session `wdqs_reimage` | 
  [production] | 
            
  | 20:21 | 
  <urbanecm@deploy1002> | 
  Synchronized wmf-config/InitialiseSettings.php: REVERT: 9dc74e45579c9b868571529171421c4bf7de41fa: Revert "Enable media change tags on wikipedias" (T266067, T282822) (duration: 01m 07s) | 
  [production] | 
            
  | 20:09 | 
  <herron@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . | 
  [production] | 
            
  | 20:09 | 
  <herron@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . | 
  [production] | 
            
  | 20:08 | 
  <herron@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . | 
  [production] | 
            
  | 20:08 | 
  <herron@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . | 
  [production] | 
            
  | 19:43 | 
  <dancy@deploy1002> | 
  Synchronized php: group1 wikis to 1.37.0-wmf.5 (duration: 01m 06s) | 
  [production] |