| 
      
        2019-12-18
      
      §
     | 
  
    
  | 06:37 | 
  <moritzm> | 
  upgrading debdeploy-client to 0.2.0 fleet-wide | 
  [production] | 
            
  | 06:36 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Slowly repool db1105:3311, db1105:3312', diff saved to https://phabricator.wikimedia.org/P9923 and previous config saved to /var/cache/conftool/dbconfig/20191218-063652-marostegui.json | 
  [production] | 
            
  | 06:30 | 
  <marostegui> | 
  Upgrade db1105 | 
  [production] | 
            
  | 06:28 | 
  <marostegui@cumin1001> | 
  dbctl commit (dc=all): 'Depool db1105:3311, db1105:3312', diff saved to https://phabricator.wikimedia.org/P9922 and previous config saved to /var/cache/conftool/dbconfig/20191218-062759-marostegui.json | 
  [production] | 
            
  | 06:24 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool pc1007 after upgrade (duration: 01m 00s) | 
  [production] | 
            
  | 06:17 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Depool pc1007 for upgrade (duration: 01m 11s) | 
  [production] | 
            
  | 06:03 | 
  <marostegui> | 
  Upgrade db2112 db2116 db2130 | 
  [production] | 
            
  | 05:59 | 
  <marostegui> | 
  Upgrade db2088, db2092 | 
  [production] | 
            
  | 05:55 | 
  <marostegui> | 
  Upgrade db2071 and db2072 | 
  [production] | 
            
  | 05:31 | 
  <marostegui> | 
  Deploy schema change on commonswiki.image on s4 primary master (db1138) - T233135 | 
  [production] | 
            
  | 04:59 | 
  <XioNoX> | 
  advertise 185.15.57.0/24 from [co|eq]dfw - T239347 | 
  [production] | 
            
  | 04:54 | 
  <XioNoX> | 
  add static routes for cloud's 185.15.57.0/29 on cr1/2-codfw - T239347 | 
  [production] | 
            
  | 02:14 | 
  <eileen> | 
  civicrm revision changed from b2d0b5d66d to 93037d6e35, config revision is 2e9bf6308b | 
  [production] | 
            
  | 00:22 | 
  <niharika29@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable Glent M0 for dewiki, enwiki and frwiki - T237365 (duration: 01m 02s) | 
  [production] | 
            
  | 00:15 | 
  <niharika29@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Disable Glent M0 A/B test - T237363 (duration: 01m 02s) | 
  [production] | 
            
  | 00:09 | 
  <niharika29@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable  on test wikis - T240736 (duration: 01m 02s) | 
  [production] | 
            
  
    | 
      
        2019-12-17
      
      §
     | 
  
    
  | 22:55 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [wikimedia/discovery/analytics@0dd9f6b]: Ship kerberos configuration for oozie jobs (duration: 15m 50s) | 
  [production] | 
            
  | 22:39 | 
  <ebernhardson@deploy1001> | 
  Started deploy [wikimedia/discovery/analytics@0dd9f6b]: Ship kerberos configuration for oozie jobs | 
  [production] | 
            
  | 22:19 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . | 
  [production] | 
            
  | 22:18 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . | 
  [production] | 
            
  | 22:10 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [wikimedia/discovery/analytics@100bf96]: Ship kerberos configuration for oozie jobs (duration: 00m 34s) | 
  [production] | 
            
  | 22:09 | 
  <ebernhardson@deploy1001> | 
  Started deploy [wikimedia/discovery/analytics@100bf96]: Ship kerberos configuration for oozie jobs | 
  [production] | 
            
  | 21:52 | 
  <cdanis> | 
  ✔️ cdanis@cp3050.esams.wmnet ~ 🕔🍵 sudo depool | 
  [production] | 
            
  | 21:04 | 
  <mholloway-shell@deploy1001> | 
  Synchronized php-1.35.0-wmf.10/extensions/MachineVision: Add more info to MachineVisionEntitySaveException message, take 2 (duration: 01m 04s) | 
  [production] | 
            
  | 21:02 | 
  <mholloway-shell@deploy1001> | 
  Synchronized php-1.35.0-wmf.11/extensions/MachineVision: Add more info to MachineVisionEntitySaveException message, take 2 (duration: 01m 04s) | 
  [production] | 
            
  | 19:49 | 
  <hashar> | 
  Restarting CI Jenkins for plugins upgrades | 
  [production] | 
            
  | 19:41 | 
  <mholloway-shell@deploy1001> | 
  Synchronized wmf-config/CommonSettings.php: Disable MachineVision email notifications (T240878) (duration: 01m 07s) | 
  [production] | 
            
  | 19:00 | 
  <mholloway-shell@deploy1001> | 
  Synchronized php-1.35.0-wmf.10/extensions/MachineVision: Add more info to MachineVisionEntitySaveException message (duration: 01m 08s) | 
  [production] | 
            
  | 18:58 | 
  <mholloway-shell@deploy1001> | 
  Synchronized php-1.35.0-wmf.11/extensions/MachineVision: Add more info to MachineVisionEntitySaveException message (duration: 01m 07s) | 
  [production] | 
            
  | 18:12 | 
  <mholloway-shell@deploy1001> | 
  Finished deploy [mobileapps/deploy@c30d801]: Update mobileapps to 5551575 (duration: 06m 38s) | 
  [production] | 
            
  | 18:06 | 
  <mholloway-shell@deploy1001> | 
  Started deploy [mobileapps/deploy@c30d801]: Update mobileapps to 5551575 | 
  [production] | 
            
  | 17:06 | 
  <moritzm> | 
  uploading debmonitor-client 0.2.0 to apt.wikimedia.org (jessie/stretch/buster) T237978 | 
  [production] | 
            
  | 16:49 | 
  <marostegui> | 
  Deploy schema change on commonswiki.logging on db1138 (s4 primary master) - T233135 | 
  [production] | 
            
  | 16:14 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.35.0-wmf.10/includes/specials/pagers/NewPagesPager.php: T240924 NewPagesPager: Fix namespace query conditions (duration: 01m 03s) | 
  [production] | 
            
  | 16:07 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.35.0-wmf.11/includes/specials/pagers/NewPagesPager.php: T240924 NewPagesPager: Fix namespace query conditions (duration: 01m 02s) | 
  [production] | 
            
  | 15:57 | 
  <jmm@deploy1001> | 
  Finished deploy [debmonitor/deploy@bb99a23]: Debmonitor release v0.2.0 - T237978 (duration: 06m 25s) | 
  [production] | 
            
  | 15:52 | 
  <ema> | 
  text@ulsfo rolling ats-backend-restart to disable compress plugin https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/558089/ T238495 | 
  [production] | 
            
  | 15:50 | 
  <jmm@deploy1001> | 
  Started deploy [debmonitor/deploy@bb99a23]: Debmonitor release v0.2.0 - T237978 | 
  [production] | 
            
  | 15:24 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . | 
  [production] | 
            
  | 15:17 | 
  <addshore> | 
  addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=10 --from-id=3929876 # T237984 (I stopped it at Processed up to page 14546856 (P501)) | 
  [production] | 
            
  | 15:14 | 
  <onimisionipe> | 
  pool maps2003 after postgres init - T239728 | 
  [production] | 
            
  | 15:13 | 
  <addshore> | 
  addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size=10 --from-id=3929876 # T237984 | 
  [production] | 
            
  | 14:46 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . | 
  [production] | 
            
  | 14:45 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . | 
  [production] | 
            
  | 14:45 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . | 
  [production] | 
            
  | 14:42 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . | 
  [production] | 
            
  | 14:42 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . | 
  [production] | 
            
  | 14:42 | 
  <akosiaris@deploy1001> | 
  helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . | 
  [production] | 
            
  | 14:40 | 
  <cdanis> | 
  tearing down tmux sessions generating nic_saturation kludge on some memcached hosts | 
  [production] | 
            
  | 14:36 | 
  <gehel> | 
  restarting elastic1054 for config change (new master) | 
  [production] |