| 2018-11-07
      
      § | 
    
  | 14:26 | <bblack> | rebooting graphite1004 | [production] | 
            
  | 14:16 | <akosiaris> | T206439 upload  apertium-separable_0.3.2-1+wmf1 apertium-lex-tools_0.2.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 14:07 | <akosiaris> | T206439 upload apertium_3.5.2-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 13:43 | <bblack> | hi | [production] | 
            
  | 13:21 | <Amir1> | ladsgroup@mwmaint1002:/srv/mediawiki/php-1.33.0-wmf.1$ mwscript sql.php --wiki=sourceswiki extensions/Wikibase/client/sql/entity_usage.sql (T208858) | [production] | 
            
  | 12:38 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 12:36 | <zfilipin@deploy1001> | Synchronized wmf-config: SWAT: [[gerrit:471948|BC: Enable Schema.org page split test (T208763)]] (duration: 00m 54s) | [production] | 
            
  | 12:35 | <akosiaris> | T206439 upload  hfst-ospell_0.5.0-1+wmf1to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 12:27 | <akosiaris> | T206439 upload cg3_1.1.7-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 12:16 | <zfilipin@deploy1001> | Synchronized wmf-config/InterwikiSortOrders.php: SWAT: [[gerrit:468337|Add dty, gor, inh, kbp and lfn to InterwikiSortOrders (T208217)]] (duration: 00m 53s) | [production] | 
            
  | 12:12 | <zfilipin@deploy1001> | Synchronized wmf-config/throttle.php: SWAT: [[gerrit:472027|New throttle rule for Art&Feminism event in Chile (T208866)]] (duration: 00m 54s) | [production] | 
            
  | 12:12 | <akosiaris> | T206439 upload hfst_3.15.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 12:08 | <zfilipin@deploy1001> | Synchronized wmf-config/throttle.php: SWAT: [[gerrit:472026|Remove expired throttle rules]] (duration: 01m 05s) | [production] | 
            
  | 11:49 | <akosiaris> | T206439 upload lttoolbox_3.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main | [production] | 
            
  | 11:49 | <akosiaris> | T206439 upload lttoolbox_3.5.0-1+wmf1 | [production] | 
            
  | 10:06 | <hashar> | CI: switched operations/puppet job to be based on Stretch ( T208422 ) and to add python3 ( T208873 ) | [production] | 
            
  | 09:25 | <_joe_> | run systemctl reset-failed on ms-be1029, had a failed debmonitor session | [production] | 
            
  | 08:16 | <kartik@deploy1001> | Finished deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 (duration: 04m 59s) | [production] | 
            
  | 08:11 | <kartik@deploy1001> | Started deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 | [production] | 
            
  | 02:55 | <ejegg> | disabled recurring charge jobs | [production] | 
            
  | 00:46 | <mutante> | tegmen - shutting down for renaming and reinstall (T208824) | [production] | 
            
  | 00:11 | <dereckson@deploy1001> | Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 44s) | [production] | 
            
  
    | 2018-11-06
      
      § | 
    
  | 23:55 | <mutante> | cp1084 - network went down, powercycled, probably T203194 | [production] | 
            
  | 22:49 | <ejegg> | updated fundraising CiviCRM from e0742d2210 to 769dcf6456 | [production] | 
            
  | 21:50 | <mutante> | icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782) | [production] | 
            
  | 21:49 | <mutante> | icinga1001 - the "MediaWiki EtcdConfig up-to-date" checks were all unknown on the new icinga server, this was because systemd unit update-etcd-mw-config-lastindex was present but service not running. that was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" to start it and the checks | [production] | 
            
  | 21:40 | <thcipriani@deploy1001> | rebuilt and synchronized wikiversions files: Group0 wikis to 1.33.0-wmf.3 | [production] | 
            
  | 20:19 | <thcipriani@deploy1001> | Finished scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache (duration: 34m 06s) | [production] | 
            
  | 19:45 | <thcipriani@deploy1001> | Started scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache | [production] | 
            
  | 19:44 | <thcipriani@deploy1001> | Pruned MediaWiki: 1.32.0-wmf.23 (duration: 07m 19s) | [production] | 
            
  | 18:36 | <thcipriani> | cutting branch for MediaWiki and extensions version 1.33.0-wmf.3 | [production] | 
            
  | 17:37 | <XioNoX> | add vlan-analytics1-a-eqiad interface-range on asw2-a-eqiad | [production] | 
            
  | 16:47 | <XioNoX> | enable cr4-ulsfo zayo transport to cr1-codfw | [production] | 
            
  | 16:23 | <akosiaris@deploy1001> | scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging] | [production] | 
            
  | 16:23 | <akosiaris@deploy1001> | scap-helm zotero install --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging] | [production] | 
            
  | 16:12 | <mutante> | einsteinium - temp disabling icinga notifications and puppet, reloading icinga (for extra caution while deploying global NRPE change) | [production] | 
            
  | 16:07 | <mutante> | planet1001 - disabling puppet, editing NRPE config, testing allowed_hosts change | [production] | 
            
  | 16:07 | <akosiaris@deploy1001> | scap-helm -h finished | [production] | 
            
  | 16:07 | <akosiaris@deploy1001> | scap-helm -h cluster staging completed | [production] | 
            
  | 16:07 | <akosiaris@deploy1001> | scap-helm -h  [namespace: -h, clusters: staging] | [production] | 
            
  | 15:59 | <banyek> | updating facts for the puppet compilers | [production] | 
            
  | 15:37 | <akosiaris> | create zotero namespace in eqiad, codfw, staging cluster T201611 | [production] | 
            
  | 15:20 | <godog> | switch all graphite read traffic to graphite1004 | [production] | 
            
  | 15:16 | <XioNoX> | push `lldp port-id-subtype interface-name` to all compatible switches - T208630 | [production] | 
            
  | 15:14 | <jiji> | scb1001/scb1002 switched nutcracker redis from rdb1001:6382 to rdb1009:6379 | [production] | 
            
  | 15:08 | <XioNoX> | push `lldp port-id-subtype interface-name` to all routers - T208630 | [production] | 
            
  | 14:40 | <jynus_> | reducing consistenct temp. on db2048 to avoid lagging | [production] | 
            
  | 14:30 | <moritzm> | installing Ruby 2.1 security updates | [production] | 
            
  | 14:22 | <godog> | add graphite1004 to graphite cluster for reads | [production] | 
            
  | 14:21 | <moritzm> | installing clamav security updates on mendelevium/ticket.wikimedia.org | [production] |