| 2019-03-18
      
      § | 
    
  | 23:54 | <maxsem@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/494551/ (duration: 00m 49s) | [production] | 
            
  | 23:45 | <maxsem@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/494551/ (duration: 00m 48s) | [production] | 
            
  | 23:33 | <maxsem@deploy1001> | Synchronized php-1.33.0-wmf.21/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/497347/ (duration: 00m 49s) | [production] | 
            
  | 23:25 | <twentyafterfour> | running puppet on phab1001 to get out of degraded state | [production] | 
            
  | 23:23 | <XioNoX> | renumber Telia transit in eqsin | [production] | 
            
  | 23:14 | <maxsem@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/497317/ (duration: 00m 49s) | [production] | 
            
  | 23:07 | <maxsem@deploy1001> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/496515/ (duration: 00m 48s) | [production] | 
            
  | 22:18 | <greg-g> | gjg@phab1001:~$ sudo /srv/phab/phabricator/bin/auth strip --all-types --user Barras # per request/verification from foks | [production] | 
            
  | 21:00 | <bd808> | Testing stashbot's ability to log events to wikitech | [production] | 
            
  | 19:57 | <bawolff@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: Enable block disables login on wikitech (duration: 00m 48s) | [production] | 
            
  | 19:56 | <bawolff@deploy1001> | Synchronized wmf-config/wikitech.php: Adjust ldap config (duration: 00m 48s) | [production] | 
            
  | 16:17 | <volans> | restarting pdfrender on scb1003 | [production] | 
            
  | 16:15 | <volans> | restarting pdfrender on scb1004 | [production] | 
            
  | 15:48 | <jiji@cumin1001> | conftool action : set/pooled=no; selector: dc=eqiad,service=cxserver,cluster=scb,name=scb.* | [production] | 
            
  | 15:45 | <jijiki> | Depool sbc* from serving cxserver on eqiad - T213195 | [production] | 
            
  | 15:06 | <papaul> | shutting down mw2206 for memtest | [production] | 
            
  | 14:47 | <gehel@cumin1001> | END (FAIL) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=99) | [production] | 
            
  | 14:46 | <gehel@cumin1001> | START - Cookbook sre.elasticsearch.e6-upgrade | [production] | 
            
  | 14:13 | <filippo@puppetmaster1001> | conftool action : set/pooled=no; selector: name=prometheus2003.codfw.wmnet | [production] | 
            
  | 13:41 | <ema> | cp-ats rolling restart to apply proxy.config.cache.ram_cache.size config change T213263 | [production] | 
            
  | 13:23 | <mvolz@deploy1001> | scap-helm citoid finished | [production] | 
            
  | 13:22 | <mvolz@deploy1001> | scap-helm citoid cluster codfw completed | [production] | 
            
  | 13:22 | <mvolz@deploy1001> | scap-helm citoid upgrade production -f citoid-codfw-values.yaml stable/citoid [namespace: citoid, clusters: codfw] | [production] | 
            
  | 13:18 | <mvolz@deploy1001> | scap-helm citoid finished | [production] | 
            
  | 13:18 | <mvolz@deploy1001> | scap-helm citoid cluster eqiad completed | [production] | 
            
  | 13:17 | <mvolz@deploy1001> | scap-helm citoid upgrade production -f citoid-eqiad-values.yaml stable/citoid [namespace: citoid, clusters: eqiad] | [production] | 
            
  | 13:03 | <arturo> | T218022 disable icinga checks for labtestservices2001.wikimedia.org | [production] | 
            
  | 12:54 | <arturo> | T218025 disable icinga checks for cloudnet2001-dev.codfw.wmnet | [production] | 
            
  | 12:49 | <mvolz@deploy1001> | scap-helm citoid finished | [production] | 
            
  | 12:49 | <mvolz@deploy1001> | scap-helm citoid cluster staging completed | [production] | 
            
  | 12:49 | <mvolz@deploy1001> | scap-helm citoid upgrade staging -f citoid-staging-values.yaml stable/citoid [namespace: citoid, clusters: staging] | [production] | 
            
  | 12:48 | <mvolz@deploy1001> | scap-helm citoid upgrade staging -f citoid-values-staging.yaml stable/citoid [namespace: citoid, clusters: staging] | [production] | 
            
  | 11:45 | <zeljkof> | EU SWAT finished | [production] | 
            
  | 11:45 | <zfilipin@deploy1001> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:496696|Enable mobile section editing on bnwiki, hewiki, zh_yuewiki (T218375)]] (duration: 00m 50s) | [production] | 
            
  | 10:51 | <_joe_> | testing safety checks for php-fpm on mwdebug2001 | [production] | 
            
  | 10:35 | <jdrewniak@deploy1001> | Synchronized portals: Wikimedia Portals Update: [[gerrit:497261| Bumping portals to master (T128546)]] (duration: 00m 48s) | [production] | 
            
  | 10:34 | <jdrewniak@deploy1001> | Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:497261| Bumping portals to master (T128546)]] (duration: 00m 49s) | [production] | 
            
  | 10:12 | <vgutierrez> | uploaded acme-chief 0.12 to apt.wikimedia.org (buster) - T218543 | [production] | 
            
  | 10:12 | <volans> | restarted irc echo on icinga2001 | [production] | 
            
  | 10:04 | <_joe_> | hot-patching the error in php7.2-fpm config | [production] | 
            
  | 10:02 | <volans> | running puppet on hosts matching  'C:php::fpm' to apply I004349ebfab34a4b7b5a65b9b10f78817e5cf193 | [production] | 
            
  | 10:00 | <volans> | running puppet on failed hosts | [production] | 
            
  | 09:57 | <volans> | temporarily stop ircecho to avoid spam | [production] | 
            
  | 09:40 | <ema> | superior-cache-analyzer_3.3.7 uploaded to stretch-wikimedia T213263 | [production] | 
            
  | 09:29 | <godog> | switch to mpm_event for prometheus apache before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/496750 | [production] | 
            
  | 08:58 | <vgutierrez> | uploaded acme-chief 0.11 to apt.wikimedia.org (buster) - T207295 | [production] | 
            
  | 08:52 | <moritzm> | restarting ferm on sessionstore, was stuck in resolving one of the -a records, which were only merged in a subsequent step (T215883) | [production] | 
            
  | 08:50 | <marostegui@deploy1001> | Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 48s) | [production] |