| 2021-09-23
      
      § | 
    
  | 08:12 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 08:04 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 08:04 | <legoktm@deploy1002> | Synchronized wmf-config/CommonSettings.php: Have SyntaxHighlight use Shellbox service on group0 wikis (2/2) (T289227) (duration: 01m 05s) | [production] | 
            
  | 08:02 | <legoktm@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Have SyntaxHighlight use Shellbox service on group0 wikis (1/2) (T289227) (duration: 01m 06s) | [production] | 
            
  | 08:01 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 07:54 | <legoktm@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Rename $wmgUseGeSHi to $wmgUseSyntaxHighlight (3/3) (duration: 01m 05s) | [production] | 
            
  | 07:52 | <legoktm@deploy1002> | Synchronized wmf-config/CommonSettings.php: Rename $wmgUseGeSHi to $wmgUseSyntaxHighlight (2/3) (duration: 01m 05s) | [production] | 
            
  | 07:49 | <legoktm@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Rename $wmgUseGeSHi to $wmgUseSyntaxHighlight (1/3) (duration: 01m 06s) | [production] | 
            
  | 07:10 | <tgr> | running `mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=$WIKI --search-index --db-table --statsd` for growthexperiments.dblist wikis | [production] | 
            
  | 07:01 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | [production] | 
            
  | 07:01 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | [production] | 
            
  | 06:59 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | [production] | 
            
  | 06:59 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | [production] | 
            
  | 06:57 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. | [production] | 
            
  | 06:57 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. | [production] | 
            
  | 06:56 | <marostegui> | Upgrade db2116 | [production] | 
            
  | 06:55 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. | [production] | 
            
  | 06:55 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. | [production] | 
            
  | 06:55 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | [production] | 
            
  | 06:55 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | [production] | 
            
  | 06:53 | <marostegui> | Upgrade db2085, db2088 and db2092 | [production] | 
            
  | 05:24 | <marostegui> | Optimize ruwiki.logging on codfw T286102 | [production] | 
            
  | 02:55 | <eileen> | civicrm revision changed from 14658445a2 to 18228490ae, config revision is 77cb7ec866 | [production] | 
            
  | 02:06 | <RoanKattouw> | Deployed patch for T291600 | [production] | 
            
  | 01:05 | <eileen> | tools revision changed from 1d67c52c12 to d90f4c91ee | [production] | 
            
  | 00:35 | <catrope@deploy1002> | Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s) | [production] | 
            
  | 00:29 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 00:27 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  
    | 2021-09-22
      
      § | 
    
  | 22:51 | <mutante> | mx2001 - re-enabled puppet | [production] | 
            
  | 20:48 | <ryankemper> | [WDQS] After puppet-merging, running puppet on `miscweb*`, and doing a `ryankemper@mwmaint1002:~$ echo 'https://query.wikidata.org/querybuilder' | mwscript purgeList.php`, https://query.wikidata.org/querybuilder is working properly again | [production] | 
            
  | 20:39 | <ryankemper> | [WDQS] Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/722958/ which should (hopefully) resolve an issue where https://query.wikidata.org/querybuilder gives a 404, whereas https://query.wikidata.org/querybuilder/ works (due to the trailing slash avoiding the rewrite regex) | [production] | 
            
  | 20:38 | <ryankemper> | `[WCQS]` `wcqs1001.eqiad.wmnet` is reachable again following the powercycle | [production] | 
            
  | 20:20 | <ryankemper> | `[WCQS]` Ran `racadm>>racadm serveraction powercycle` on `wcqs1001.mgmt.eqiad.wmnet` | [production] | 
            
  | 20:18 | <ryankemper> | `[WCQS]` `wcqs1001` is ssh unreachable (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=wcqs1001&service=SSH), will try restarting from mgmt console | [production] | 
            
  | 19:29 | <dduvall> | 1.38.0-wmf.1 promoted to group1. no new errors or rising error rates (T281165) | [production] | 
            
  | 19:22 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:20 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:20 | <dduvall@deploy1002> | Synchronized php: group1 wikis to 1.38.0-wmf.1 (duration: 01m 11s) | [production] | 
            
  | 19:18 | <dduvall@deploy1002> | rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.1 | [production] | 
            
  | 19:12 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:11 | <dduvall@deploy1002> | Synchronized php-1.38.0-wmf.1/extensions/CentralAuth: Backport: [[gerrit:722896|Avoid $wgUser deprecation warnings (T291515)]] (duration: 01m 06s) | [production] | 
            
  | 19:10 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:33 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:32 | <legoktm@deploy1002> | Synchronized php-1.38.0-wmf.1/extensions/GrowthExperiments/modules/help/ext.growthExperiments.PostEditPanel.js: Post-edit Panel: Set task.pageviews to null rather than undefined (T291510) (duration: 01m 05s) | [production] | 
            
  | 18:31 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:23 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:21 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:13 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:12 | <legoktm@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: logging: send DuplicateParse bucket to Logstash (duration: 01m 05s) | [production] | 
            
  | 18:11 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] |