| 2021-09-29
      
      ยง | 
    
  | 20:34 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 20:31 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 20:22 | <jhuneidi@deploy1002> | Synchronized php: group1 wikis to 1.38.0-wmf.2  refs T281166 (duration: 01m 08s) | [production] | 
            
  | 20:21 | <jhuneidi@deploy1002> | rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.2  refs T281166 | [production] | 
            
  | 20:20 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 20:16 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 20:16 | <jhuneidi@deploy1002> | Finished scap: Fix pywikibot feature detection (duration: 13m 38s) | [production] | 
            
  | 20:02 | <jhuneidi@deploy1002> | Started scap: Fix pywikibot feature detection | [production] | 
            
  | 19:17 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:13 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:06 | <legoktm@deploy1002> | Synchronized php-1.38.0-wmf.2/extensions/timeline/scripts/renderTimeline.sh: Fix passing temp directory to EasyTimeline.pl (duration: 01m 07s) | [production] | 
            
  | 18:59 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:55 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:52 | <dancy@deploy1002> | Synchronized php-1.38.0-wmf.2/skins/MinervaNeue/resources/skins.minerva.base.styles/ui.less: Backport: [[gerrit:724787|Search header should be vertically centered, not top aligned(take 2) (T292071)]] (duration: 01m 08s) | [production] | 
            
  | 17:26 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 17:22 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 17:14 | <ladsgroup@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724776|Fully enable change dispatching via jobs on test wikis]], Part I (duration: 01m 09s) | [production] | 
            
  | 17:13 | <ladsgroup@deploy1002> | Synchronized wmf-config/Wikibase.php: Config: [[gerrit:724776|Fully enable change dispatching via jobs on test wikis]], Part I (duration: 01m 07s) | [production] | 
            
  | 16:52 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:47 | <pt1979@cumin2002> | END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host thumbor2006.codfw.wmnet | [production] | 
            
  | 16:47 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:43 | <akosiaris> | start hbal -L -G row_B -X on ganeti01.svc.codfw.wmnet . Rows C and D are fine | [production] | 
            
  | 16:42 | <akosiaris> | start hbal -L -G row_A -X on ganeti01.svc.codfw.wmnet | [production] | 
            
  | 16:40 | <akosiaris> | migrate kubemaster2001 off ganeti2007 and to ganeti2008 due to memory starvation on ganeti2007 | [production] | 
            
  | 16:37 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:34 | <pt1979@cumin2002> | START - Cookbook sre.experimental.reimage for host thumbor2006.codfw.wmnet | [production] | 
            
  | 16:33 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:25 | <lucaswerkmeister-wmde@deploy1002> | Synchronized php-1.38.0-wmf.2/extensions/WikimediaBadges/: Backport: [[gerrit:724561|Handle missing items in WikibaseClientSiteLinksForItemHandler (T291953)]] (duration: 01m 08s) | [production] | 
            
  | 16:24 | <lucaswerkmeister-wmde@deploy1002> | Synchronized php-1.38.0-wmf.1/extensions/WikimediaBadges/: Backport: [[gerrit:724560|Handle missing items in WikibaseClientSiteLinksForItemHandler (T291953)]] (duration: 01m 10s) | [production] | 
            
  | 15:58 | <pt1979@cumin2002> | END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host thumbor2006.codfw.wmnet | [production] | 
            
  | 15:53 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 15:50 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 15:45 | <Amir1> | disabled cron dispatching for mediawikiwiki | [production] | 
            
  | 15:44 | <ladsgroup@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724765|Enable change dispatching via jobs in wikidatawiki (T48643)]] (duration: 01m 08s) | [production] | 
            
  | 15:44 | <pt1979@cumin2002> | START - Cookbook sre.experimental.reimage for host thumbor2006.codfw.wmnet | [production] | 
            
  | 15:39 | <filippo@cumin1001> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2035.codfw.wmnet | [production] | 
            
  | 15:39 | <ladsgroup@deploy1002> | Synchronized php-1.38.0-wmf.2/extensions/Wikibase/client: Backport: [[gerrit:724558|Track time until dispatched recent changes are inserted (T291962)]] (duration: 01m 10s) | [production] | 
            
  | 15:24 | <pt1979@cumin2002> | END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host thumbor2006.codfw.wmnet | [production] | 
            
  | 15:22 | <filippo@cumin1001> | START - Cookbook sre.hosts.reboot-single for host ms-be2035.codfw.wmnet | [production] | 
            
  | 15:21 | <jmm@cumin2002> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet | [production] | 
            
  | 15:12 | <jmm@cumin2002> | START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet | [production] | 
            
  | 15:02 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . | [production] | 
            
  | 14:51 | <jmm@cumin2002> | END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet | [production] | 
            
  | 14:45 | <jmm@cumin2002> | START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet | [production] | 
            
  | 14:38 | <jmm@cumin2002> | END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | [production] | 
            
  | 14:35 | <jmm@cumin2002> | START - Cookbook sre.dns.netbox | [production] | 
            
  | 14:25 | <jmm@cumin2002> | END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2001.codfw.wmnet | [production] | 
            
  | 14:17 | <jmm@cumin2002> | START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet | [production] | 
            
  | 14:08 | <dcausse@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . | [production] | 
            
  | 14:07 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | [production] |