| 
      
        2025-09-22
      
      §
     | 
  
    
  | 09:18 | 
  <klausman@deploy1003> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 09:16 | 
  <moritzm> | 
  prune now obsolete nginx packages from install* T329529 | 
  [production] | 
            
  | 09:14 | 
  <kharlan@deploy1003> | 
  kharlan: Backport for [[gerrit:1190191|hCaptcha: Log error message from upstream]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 09:11 | 
  <klausman@deploy1003> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 09:09 | 
  <bwojtowicz@deploy1003> | 
  helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 09:08 | 
  <kharlan@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1190191|hCaptcha: Log error message from upstream]] | 
  [production] | 
            
  | 09:06 | 
  <klausman@cumin2002> | 
  END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check 2 services: maintenance | 
  [production] | 
            
  | 09:06 | 
  <klausman@cumin2002> | 
  START - Cookbook sre.discovery.service-route check 2 services: maintenance | 
  [production] | 
            
  | 09:05 | 
  <klausman@cumin2002> | 
  END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check 2 services: maintenance | 
  [production] | 
            
  | 09:05 | 
  <klausman@cumin2002> | 
  START - Cookbook sre.discovery.service-route check 2 services: maintenance | 
  [production] | 
            
  | 09:05 | 
  <tappof> | 
  bump space for prometheus k8s-dse in eqiad | 
  [production] | 
            
  | 09:01 | 
  <klausman@deploy1003> | 
  helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. | 
  [production] | 
            
  | 09:01 | 
  <klausman@deploy1003> | 
  helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. | 
  [production] | 
            
  | 08:53 | 
  <awight> | 
  finished special dewiki sub-referencing window | 
  [production] | 
            
  | 08:53 | 
  <awight@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1190189|Fix pulsating dot not disappearing for logged in users (T403693)]] (duration: 12m 26s) | 
  [production] | 
            
  | 08:47 | 
  <awight@deploy1003> | 
  awight, wmde-fisch: Continuing with sync | 
  [production] | 
            
  | 08:46 | 
  <awight@deploy1003> | 
  awight, wmde-fisch: Backport for [[gerrit:1190189|Fix pulsating dot not disappearing for logged in users (T403693)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 08:45 | 
  <brouberol@deploy1003> | 
  helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. | 
  [production] | 
            
  | 08:44 | 
  <brouberol@deploy1003> | 
  helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. | 
  [production] | 
            
  | 08:40 | 
  <awight@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1190189|Fix pulsating dot not disappearing for logged in users (T403693)]] | 
  [production] | 
            
  | 08:33 | 
  <klausman@deploy1003> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 08:23 | 
  <awight@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1189949|Enable sub-references on dewiki (T398669)]] (duration: 17m 26s) | 
  [production] | 
            
  | 08:18 | 
  <awight@deploy1003> | 
  wmde-fisch, awight: Continuing with sync | 
  [production] | 
            
  | 08:12 | 
  <bwojtowicz@deploy1003> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 08:12 | 
  <awight@deploy1003> | 
  wmde-fisch, awight: Backport for [[gerrit:1189949|Enable sub-references on dewiki (T398669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 08:06 | 
  <awight@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1189949|Enable sub-references on dewiki (T398669)]] | 
  [production] | 
            
  | 08:02 | 
  <moritzm> | 
  upgrading Envoy on webperf hosts T403663 | 
  [production] | 
            
  | 08:02 | 
  <awight> | 
  beginning dewiki sub-referencing deployment window | 
  [production] | 
            
  | 08:00 | 
  <bwojtowicz@deploy1003> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 07:39 | 
  <phuedx@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1189877|xLab: Fix instrument to produce valid events (T404420)]] (duration: 15m 28s) | 
  [production] | 
            
  | 07:31 | 
  <phuedx@deploy1003> | 
  phuedx: Continuing with sync | 
  [production] | 
            
  | 07:30 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts install1004.wikimedia.org | 
  [production] | 
            
  | 07:30 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | 
  [production] | 
            
  | 07:30 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install1004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | 
  [production] | 
            
  | 07:29 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: install1004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | 
  [production] | 
            
  | 07:29 | 
  <phuedx@deploy1003> | 
  phuedx: Backport for [[gerrit:1189877|xLab: Fix instrument to produce valid events (T404420)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 07:23 | 
  <phuedx@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1189877|xLab: Fix instrument to produce valid events (T404420)]] | 
  [production] | 
            
  | 07:18 | 
  <kharlan@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1189879|hCaptcha: Enable on API account creation on test2wiki (T405107)]] (duration: 37m 41s) | 
  [production] | 
            
  | 07:17 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 07:08 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.hosts.decommission for hosts install1004.wikimedia.org | 
  [production] | 
            
  | 07:05 | 
  <kharlan@deploy1003> | 
  kharlan: Continuing with sync | 
  [production] | 
            
  | 07:04 | 
  <kharlan@deploy1003> | 
  kharlan: Backport for [[gerrit:1189879|hCaptcha: Enable on API account creation on test2wiki (T405107)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 06:40 | 
  <kharlan@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1189879|hCaptcha: Enable on API account creation on test2wiki (T405107)]] | 
  [production] | 
            
  | 01:29 | 
  <eileen> | 
  civicrm upgraded from 793af994 to 2c830cae | 
  [production] | 
            
  | 01:12 | 
  <mwpresync@deploy1003> | 
  Finished scap build-images: Publishing wmf/next image (duration: 12m 14s) | 
  [production] | 
            
  | 01:00 | 
  <mwpresync@deploy1003> | 
  Started scap build-images: Publishing wmf/next image | 
  [production] |