| 
      
        2025-09-22
      
      ยง
     | 
  
    
  | 12:40 | 
  <dreamyjazz@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1189869|SI: Add a configuration flag to hide SI even if the feature is enabled (T405076)]], [[gerrit:1189890|CheckUser: Enable SI on enwiki and frwiki while hiding special page (T405109)]] | 
  [production] | 
            
  | 12:39 | 
  <dcaro@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api | 
  [toolsbeta] | 
            
  | 12:38 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet | 
  [production] | 
            
  | 12:34 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet | 
  [production] | 
            
  | 12:34 | 
  <dcaro@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.component.deploy for component components-api | 
  [toolsbeta] | 
            
  | 12:28 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet | 
  [production] | 
            
  | 12:17 | 
  <fceratto@cumin1002> | 
  END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis tokwiki, mswikiquote, thwikimedia in section s5 | 
  [production] | 
            
  | 12:08 | 
  <wmbot~godog@r5> | 
  END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) | 
  [testlabs] | 
            
  | 11:58 | 
  <wmbot~godog@r5> | 
  START - Cookbook wmcs.nfs.add_server | 
  [testlabs] | 
            
  | 11:50 | 
  <fceratto@cumin1002> | 
  START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis tokwiki, mswikiquote, thwikimedia in section s5 | 
  [production] | 
            
  | 11:41 | 
  <moritzm> | 
  upgrading Envoy on puppetboard hosts T403663 | 
  [production] | 
            
  | 11:41 | 
  <wmbot~godog@r5> | 
  END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) | 
  [testlabs] | 
            
  | 11:41 | 
  <fceratto@cumin1002> | 
  END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Managing sanitization for wikis tokwiki, mswikiquote, thwikimedia in section s5 | 
  [production] | 
            
  | 11:30 | 
  <wmbot~godog@r5> | 
  START - Cookbook wmcs.nfs.add_server | 
  [testlabs] | 
            
  | 11:19 | 
  <zabe@deploy1003> | 
  mwscript-k8s job started: extensions/CentralAuth/maintenance/attachAccount.php --wiki=thwikimedia --userlist users.txt | 
  [production] | 
            
  | 11:15 | 
  <zabe@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1189981|Attach thwikimedia to SUL (T400001)]], [[gerrit:1189985|Set timezone and project namespace for thwikimedia (T400001)]] (duration: 11m 55s) | 
  [production] | 
            
  | 11:09 | 
  <zabe@deploy1003> | 
  zabe: Continuing with sync | 
  [production] | 
            
  | 11:08 | 
  <zabe@deploy1003> | 
  zabe: Backport for [[gerrit:1189981|Attach thwikimedia to SUL (T400001)]], [[gerrit:1189985|Set timezone and project namespace for thwikimedia (T400001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 11:03 | 
  <zabe@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1189981|Attach thwikimedia to SUL (T400001)]], [[gerrit:1189985|Set timezone and project namespace for thwikimedia (T400001)]] | 
  [production] | 
            
  | 10:48 | 
  <James_F> | 
  Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job | 
  [releng] | 
            
  | 10:48 | 
  <fceratto@cumin1002> | 
  START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis tokwiki, mswikiquote, thwikimedia in section s5 | 
  [production] | 
            
  | 10:48 | 
  <fceratto@cumin1002> | 
  END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Checking sanitization for wikis tokwiki in section s5 | 
  [production] | 
            
  | 10:43 | 
  <fceratto@cumin1002> | 
  START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis tokwiki in section s5 | 
  [production] | 
            
  | 10:40 | 
  <James_F> | 
  Zuul: [mediawiki/extensions/CheckUser] Remove dependency on GuidedTour, for T264359 | 
  [releng] | 
            
  | 10:38 | 
  <btullis@cumin1003> | 
  END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-backup-datanode[1032,1034-1046].eqiad.wmnet | 
  [production] | 
            
  | 10:38 | 
  <btullis@cumin1003> | 
  END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | 
  [production] | 
            
  | 10:38 | 
  <btullis@cumin1003> | 
  END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-datanode[1032,1034-1046].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" | 
  [production] | 
            
  | 10:36 | 
  <btullis@cumin1003> | 
  START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-datanode[1032,1034-1046].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" | 
  [production] | 
            
  | 10:32 | 
  <btullis@cumin1003> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 10:29 | 
  <fabfur> | 
  temporary updating haproxykafka version on cp5021 to check for non-parsable character (T404427) | 
  [production] | 
            
  | 10:12 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps1011.eqiad.wmnet with OS bookworm | 
  [production] | 
            
  | 10:10 | 
  <James_F> | 
  sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-docs-publish/ # T405166 | 
  [releng] | 
            
  | 10:08 | 
  <moritzm> | 
  installing qemu security updates | 
  [production] | 
            
  | 09:59 | 
  <joelyrookewmde> | 
  Finished populateSitesTable for mswikiquote (T404704) | 
  [production] | 
            
  | 09:56 | 
  <moritzm> | 
  installing modsecurity-apache security updates | 
  [production] | 
            
  | 09:53 | 
  <jmm@cumin2002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps1011.eqiad.wmnet with reason: host reimage | 
  [production] | 
            
  | 09:46 | 
  <jmm@cumin2002> | 
  START - Cookbook sre.hosts.downtime for 2:00:00 on maps1011.eqiad.wmnet with reason: host reimage | 
  [production] | 
            
  | 09:45 | 
  <moritzm> | 
  installing imagemagick security updates | 
  [production] | 
            
  | 09:43 | 
  <ladsgroup@deploy1003> | 
  Finished scap sync-world: Backport for [[gerrit:1190205|User: Reduce locking severity of ::getInstanceForUpdate()]] (duration: 11m 59s) | 
  [production] | 
            
  | 09:39 | 
  <klausman@cumin2002> | 
  END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool inference in eqiad: maintenance | 
  [production] | 
            
  | 09:38 | 
  <ladsgroup@deploy1003> | 
  ladsgroup: Continuing with sync | 
  [production] | 
            
  | 09:37 | 
  <ladsgroup@deploy1003> | 
  ladsgroup: Backport for [[gerrit:1190205|User: Reduce locking severity of ::getInstanceForUpdate()]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. | 
  [production] | 
            
  | 09:36 | 
  <joelyrookewmde@deploy1003> | 
  mwscript-k8s job started: foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https  # T404704 | 
  [production] | 
            
  | 09:34 | 
  <klausman@cumin2002> | 
  START - Cookbook sre.discovery.service-route pool inference in eqiad: maintenance | 
  [production] | 
            
  | 09:31 | 
  <ladsgroup@deploy1003> | 
  Started scap sync-world: Backport for [[gerrit:1190205|User: Reduce locking severity of ::getInstanceForUpdate()]] | 
  [production] | 
            
  | 09:30 | 
  <klausman@deploy1003> | 
  helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 09:29 | 
  <btullis@cumin1003> | 
  START - Cookbook sre.hosts.decommission for hosts an-backup-datanode[1032,1034-1046].eqiad.wmnet | 
  [production] | 
            
  | 09:29 | 
  <moritzm> | 
  installing libfastjson security updates | 
  [production] | 
            
  | 09:28 | 
  <bwojtowicz@deploy1003> | 
  helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | 
  [production] | 
            
  | 09:27 | 
  <klausman@cumin2002> | 
  END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool inference in eqiad: maintenance | 
  [production] |