| 
      
        2024-09-20
      
      ยง
     | 
  
    
  | 16:01 | 
  <xSavitar> | 
  T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a' | 
  [production] | 
            
  | 16:01 | 
  <xSavitar> | 
  T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a' | 
  [production] | 
            
  | 14:20 | 
  <Dreamy_Jazz> | 
  Finished running`foreachwiki maintenance/fixAutoblockLogTitles.php` on a tmux session for T373929 | 
  [production] | 
            
  | 14:08 | 
  <Dreamy_Jazz> | 
  Running `foreachwiki maintenance/fixAutoblockLogTitles.php` on a tmux session for T373929 | 
  [production] | 
            
  | 14:04 | 
  <kcvelaga@deploy1003> | 
  Finished deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062 (duration: 01m 47s) | 
  [production] | 
            
  | 14:03 | 
  <kcvelaga@deploy1003> | 
  Started deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062 | 
  [production] | 
            
  | 13:20 | 
  <moritzm> | 
  uploaded debmonitor-client 0.4.0-3 for buster/bullseye/bookworm to apt.wikimedia.org T216832 | 
  [production] | 
            
  | 13:09 | 
  <jiji@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 13:09 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 12:45 | 
  <jiji@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:38 | 
  <jiji@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 12:37 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 2:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 12:36 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:24 | 
  <jiji@cumin1002> | 
  END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:18 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:16 | 
  <jiji@cumin1002> | 
  END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:11 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL | 
  [production] | 
            
  | 12:07 | 
  <jiji@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 12:07 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 2:00:00 on mw2425.codfw.wmnet with reason: reimage | 
  [production] | 
            
  | 11:58 | 
  <jiji@cumin1002> | 
  END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2424.codfw.wmnet | 
  [production] | 
            
  | 11:57 | 
  <jiji@cumin1002> | 
  START - Cookbook sre.k8s.pool-depool-node depool for host mw2424.codfw.wmnet | 
  [production] | 
            
  | 11:46 | 
  <cgoubert@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID | 
  [production] | 
            
  | 11:46 | 
  <cgoubert@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID | 
  [production] | 
            
  | 11:18 | 
  <moritzm> | 
  installing links2 updates from Bullseye point release | 
  [production] | 
            
  | 11:11 | 
  <moritzm> | 
  installing nano updates from Bullseye point update | 
  [production] | 
            
  | 11:00 | 
  <aborrero@cloudcumin1001> | 
  END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | 
  [admin] | 
            
  | 11:00 | 
  <aborrero@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | 
  [admin] | 
            
  | 10:59 | 
  <aborrero@cloudcumin1001> | 
  END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | 
  [admin] | 
            
  | 10:59 | 
  <aborrero@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | 
  [admin] | 
            
  | 10:11 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f3-eqiad | 
  [production] | 
            
  | 10:11 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device lsw1-f3-eqiad | 
  [production] | 
            
  | 10:08 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-eqiad | 
  [production] | 
            
  | 10:08 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device lsw1-f2-eqiad | 
  [production] | 
            
  | 10:08 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e3-eqiad | 
  [production] | 
            
  | 10:07 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device lsw1-e3-eqiad | 
  [production] | 
            
  | 10:07 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-eqiad | 
  [production] | 
            
  | 10:07 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device lsw1-e2-eqiad | 
  [production] | 
            
  | 10:03 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-eqiad | 
  [production] | 
            
  | 10:03 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device lsw1-e1-eqiad | 
  [production] | 
            
  | 09:42 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-f4-eqiad | 
  [production] | 
            
  | 09:40 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device cloudsw1-f4-eqiad | 
  [production] | 
            
  | 09:38 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-e4-eqiad | 
  [production] | 
            
  | 09:35 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device cloudsw1-e4-eqiad | 
  [production] | 
            
  | 09:34 | 
  <ayounsi@cumin1002> | 
  END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-b1-codfw | 
  [production] | 
            
  | 09:32 | 
  <ayounsi@cumin1002> | 
  START - Cookbook sre.network.tls for network device cloudsw1-b1-codfw | 
  [production] | 
            
  | 08:54 | 
  <btullis@cumin1002> | 
  END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster | 
  [production] | 
            
  | 08:37 | 
  <arnaudb@cumin1002> | 
  dbctl commit (dc=all): 'depool es1022 - T375257', diff saved to https://phabricator.wikimedia.org/P69377 and previous config saved to /var/cache/conftool/dbconfig/20240920-083722-arnaudb.json | 
  [production] | 
            
  | 08:32 | 
  <jgleeson|away> | 
  payments config changed from eee33e37 to e140fab5 | 
  [production] | 
            
  | 08:27 | 
  <aborrero@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T373740) | 
  [admin] | 
            
  | 08:27 | 
  <aborrero@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T373740) | 
  [admin] |