|
2025-01-30
ยง
|
| 09:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72846 and previous config saved to /var/cache/conftool/dbconfig/20250130-095706-marostegui.json |
[production] |
| 09:57 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T384592)', diff saved to https://phabricator.wikimedia.org/P72845 and previous config saved to /var/cache/conftool/dbconfig/20250130-095644-marostegui.json |
[production] |
| 09:56 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A |
[production] |
| 09:56 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2039.codfw.wmnet to cluster codfw and group A |
[production] |
| 09:56 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti2039.codfw.wmnet to cluster codfw and group A |
[production] |
| 09:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72844 and previous config saved to /var/cache/conftool/dbconfig/20250130-095524-root.json |
[production] |
| 09:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet |
[production] |
| 09:46 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Index rebuild |
[production] |
| 09:46 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet |
[production] |
| 09:46 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1183.eqiad.wmnet |
[production] |
| 09:43 |
<moritzm> |
removed profile::ssh::server::disable_nist_kex: false from Toolforge Hiera settings (so that the defaults apply like for the rest of Cloud VPS) |
[production] |
| 09:41 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2030.codfw.wmnet with OS bookworm |
[production] |
| 09:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P72843 and previous config saved to /var/cache/conftool/dbconfig/20250130-094137-marostegui.json |
[production] |
| 09:40 |
<root@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1183.eqiad.wmnet |
[production] |
| 09:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72842 and previous config saved to /var/cache/conftool/dbconfig/20250130-094018-root.json |
[production] |
| 09:39 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 09:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1183 T385147', diff saved to https://phabricator.wikimedia.org/P72841 and previous config saved to /var/cache/conftool/dbconfig/20250130-093927-marostegui.json |
[production] |
| 09:38 |
<marostegui@cumin2002> |
dbctl commit (dc=all): 'Promote db1230 to s5 primary T385147', diff saved to https://phabricator.wikimedia.org/P72840 and previous config saved to /var/cache/conftool/dbconfig/20250130-093845-marostegui.json |
[production] |
| 09:38 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 09:38 |
<marostegui> |
Starting s5 eqiad failover from db1183 to db1230 - T385147 |
[production] |
| 09:32 |
<marostegui@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385147 |
[production] |
| 09:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1230 with weight 0 T385147', diff saved to https://phabricator.wikimedia.org/P72839 and previous config saved to /var/cache/conftool/dbconfig/20250130-093221-root.json |
[production] |
| 09:28 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/kartotherian: sync |
[production] |
| 09:28 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/kartotherian: sync |
[production] |
| 09:26 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/kartotherian: sync |
[production] |
| 09:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P72838 and previous config saved to /var/cache/conftool/dbconfig/20250130-092630-marostegui.json |
[production] |
| 09:26 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/kartotherian: sync |
[production] |
| 09:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1161 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72837 and previous config saved to /var/cache/conftool/dbconfig/20250130-092513-root.json |
[production] |
| 09:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2030.codfw.wmnet with reason: host reimage |
[production] |
| 09:18 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2030.codfw.wmnet with reason: host reimage |
[production] |
| 09:16 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.14 refs T382365 |
[production] |
| 09:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T384592)', diff saved to https://phabricator.wikimedia.org/P72836 and previous config saved to /var/cache/conftool/dbconfig/20250130-091123-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72835 and previous config saved to /var/cache/conftool/dbconfig/20250130-091007-root.json |
[production] |
| 09:08 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/kartotherian: sync |
[production] |
| 09:08 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/kartotherian: sync |
[production] |
| 08:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1161 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72834 and previous config saved to /var/cache/conftool/dbconfig/20250130-085502-root.json |
[production] |
| 08:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2030.codfw.wmnet with OS bookworm |
[production] |
| 08:48 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2030.codfw.wmnet with OS bookworm |
[production] |
| 08:45 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2095,2175,2186].codfw.wmnet |
[production] |
| 08:45 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2095,2175,2186].codfw.wmnet |
[production] |
| 08:45 |
<jayme@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2095,2175,2186].codfw.wmnet |
[production] |
| 08:45 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2095,2175,2186].codfw.wmnet |
[production] |
| 08:33 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2030.codfw.wmnet with OS bookworm |
[production] |
| 08:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti2030.codfw.wmnet |
[production] |
| 08:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet |
[production] |
| 08:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet |
[production] |
| 08:09 |
<marostegui@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Rebuild and upgrade dbstore1007:s4 |
[production] |
| 07:49 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Index rebuild |
[production] |
| 07:48 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1161.eqiad.wmnet |
[production] |