|
2025-06-10
ยง
|
| 09:46 |
<moritzm> |
installing postgresql-15 security updates |
[production] |
| 09:45 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) es2033 - Upgrading es2033.codfw.wmnet |
[production] |
| 09:45 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.depool es2033 - Upgrading es2033.codfw.wmnet |
[production] |
| 09:44 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.upgrade for es2033.codfw.wmnet |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P77450 and previous config saved to /var/cache/conftool/dbconfig/20250610-094429-root.json |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2031 to es2 master T395241', diff saved to https://phabricator.wikimedia.org/P77449 and previous config saved to /var/cache/conftool/dbconfig/20250610-094401-root.json |
[production] |
| 09:43 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host ganeti2038.codfw.wmnet |
[production] |
| 09:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77448 and previous config saved to /var/cache/conftool/dbconfig/20250610-094345-root.json |
[production] |
| 09:43 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2032.codfw.wmnet |
[production] |
| 09:39 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:39 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1187 T395989', diff saved to https://phabricator.wikimedia.org/P77447 and previous config saved to /var/cache/conftool/dbconfig/20250610-093846-marostegui.json |
[production] |
| 09:38 |
<akosiaris@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 09:37 |
<akosiaris@deploy1003> |
helmfile [codfw] START helmfile.d/services/wikifunctions: apply |
[production] |
| 09:37 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) es2032 - Upgrading es2032.codfw.wmnet |
[production] |
| 09:37 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.depool es2032 - Upgrading es2032.codfw.wmnet |
[production] |
| 09:36 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.upgrade for es2032.codfw.wmnet |
[production] |
| 09:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2030 to es1 master T395241', diff saved to https://phabricator.wikimedia.org/P77445 and previous config saved to /var/cache/conftool/dbconfig/20250610-093628-root.json |
[production] |
| 09:36 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:35 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P77444 and previous config saved to /var/cache/conftool/dbconfig/20250610-093518-fceratto.json |
[production] |
| 09:33 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2038.codfw.wmnet |
[production] |
| 09:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2148 (T396130)', diff saved to https://phabricator.wikimedia.org/P77443 and previous config saved to /var/cache/conftool/dbconfig/20250610-093252-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance |
[production] |
| 09:31 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1230.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:31 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:27 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:26 |
<fnegri@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1019.eqiad.wmnet with reason: Upgrading clouddbs T394372 |
[production] |
| 09:26 |
<jynus> |
upgrade db2197 to MariaDB 10.11 T394487 |
[production] |
| 09:24 |
<fnegri@cumin1003> |
conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet |
[production] |
| 09:20 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170 (T395241)', diff saved to https://phabricator.wikimedia.org/P77442 and previous config saved to /var/cache/conftool/dbconfig/20250610-092011-fceratto.json |
[production] |
| 09:17 |
<jynus@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2197.codfw.wmnet,dbprov2003.codfw.wmnet with reason: Downtime hosts for MariaDB 10.11 upgrade |
[production] |
| 09:15 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:11 |
<taavi> |
delete old proxy-03,04 instances T379175 |
[project-proxy] |
| 09:10 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1170 (T395241)', diff saved to https://phabricator.wikimedia.org/P77441 and previous config saved to /var/cache/conftool/dbconfig/20250610-091040-fceratto.json |
[production] |
| 09:10 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T395241)', diff saved to https://phabricator.wikimedia.org/P77440 and previous config saved to /var/cache/conftool/dbconfig/20250610-091016-fceratto.json |
[production] |
| 09:08 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:07 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1245.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:06 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1216.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210 (T396130)', diff saved to https://phabricator.wikimedia.org/P77439 and previous config saved to /var/cache/conftool/dbconfig/20250610-090635-marostegui.json |
[production] |
| 08:55 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P77438 and previous config saved to /var/cache/conftool/dbconfig/20250610-085508-fceratto.json |
[production] |
| 08:54 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 08:53 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 08:53 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 08:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P77437 and previous config saved to /var/cache/conftool/dbconfig/20250610-085128-marostegui.json |
[production] |
| 08:40 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 08:40 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P77436 and previous config saved to /var/cache/conftool/dbconfig/20250610-084002-fceratto.json |
[production] |
| 08:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P77435 and previous config saved to /var/cache/conftool/dbconfig/20250610-083622-marostegui.json |
[production] |
| 08:24 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T395241)', diff saved to https://phabricator.wikimedia.org/P77434 and previous config saved to /var/cache/conftool/dbconfig/20250610-082454-fceratto.json |
[production] |
| 08:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1210 (T396130)', diff saved to https://phabricator.wikimedia.org/P77433 and previous config saved to /var/cache/conftool/dbconfig/20250610-082114-marostegui.json |
[production] |