|
2022-03-16
ยง
|
| 10:55 |
<vgutierrez> |
rolling upgrade to HAProxy 2.4.15 on cache nodes |
[production] |
| 10:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P22686 and previous config saved to /var/cache/conftool/dbconfig/20220316-104858-marostegui.json |
[production] |
| 10:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T297189)', diff saved to https://phabricator.wikimedia.org/P22685 and previous config saved to /var/cache/conftool/dbconfig/20220316-104637-marostegui.json |
[production] |
| 10:42 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1024.eqiad.wmnet with OS bullseye |
[production] |
| 10:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:40 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P22684 and previous config saved to /var/cache/conftool/dbconfig/20220316-103353-marostegui.json |
[production] |
| 10:28 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1024.eqiad.wmnet with OS bullseye |
[production] |
| 10:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1181 (T298294)', diff saved to https://phabricator.wikimedia.org/P22683 and previous config saved to /var/cache/conftool/dbconfig/20220316-101848-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1181 (T298294)', diff saved to https://phabricator.wikimedia.org/P22682 and previous config saved to /var/cache/conftool/dbconfig/20220316-101729-marostegui.json |
[production] |
| 10:17 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 16:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 10:15 |
<vgutierrez> |
rolling restart of ats-tls and ats-backend to catch up on OpenSSL updates |
[production] |
| 10:15 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298294)', diff saved to https://phabricator.wikimedia.org/P22681 and previous config saved to /var/cache/conftool/dbconfig/20220316-101502-marostegui.json |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3315 (T297189)', diff saved to https://phabricator.wikimedia.org/P22680 and previous config saved to /var/cache/conftool/dbconfig/20220316-100527-marostegui.json |
[production] |
| 10:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T297189)', diff saved to https://phabricator.wikimedia.org/P22679 and previous config saved to /var/cache/conftool/dbconfig/20220316-100519-marostegui.json |
[production] |
| 10:04 |
<vgutierrez> |
vgutierrez@apt1001:~$ sudo -i reprepro --component thirdparty/haproxy24 update buster-wikimedia |
[production] |
| 10:01 |
<moritzm> |
installing openssl security updates |
[production] |
| 09:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P22678 and previous config saved to /var/cache/conftool/dbconfig/20220316-095957-marostegui.json |
[production] |
| 09:56 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1069.eqiad.wmnet with OS stretch |
[production] |
| 09:55 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1070.eqiad.wmnet with OS stretch |
[production] |
| 09:55 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1071.eqiad.wmnet with OS buster |
[production] |
| 09:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P22677 and previous config saved to /var/cache/conftool/dbconfig/20220316-095014-marostegui.json |
[production] |
| 09:46 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:46 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P22676 and previous config saved to /var/cache/conftool/dbconfig/20220316-094452-marostegui.json |
[production] |
| 09:36 |
<dcausse> |
T293862: manually restarted blazegraph on wdqs1010 with "-agentpath:/usr/lib/libjvmquake.so=1000,1,0,warn=30,touch=/tmp/jvmquake" |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P22675 and previous config saved to /var/cache/conftool/dbconfig/20220316-093509-marostegui.json |
[production] |
| 09:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T298294)', diff saved to https://phabricator.wikimedia.org/P22674 and previous config saved to /var/cache/conftool/dbconfig/20220316-092947-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3317 (T298294)', diff saved to https://phabricator.wikimedia.org/P22673 and previous config saved to /var/cache/conftool/dbconfig/20220316-092742-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T298294)', diff saved to https://phabricator.wikimedia.org/P22672 and previous config saved to /var/cache/conftool/dbconfig/20220316-092735-marostegui.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T297189)', diff saved to https://phabricator.wikimedia.org/P22671 and previous config saved to /var/cache/conftool/dbconfig/20220316-092004-marostegui.json |
[production] |
| 09:16 |
<moritzm> |
revert mx1001/mx2001 to the Bullseye version of Exim T303738 |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 T303498', diff saved to https://phabricator.wikimedia.org/P22670 and previous config saved to /var/cache/conftool/dbconfig/20220316-091533-marostegui.json |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22669 and previous config saved to /var/cache/conftool/dbconfig/20220316-091229-marostegui.json |
[production] |
| 09:09 |
<ayounsi@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1024.eqiad.wmnet with OS bullseye |
[production] |
| 08:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P22668 and previous config saved to /var/cache/conftool/dbconfig/20220316-085724-marostegui.json |
[production] |
| 08:55 |
<ayounsi@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1024.eqiad.wmnet with OS bullseye |
[production] |