|
2024-07-30
ยง
|
| 22:45 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1242.eqiad.wmnet with reason: host reimage |
[production] |
| 22:41 |
<eileen> |
config revision changed from d2484ce6 to e8cc0ed6 |
[production] |
| 22:35 |
<eileen> |
config revision changed from 10ead940 to d2484ce6 |
[production] |
| 22:34 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host wikikube-worker1244.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 22:32 |
<eileen> |
civicrm upgraded from 5ac353bd to 3db16342 |
[production] |
| 22:32 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-worker1245.eqiad.wmnet with OS bullseye |
[production] |
| 22:28 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-worker1242.eqiad.wmnet with OS bullseye |
[production] |
| 22:28 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bullseye |
[production] |
| 22:28 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.reimage for host wikikube-worker1241.eqiad.wmnet with OS bullseye |
[production] |
| 21:53 |
<urbanecm@deploy1003> |
Finished scap: Backport for [[gerrit:1058258|Fix resource response to use JSON content type header (T263870)]], [[gerrit:1058259|Fix resource response to use JSON content type header (T263870)]] (duration: 08m 09s) |
[production] |
| 21:45 |
<urbanecm@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1058258|Fix resource response to use JSON content type header (T263870)]], [[gerrit:1058259|Fix resource response to use JSON content type header (T263870)]] |
[production] |
| 21:23 |
<cjming@deploy1003> |
Finished scap: Backport for [[gerrit:1046732|Deploy MetricsPlatform to beta cluster (T366234)]] (duration: 11m 41s) |
[production] |
| 21:18 |
<cjming@deploy1003> |
cjming: Continuing with sync |
[production] |
| 21:14 |
<cjming@deploy1003> |
cjming: Backport for [[gerrit:1046732|Deploy MetricsPlatform to beta cluster (T366234)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:11 |
<cjming@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1046732|Deploy MetricsPlatform to beta cluster (T366234)]] |
[production] |
| 21:06 |
<cjming@deploy1003> |
Finished scap: Backport for [[gerrit:1057976|Enable Parsoid Read Views on {en,he}wikivoyage (T365367)]] (duration: 13m 18s) |
[production] |
| 21:01 |
<cjming@deploy1003> |
cjming, cscott: Continuing with sync |
[production] |
| 20:58 |
<cjming@deploy1003> |
cjming, cscott: Backport for [[gerrit:1057976|Enable Parsoid Read Views on {en,he}wikivoyage (T365367)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:52 |
<cjming@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1057976|Enable Parsoid Read Views on {en,he}wikivoyage (T365367)]] |
[production] |
| 20:48 |
<cjming@deploy1003> |
Finished scap: Backport for [[gerrit:1058167|Add NetworkSession extension (T355267)]] (duration: 45m 08s) |
[production] |
| 20:40 |
<cjming@deploy1003> |
ebernhardson, cjming: Continuing with sync |
[production] |
| 20:38 |
<cjming@deploy1003> |
ebernhardson, cjming: Backport for [[gerrit:1058167|Add NetworkSession extension (T355267)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:16 |
<godog> |
bounce benthos@webrequest_live.service on centrallog for excessive lag |
[production] |
| 20:06 |
<topranks> |
re-enable BGP to lvs2011 on lsw1-a2-codfw (restores as primary for traffic) T370891 |
[production] |
| 20:03 |
<cjming@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1058167|Add NetworkSession extension (T355267)]] |
[production] |
| 19:58 |
<topranks> |
rebooting lvs2011 to force new network config T370891 |
[production] |
| 19:37 |
<eileen> |
civicrm upgraded from 5e72c64f to 5ac353bd |
[production] |
| 19:29 |
<topranks> |
disable BGP to lvs2011 on lsw1-a2-codfw (moves traffic to lvs2014) in advnace of vlan change T370891 |
[production] |
| 19:29 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs2011.codfw.wmnet with reason: reconfigure vlans on lvs2011 |
[production] |
| 19:29 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on lvs2011.codfw.wmnet with reason: reconfigure vlans on lvs2011 |
[production] |
| 19:28 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a2-codfw.mgmt with reason: reconfigure vlans on lvs2011 |
[production] |
| 19:28 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on lsw1-a2-codfw.mgmt with reason: reconfigure vlans on lvs2011 |
[production] |
| 19:21 |
<ryankemper@deploy1003> |
Finished deploy [wdqs/wdqs@316bf7f]: 0.3.145 (duration: 07m 59s) |
[production] |
| 19:13 |
<ryankemper@deploy1003> |
Started deploy [wdqs/wdqs@316bf7f]: 0.3.145 |
[production] |
| 18:53 |
<brennen@deploy1003> |
rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.16 refs T366961 |
[production] |
| 18:39 |
<topranks> |
re-enabling BGP to lvs2012 from lsw1-b2-codfw T370862 |
[production] |
| 18:33 |
<brennen> |
1.43.0-wmf.16 train (T366961): blockers resolved, rolling to group0 |
[production] |
| 18:31 |
<brennen@deploy1003> |
Finished scap: Backport for [[gerrit:1058211|Bump wikimedia/parsoid to 0.20.0-a16 (T371376 T371126)]] (duration: 08m 54s) |
[production] |
| 18:29 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs2012.codfw.wmnet with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:29 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lvs2012.codfw.wmnet with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:29 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lsw1-b2-codfw.mgmt with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:29 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lsw1-b2-codfw.mgmt with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:27 |
<topranks> |
rebooting lvs2012 (again) to force new network config T370862 |
[production] |
| 18:26 |
<brennen@deploy1003> |
brennen, cscott: Continuing with sync |
[production] |
| 18:24 |
<brennen@deploy1003> |
brennen, cscott: Backport for [[gerrit:1058211|Bump wikimedia/parsoid to 0.20.0-a16 (T371376 T371126)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 18:22 |
<brennen@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1058211|Bump wikimedia/parsoid to 0.20.0-a16 (T371376 T371126)]] |
[production] |
| 18:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repool db1174', diff saved to https://phabricator.wikimedia.org/P67083 and previous config saved to /var/cache/conftool/dbconfig/20240730-181331-ladsgroup.json |
[production] |
| 18:13 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs2012.codfw.wmnet with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:13 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lvs2012.codfw.wmnet with reason: reconfigure vlans on lvs2012 |
[production] |
| 18:12 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depool db1174', diff saved to https://phabricator.wikimedia.org/P67082 and previous config saved to /var/cache/conftool/dbconfig/20240730-181242-ladsgroup.json |
[production] |