|
2021-04-19
ยง
|
| 11:01 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudgw1001.eqiad.wmnet |
[production] |
| 10:46 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:681008| Bumping portals to master (T128546)]] (duration: 00m 57s) |
[production] |
| 10:45 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:681008| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 10:34 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on theemin.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on theemin.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:24 |
<hnowlan> |
imported 1.16.3 into envoy-future |
[production] |
| 10:22 |
<moritzm> |
reimaging theemin to bullseye |
[production] |
| 10:15 |
<dcausse> |
depooling wdqs1005 |
[production] |
| 10:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1156.eqiad.wmnet with reason: REIMAGE |
[production] |
| 10:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1156.eqiad.wmnet with reason: REIMAGE |
[production] |
| 10:05 |
<arturo> |
aborrero@apt1001:~ $ sudo -i reprepro --component thirdparty/kubeadm-k8s-1-18 update buster-wikimedia |
[production] |
| 10:04 |
<arturo> |
aborrero@apt1001:~ $ sudo -i reprepro --delete clearvanished (remove old buster-wikimedia|thirdparty/kubeadm-k8s-1-15,16 repos and packages) |
[production] |
| 09:56 |
<ema> |
cp3051: varnish-frontend-restart to apply exp policy settings changes starting from empty cache T275809 |
[production] |
| 09:53 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thanos-fe2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:51 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on thanos-fe2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 09:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1161.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:39 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1161.eqiad.wmnet with reason: REIMAGE |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15454 and previous config saved to /var/cache/conftool/dbconfig/20210419-092251-root.json |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1161 T280492', diff saved to https://phabricator.wikimedia.org/P15453 and previous config saved to /var/cache/conftool/dbconfig/20210419-092234-marostegui.json |
[production] |
| 09:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 100%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15452 and previous config saved to /var/cache/conftool/dbconfig/20210419-091535-root.json |
[production] |
| 09:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 90%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15451 and previous config saved to /var/cache/conftool/dbconfig/20210419-090747-root.json |
[production] |
| 09:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 75%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15450 and previous config saved to /var/cache/conftool/dbconfig/20210419-090031-root.json |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 80%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15449 and previous config saved to /var/cache/conftool/dbconfig/20210419-085243-root.json |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 T272008', diff saved to https://phabricator.wikimedia.org/P15448 and previous config saved to /var/cache/conftool/dbconfig/20210419-084834-marostegui.json |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 50%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15447 and previous config saved to /var/cache/conftool/dbconfig/20210419-084528-root.json |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 T272008', diff saved to https://phabricator.wikimedia.org/P15446 and previous config saved to /var/cache/conftool/dbconfig/20210419-084523-marostegui.json |
[production] |
| 08:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 70%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15445 and previous config saved to /var/cache/conftool/dbconfig/20210419-083740-root.json |
[production] |
| 08:35 |
<ema> |
restart debmonitor-client.service on cp4030, dns5002, an-worker1106 T280484 |
[production] |
| 08:34 |
<marostegui> |
Testing log |
[production] |
| 08:30 |
<marostegui> |
Restart m5 master - wikitech will go down T279657 |
[production] |
| 08:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1085 (re)pooling @ 100%: Repool db1085', diff saved to https://phabricator.wikimedia.org/P15444 and previous config saved to /var/cache/conftool/dbconfig/20210419-083021-root.json |
[production] |
| 08:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 25%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15443 and previous config saved to /var/cache/conftool/dbconfig/20210419-083018-root.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079 T272008', diff saved to https://phabricator.wikimedia.org/P15442 and previous config saved to /var/cache/conftool/dbconfig/20210419-082559-marostegui.json |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 60%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15441 and previous config saved to /var/cache/conftool/dbconfig/20210419-082236-root.json |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1082 (re)pooling @ 100%: Repool db1082', diff saved to https://phabricator.wikimedia.org/P15440 and previous config saved to /var/cache/conftool/dbconfig/20210419-082000-root.json |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1085 (re)pooling @ 75%: Repool db1085', diff saved to https://phabricator.wikimedia.org/P15439 and previous config saved to /var/cache/conftool/dbconfig/20210419-081517-root.json |
[production] |
| 08:07 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on labstore1004.eqiad.wmnet with reason: Restarting mysql |
[production] |
| 08:07 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on labstore1004.eqiad.wmnet with reason: Restarting mysql |
[production] |
| 08:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15438 and previous config saved to /var/cache/conftool/dbconfig/20210419-080732-root.json |
[production] |
| 08:07 |
<godog> |
swift eqiad-prod: less weight for ms-be[1019-1026] / more weight to ms-be106[0-3] - T272836 |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1082 (re)pooling @ 75%: Repool db1082', diff saved to https://phabricator.wikimedia.org/P15437 and previous config saved to /var/cache/conftool/dbconfig/20210419-080456-root.json |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15436 and previous config saved to /var/cache/conftool/dbconfig/20210419-080454-root.json |
[production] |
| 08:03 |
<moritzm> |
installing python-bleach security updates |
[production] |
| 08:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1085 (re)pooling @ 50%: Repool db1085', diff saved to https://phabricator.wikimedia.org/P15435 and previous config saved to /var/cache/conftool/dbconfig/20210419-080013-root.json |
[production] |
| 07:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 40%: Slowly pool db1179 for the first time in s3 T275633', diff saved to https://phabricator.wikimedia.org/P15434 and previous config saved to /var/cache/conftool/dbconfig/20210419-075229-root.json |
[production] |
| 07:51 |
<moritzm> |
upgrade mwdebug2002 to PHP 7.2.34 |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1082 (re)pooling @ 50%: Repool db1082', diff saved to https://phabricator.wikimedia.org/P15433 and previous config saved to /var/cache/conftool/dbconfig/20210419-074953-root.json |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool db1121', diff saved to https://phabricator.wikimedia.org/P15432 and previous config saved to /var/cache/conftool/dbconfig/20210419-074950-root.json |
[production] |
| 07:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1085 (re)pooling @ 25%: Repool db1085', diff saved to https://phabricator.wikimedia.org/P15431 and previous config saved to /var/cache/conftool/dbconfig/20210419-074510-root.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1085 T272008', diff saved to https://phabricator.wikimedia.org/P15430 and previous config saved to /var/cache/conftool/dbconfig/20210419-074155-marostegui.json |
[production] |