|
2021-02-17
ยง
|
| 18:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1350.eqiad.wmnet |
[production] |
| 18:26 |
<effie> |
enable puppet on mw* |
[production] |
| 18:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1344.eqiad.wmnet |
[production] |
| 18:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1343.eqiad.wmnet |
[production] |
| 18:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1275.eqiad.wmnet |
[production] |
| 18:20 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@c5c4b2d]: Remove graphoid T242855 |
[production] |
| 18:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1350.eqiad.wmnet |
[production] |
| 18:14 |
<mutante> |
mw1350 - powercycled via mgmt |
[production] |
| 18:11 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1343.eqiad.wmnet |
[production] |
| 18:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1344.eqiad.wmnet |
[production] |
| 18:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1275.eqiad.wmnet |
[production] |
| 18:07 |
<effie> |
disable puppet on mw* in eqiad |
[production] |
| 17:36 |
<godog> |
roll-restart logstash7 in codfw/eqiad to apply ulogd filters - T234565 |
[production] |
| 17:28 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1035.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:26 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1035.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:16 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1343.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1344.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:13 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1343.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1275.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:12 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1344.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:10 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1275.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:07 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:05 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1350.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1350.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:58 |
<jiji@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:46 |
<godog> |
roll-restart logstash to apply ulogd filter - T234565 |
[production] |
| 16:42 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 16:41 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 16:33 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 16:32 |
<moritzm> |
installing intel-microcode security updates on buster |
[production] |
| 16:23 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 16:08 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 16:06 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@b5f4a3e]: (no justification provided) (duration: 00m 30s) |
[production] |
| 16:05 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@b5f4a3e]: (no justification provided) |
[production] |
| 15:36 |
<cdanis> |
T275028 rolling restart done; check for fetch failures once caches re-fill |
[production] |
| 15:34 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host htmldumper1001.eqiad.wmnet |
[production] |
| 15:31 |
<moritzm> |
uploaded jasper 1.900.1-debian1-2.4+deb8u6+wmf3 to apt.wikimedia.org |
[production] |
| 15:28 |
<root@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host htmldumper1001.eqiad.wmnet |
[production] |
| 15:26 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1003.eqiad.wmnet |
[production] |
| 15:20 |
<root@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host dumpsdata1003.eqiad.wmnet |
[production] |
| 15:17 |
<root@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet |
[production] |
| 15:13 |
<root@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet |
[production] |
| 14:26 |
<cdanis> |
starting rolling restart of cp-upload@eqsin varnish-fe T275028 |
[production] |
| 13:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14396 and previous config saved to /var/cache/conftool/dbconfig/20210217-135533-root.json |
[production] |
| 13:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 80%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14395 and previous config saved to /var/cache/conftool/dbconfig/20210217-134030-root.json |
[production] |
| 13:30 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 13:30 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 13:28 |
<moritzm> |
installing libzstd security updates on Buster |
[production] |
| 13:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1172 (re)pooling @ 60%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14393 and previous config saved to /var/cache/conftool/dbconfig/20210217-132526-root.json |
[production] |
| 13:19 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:664593|Enable Wikibase Repo ID generator rate limiting on Wikidata (T272032)]] (duration: 01m 11s) |
[production] |