|
2020-04-02
§
|
| 16:34 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:34 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 01m 05s) |
[production] |
| 16:33 |
<jforrester@deploy1001> |
sync-file aborted: T249014 [siwiki] Change wgSitename to drop the ',' (duration: 00m 00s) |
[production] |
| 16:32 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T249014 [siwiki] Change wgSitename to drop the ',' (duration: 01m 07s) |
[production] |
| 16:30 |
<joal@deploy1001> |
Started deploy [analytics/refinery@5b254c8]: Regular analytics weekly train [analytics/refinery@5b254c8] |
[production] |
| 16:19 |
<XioNoX> |
upgrade netflow4001's fastnetmon to 1.1.4 - T240658 |
[production] |
| 14:56 |
<XioNoX> |
push new test switch config for cloudvirt2001 - T248425 |
[production] |
| 14:33 |
<vgutierrez> |
Enable inbound TLSv1.3 in upload@codfw - T170567 |
[production] |
| 14:33 |
<vgutierrez> |
Enable TLS Session tickets in codfw - T245616 |
[production] |
| 14:24 |
<jbond42> |
updating bluez on ganeti and cloudvirt |
[production] |
| 14:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10865 and previous config saved to /var/cache/conftool/dbconfig/20200402-142338-marostegui.json |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10864 and previous config saved to /var/cache/conftool/dbconfig/20200402-141802-marostegui.json |
[production] |
| 14:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10863 and previous config saved to /var/cache/conftool/dbconfig/20200402-141335-marostegui.json |
[production] |
| 14:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 after schema change', diff saved to https://phabricator.wikimedia.org/P10862 and previous config saved to /var/cache/conftool/dbconfig/20200402-141149-marostegui.json |
[production] |
| 13:50 |
<marostegui> |
Compress wbqc_constraints on testcommonswiki and commonswiki (empty tables) - T248967 |
[production] |
| 13:44 |
<vgutierrez> |
update puppet compiler facts |
[production] |
| 13:40 |
<marostegui> |
Deploy schema change on db1111 |
[production] |
| 13:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111 for schema change', diff saved to https://phabricator.wikimedia.org/P10861 and previous config saved to /var/cache/conftool/dbconfig/20200402-133956-marostegui.json |
[production] |
| 13:32 |
<gehel> |
OSM data reimport on maps2004 - T249086 |
[production] |
| 12:55 |
<mutante> |
mw1390 - mw1399 - pooled and active but status "staged" in netbox, fixing to 'active' |
[production] |
| 12:52 |
<mutante> |
mw1297 - is pooled and serving traffic but status "staged" in netbox. set to "active" |
[production] |
| 11:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087 after schema change', diff saved to https://phabricator.wikimedia.org/P10858 and previous config saved to /var/cache/conftool/dbconfig/20200402-114020-marostegui.json |
[production] |
| 10:56 |
<mutante> |
decom planet1001 (T248863) |
[production] |
| 10:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:55 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:19 |
<marostegui> |
Deploy schema change on db1087, this will generate lag on s8 on wiki replicas |
[production] |
| 10:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for schema change', diff saved to https://phabricator.wikimedia.org/P10857 and previous config saved to /var/cache/conftool/dbconfig/20200402-101920-marostegui.json |
[production] |
| 10:17 |
<elukey> |
set up TLS encryption for all pmacct instances on netflow* to Kafka Jumbo |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1104 after schema change', diff saved to https://phabricator.wikimedia.org/P10856 and previous config saved to /var/cache/conftool/dbconfig/20200402-101747-marostegui.json |
[production] |
| 09:47 |
<marostegui> |
Remove haproxy@10.64.37.14 from labsdb hosts - T231280 T248944 |
[production] |
| 09:40 |
<gehel> |
CORRECTION: depool maps2004 for data reimport - T249086 |
[production] |
| 09:40 |
<gehel> |
depool wdqs2004 for data reimport - T249086 |
[production] |
| 09:33 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) (duration: 00m 18s) |
[production] |
| 09:32 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@9f2ba2c]: (no justification provided) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Finished deploy [docker-pkg/deploy@4f86d77]: (no justification provided) (duration: 00m 09s) |
[production] |
| 09:28 |
<oblivian@deploy1001> |
Started deploy [docker-pkg/deploy@4f86d77]: (no justification provided) |
[production] |
| 08:51 |
<marostegui> |
Deploy schema change db1104 |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104 for schema change', diff saved to https://phabricator.wikimedia.org/P10854 and previous config saved to /var/cache/conftool/dbconfig/20200402-085057-marostegui.json |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1092 after schema change', diff saved to https://phabricator.wikimedia.org/P10853 and previous config saved to /var/cache/conftool/dbconfig/20200402-085019-marostegui.json |
[production] |
| 08:28 |
<gehel> |
repooling wdqs1006 - catched up on lag |
[production] |
| 08:22 |
<vgutierrez> |
Enable inbound TLSv1.3 in upload@esams - T170567 |
[production] |
| 08:21 |
<vgutierrez> |
Enable TLS Session tickets in esams - T245616 |
[production] |
| 07:45 |
<moritzm> |
bounced ferm on ms-be1040 |
[production] |
| 07:27 |
<marostegui> |
Deploy schema change on db1092 |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P10850 and previous config saved to /var/cache/conftool/dbconfig/20200402-072730-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P10849 and previous config saved to /var/cache/conftool/dbconfig/20200402-072500-marostegui.json |
[production] |
| 05:49 |
<marostegui> |
Deploy schema change on db1101:3318 |
[production] |
| 05:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P10848 and previous config saved to /var/cache/conftool/dbconfig/20200402-054931-marostegui.json |
[production] |
| 05:29 |
<elukey> |
powercycle analytics1045 (host not responsive to ssh, weird chars showed in mgmt serial console) |
[production] |