|
2020-04-11
§
|
| 15:39 |
<vgutierrez> |
restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
| 15:07 |
<Krenair> |
Migrated from deployment-cache-text05 (stretch) to deployment-cache-text06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance - T250006 |
[releng] |
| 14:52 |
<Krenair> |
Migrated from deployment-cache-upload05 (stretch) to deployment-cache-upload06 (buster) - class stopped working on stretch with https://gerrit.wikimedia.org/r/c/operations/puppet/+/584553 - shut down old instance which coincidentally would turn one year old tomorrow |
[releng] |
| 09:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) |
[production] |
| 09:20 |
<elukey@cumin1001> |
START - Cookbook sre.presto.roll-restart-workers |
[production] |
| 09:19 |
<elukey> |
set hive-security: read-only for the Presto hive connector and roll restart the cluster |
[analytics] |
| 07:01 |
<vgutierrez> |
restart ats-tls on cp[1079,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
|
2020-04-10
§
|
| 23:01 |
<hashar> |
Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/588054/ |
[releng] |
| 21:33 |
<bd808> |
Rebuilding all Docker images for the Kubernetes cluster (T249843) |
[tools] |
| 21:27 |
<bd808> |
Rebuilding all Docker images for the Kubernetes cluster (T249843) |
[tools] |
| 21:12 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'db1111 seems overloaded', diff saved to https://phabricator.wikimedia.org/P10954 and previous config saved to /var/cache/conftool/dbconfig/20200410-211202-cdanis.json |
[production] |
| 21:02 |
<andrewbogott> |
deleted a bunch of canary VMS and other miscellaneous testing VMs which (I hope) were mine |
[testlabs] |
| 19:37 |
<cdanis> |
cdanis@re0.cr1-codfw> clear bfd session address 208.80.153.220 |
[production] |
| 19:36 |
<bstorm_> |
after testing deploying toollabs-webservice 0.67 to tools repos T249843 |
[tools] |
| 19:32 |
<bstorm_> |
deployed webservice 0.67 T249843 |
[toolsbeta] |
| 19:28 |
<longma> |
Updating docker-pkg files on contint2001 for https://gerrit.wikimedia.org/r/c/integration/config/+/588029 |
[releng] |
| 18:59 |
<bstorm_> |
delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test T249946 |
[toolsbeta] |
| 18:40 |
<James_F> |
Docker: Publishing composer-php70:0.2.0 and cascade. |
[releng] |
| 17:12 |
<twentyafterfour> |
deploying docker image on contint2001 with `tox -e fabric -- deploy_docker` |
[releng] |
| 17:08 |
<James_F> |
Docker: Publishing release-notes:0.0.5 on contint2001 |
[releng] |
| 16:46 |
<hashar> |
contint1001: deleting docker-pkg maintained images. They are in the registry anyway. # T224591 |
[releng] |
| 16:37 |
<wm-bot> |
<peterbowman> Deploy 51e4941 (update wiki-java, MissingRefs webapp) |
[tools.pbbot] |
| 16:31 |
<elukey> |
enable TLS from kafkatee to Kafka on analytics1030 (test instance) |
[analytics] |
| 15:45 |
<elukey> |
migrate data_purge timers from an-coord1001 to an-launcher1001 |
[analytics] |
| 15:03 |
<vgutierrez> |
restart ats-tls on cp1083 and cp1085 - T249335 |
[production] |
| 14:53 |
<arturo> |
live-hacking tools-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/587991 for T249837 |
[tools] |
| 13:14 |
<hashar@deploy1001> |
Finished deploy [zuul/deploy@4a69913]: (no justification provided) (duration: 00m 40s) |
[production] |
| 13:14 |
<hashar@deploy1001> |
Started deploy [zuul/deploy@4a69913]: (no justification provided) |
[production] |
| 13:12 |
<mutante> |
restarted and re-armed keyholder on deploy1001 to pick up changes for zuul scap deploy |
[production] |
| 12:12 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 12:11 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 12:10 |
<mutante> |
Creating VM people1002.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=A vcpus=1 memory=2GB disk=80GB link=private. (T249907) |
[production] |
| 12:10 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 12:10 |
<mutante> |
Creating VM people1002.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=A vcpus=1 memory=2GB disk=80GB link=private. This may take a few minutes. |
[production] |
| 12:10 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 12:09 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 12:09 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:47 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'mathoid' for release 'canary' . |
[production] |
| 11:47 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'mathoid' for release 'production' . |
[production] |
| 11:44 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'mathoid' for release 'production' . |
[production] |
| 11:39 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 09:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more weight to db1089', diff saved to https://phabricator.wikimedia.org/P10953 and previous config saved to /var/cache/conftool/dbconfig/20200410-094359-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more weight to db1089', diff saved to https://phabricator.wikimedia.org/P10952 and previous config saved to /var/cache/conftool/dbconfig/20200410-093129-marostegui.json |
[production] |
| 09:11 |
<elukey> |
move druid_load jobs from an-coord1001 to an-launcher1001 |
[analytics] |
| 08:52 |
<hashar@deploy1001> |
Finished deploy [zuul/deploy@4a69913]: (no justification provided) (duration: 00m 16s) |
[production] |
| 08:51 |
<hashar@deploy1001> |
Started deploy [zuul/deploy@4a69913]: (no justification provided) |
[production] |
| 08:46 |
<hashar@deploy1001> |
Finished deploy [zuul/deploy@5a0a03a]: (no justification provided) (duration: 02m 20s) |
[production] |
| 08:44 |
<hashar@deploy1001> |
Started deploy [zuul/deploy@5a0a03a]: (no justification provided) |
[production] |
| 08:39 |
<mutante> |
deploy1001 - keyholder disarm, keyholder arm |
[production] |
| 08:32 |
<mutante> |
fix comment in deployment ssh key for zuul to include the path to the key on deploy1001 |
[production] |