|
2020-06-10
§
|
| 11:02 |
<marostegui> |
Stop MySQL on db1094 to clone db1127 |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1094 moving to clone db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11453 and previous config saved to /var/cache/conftool/dbconfig/20200610-110204-marostegui.json |
[production] |
| 10:57 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:54 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1127 moving it to s7 T253217', diff saved to https://phabricator.wikimedia.org/P11452 and previous config saved to /var/cache/conftool/dbconfig/20200610-103742-marostegui.json |
[production] |
| 10:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11451 and previous config saved to /var/cache/conftool/dbconfig/20200610-102805-marostegui.json |
[production] |
| 10:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: IV – Drop flag to load (duration: 01m 05s) |
[production] |
| 10:23 |
<jayme> |
T254581 re-enabled puppet on all mw, api and jobrunner servers |
[production] |
| 10:20 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T254036 Undeploy CollaborationKit: III – Drop ability to load (duration: 01m 05s) |
[production] |
| 10:16 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: II – Disable on Test Wikipedia (duration: 01m 37s) |
[production] |
| 10:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11450 and previous config saved to /var/cache/conftool/dbconfig/20200610-101407-marostegui.json |
[production] |
| 10:12 |
<moritzm> |
upgrading remaining API servers in codfw to PHP 7.2.31 |
[production] |
| 10:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11449 and previous config saved to /var/cache/conftool/dbconfig/20200610-100834-marostegui.json |
[production] |
| 10:03 |
<jynus> |
cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516 |
[production] |
| 10:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1103 into x1', diff saved to https://phabricator.wikimedia.org/P11448 and previous config saved to /var/cache/conftool/dbconfig/20200610-100306-marostegui.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11447 and previous config saved to /var/cache/conftool/dbconfig/20200610-100037-marostegui.json |
[production] |
| 09:35 |
<volans> |
imported 0.0.38-1+deb10u1 into buster-wikimedia APT - T245114 |
[production] |
| 09:35 |
<marostegui> |
Stop mysql on db1127 to clone db1103 |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1137 for cloning db1103 - T253217', diff saved to https://phabricator.wikimedia.org/P11443 and previous config saved to /var/cache/conftool/dbconfig/20200610-093440-marostegui.json |
[production] |
| 09:31 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:31 |
<godog> |
configure thanos-be1* HDDs as raid0 - T252186 |
[production] |
| 09:26 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1103 to dbctl, depooled T253217', diff saved to https://phabricator.wikimedia.org/P11442 and previous config saved to /var/cache/conftool/dbconfig/20200610-092603-marostegui.json |
[production] |
| 09:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1103:3312 and db1103:3314', diff saved to https://phabricator.wikimedia.org/P11441 and previous config saved to /var/cache/conftool/dbconfig/20200610-092406-marostegui.json |
[production] |
| 09:14 |
<jayme> |
T254581 disabling puppet on all mw, api and jobrunner servers to move termbox envoy config to TLS |
[production] |
| 09:08 |
<kormat@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:08 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:50 |
<XioNoX> |
make asw1-eqsin interfaces Homer like - T250429 |
[production] |
| 08:45 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 08:45 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:17 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 08:14 |
<kormat@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:14 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:13 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 07:52 |
<kormat> |
reimaging db1077 T252027 |
[production] |
| 07:36 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 07:36 |
<XioNoX> |
make asw2-ulsfo interfaces Homer like - T250429 |
[production] |
| 07:33 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 07:31 |
<moritzm> |
upgrade mw1298-mw1309 (job runners) to PHP 7.2.31 |
[production] |
| 07:26 |
<XioNoX> |
trunk public vlan to esams ganeti hosts - T254157 |
[production] |
| 07:16 |
<XioNoX> |
trunk public vlan to eqsin ganeti hosts - T254157 |
[production] |
| 07:15 |
<moritzm> |
upgrade remaining API servers in eqiad to PHP 7.2.31 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103 for reimage - T253217', diff saved to https://phabricator.wikimedia.org/P11439 and previous config saved to /var/cache/conftool/dbconfig/20200610-070822-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2113 after on-site maintenance T251570', diff saved to https://phabricator.wikimedia.org/P11438 and previous config saved to /var/cache/conftool/dbconfig/20200610-070508-marostegui.json |
[production] |
| 06:53 |
<XioNoX> |
trunk public vlan to ulsfo ganeti hosts - T254157 |
[production] |
| 05:10 |
<marostegui> |
Deploy schema change on s3 master with 2 minutes sleep between wikis - T206103 |
[production] |