|
2020-06-10
§
|
| 10:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11449 and previous config saved to /var/cache/conftool/dbconfig/20200610-100834-marostegui.json |
[production] |
| 10:03 |
<jynus> |
cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516 |
[production] |
| 10:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1103 into x1', diff saved to https://phabricator.wikimedia.org/P11448 and previous config saved to /var/cache/conftool/dbconfig/20200610-100306-marostegui.json |
[production] |
| 10:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11447 and previous config saved to /var/cache/conftool/dbconfig/20200610-100037-marostegui.json |
[production] |
| 09:35 |
<volans> |
imported 0.0.38-1+deb10u1 into buster-wikimedia APT - T245114 |
[production] |
| 09:35 |
<marostegui> |
Stop mysql on db1127 to clone db1103 |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1137 for cloning db1103 - T253217', diff saved to https://phabricator.wikimedia.org/P11443 and previous config saved to /var/cache/conftool/dbconfig/20200610-093440-marostegui.json |
[production] |
| 09:31 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 09:31 |
<godog> |
configure thanos-be1* HDDs as raid0 - T252186 |
[production] |
| 09:26 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1103 to dbctl, depooled T253217', diff saved to https://phabricator.wikimedia.org/P11442 and previous config saved to /var/cache/conftool/dbconfig/20200610-092603-marostegui.json |
[production] |
| 09:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1103:3312 and db1103:3314', diff saved to https://phabricator.wikimedia.org/P11441 and previous config saved to /var/cache/conftool/dbconfig/20200610-092406-marostegui.json |
[production] |
| 09:14 |
<jayme> |
T254581 disabling puppet on all mw, api and jobrunner servers to move termbox envoy config to TLS |
[production] |
| 09:08 |
<kormat@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 09:08 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:50 |
<XioNoX> |
make asw1-eqsin interfaces Homer like - T250429 |
[production] |
| 08:45 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 08:45 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 08:45 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:17 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 08:14 |
<kormat@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 08:14 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:13 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 07:52 |
<kormat> |
reimaging db1077 T252027 |
[production] |
| 07:36 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 07:36 |
<XioNoX> |
make asw2-ulsfo interfaces Homer like - T250429 |
[production] |
| 07:33 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 07:31 |
<moritzm> |
upgrade mw1298-mw1309 (job runners) to PHP 7.2.31 |
[production] |
| 07:26 |
<XioNoX> |
trunk public vlan to esams ganeti hosts - T254157 |
[production] |
| 07:16 |
<XioNoX> |
trunk public vlan to eqsin ganeti hosts - T254157 |
[production] |
| 07:15 |
<moritzm> |
upgrade remaining API servers in eqiad to PHP 7.2.31 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103 for reimage - T253217', diff saved to https://phabricator.wikimedia.org/P11439 and previous config saved to /var/cache/conftool/dbconfig/20200610-070822-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2113 after on-site maintenance T251570', diff saved to https://phabricator.wikimedia.org/P11438 and previous config saved to /var/cache/conftool/dbconfig/20200610-070508-marostegui.json |
[production] |
| 06:53 |
<XioNoX> |
trunk public vlan to ulsfo ganeti hosts - T254157 |
[production] |
| 05:10 |
<marostegui> |
Deploy schema change on s3 master with 2 minutes sleep between wikis - T206103 |
[production] |
|
2020-06-09
§
|
| 23:18 |
<Reedy> |
run namespaceDupes.php --fix for hiwikibooks T254012 |
[production] |
| 23:10 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T254706 T254012 T241893 (duration: 01m 06s) |
[production] |
| 23:03 |
<Reedy> |
created wikilove_log on slwiki T254706 |
[production] |
| 20:00 |
<jhuneidi@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.32 (duration: 05m 11s) |
[production] |
| 19:51 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.36 |
[production] |
| 19:42 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.36 (duration: 57m 47s) |
[production] |
| 19:29 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:26 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:26 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:23 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:10 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:07 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:07 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:05 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |