|
2019-09-11
§
|
| 10:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:01 |
<jynus> |
stopping and upgrading db1074 |
[production] |
| 09:56 |
<jynus> |
upgrading mariadb client libary on mariadb root clients |
[production] |
| 09:46 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 50% - T219150 (duration: 01m 03s) |
[production] |
| 09:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a (duration: 12m 15s) |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 (duration: 13m 18s) |
[production] |
| 09:19 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 |
[production] |
| 09:16 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 (duration: 03m 59s) |
[production] |
| 09:13 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 |
[production] |
| 09:11 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 (duration: 03m 24s) |
[production] |
| 09:08 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 |
[production] |
| 08:36 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 (duration: 02m 13s) |
[production] |
| 08:34 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition (duration: 00m 34s) |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition |
[production] |
| 08:22 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 (duration: 02m 45s) |
[production] |
| 08:19 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 |
[production] |
| 08:13 |
<elukey> |
add thirdparty/amd-rocm271 to buster-wikimedia and update it with ROCm 2.7.1 packages |
[production] |
| 08:09 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:07 |
<elukey> |
execute reprepro clearvanished on install1002 to clear buster-wikimedia|thirdparty/amd-rocm27 (not used anymore) |
[production] |
| 08:07 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P9088 and previous config saved to /var/cache/conftool/dbconfig/20190911-080450-marostegui.json |
[production] |
| 07:52 |
<moritzm> |
reimaging restbase-dev1005 to Stretch T224554 |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9087 and previous config saved to /var/cache/conftool/dbconfig/20190911-075139-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9086 and previous config saved to /var/cache/conftool/dbconfig/20190911-073335-marostegui.json |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9085 and previous config saved to /var/cache/conftool/dbconfig/20190911-072344-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9084 and previous config saved to /var/cache/conftool/dbconfig/20190911-071450-marostegui.json |
[production] |
| 07:07 |
<marostegui> |
Stop MySQL on db1122 to reboot for a kernel upgrade T230785 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json |
[production] |
| 07:00 |
<hashar> |
Restarting Gerrit - T224448 |
[production] |
| 06:58 |
<hashar> |
Restarting Gerrit |
[production] |
| 06:45 |
<marostegui> |
Drop unused database puppet on m1 - T231539 |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9082 and previous config saved to /var/cache/conftool/dbconfig/20190911-061924-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9081 and previous config saved to /var/cache/conftool/dbconfig/20190911-061659-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2048, will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P9080 and previous config saved to /var/cache/conftool/dbconfig/20190911-054855-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2112 to s1 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P9079 and previous config saved to /var/cache/conftool/dbconfig/20190911-054753-marostegui.json |
[production] |
| 05:29 |
<marostegui> |
Switchover s1 codfw master db2048 -> db2112 T230106 |
[production] |
| 03:31 |
<eileen> |
civicrm revision changed from b343642c76 to 53aeba6318, config revision is 3e22a80bc8 |
[production] |
|
2019-09-10
§
|
| 20:46 |
<ejegg> |
updated payments-wiki from 15baf7f58b to 5432f9c3a4 |
[production] |
| 20:24 |
<XioNoX> |
add MSS clamp on install1002 - T2324563 |
[production] |
| 20:20 |
<XioNoX> |
add MSS clamp on archiva1001 - T232456 |
[production] |
| 18:42 |
<herron> |
rolling out "Aggregate IPsec Tunnel Status” icinga check, please disregard for the time being if it alerts |
[production] |
| 18:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T229863 Remove EventBusRCFeedEngine eventServiceName (duration: 01m 05s) |
[production] |
| 18:15 |
<XioNoX> |
rollback test add static route on bast3002 to force advmss |
[production] |
| 18:10 |
<XioNoX> |
test add static route on bast3002 to force advmss |
[production] |
| 17:58 |
<jforrester@deploy1001> |
Synchronized wmf-config/logging.php: T232042 Direct Parsoid/PHP rt-testing log events to a different target (duration: 01m 02s) |
[production] |
| 17:56 |
<jforrester@deploy1001> |
Synchronized wmf-config/ProductionServices.php: T232122 Stop setting production value for eventlogging-service (duration: 01m 00s) |
[production] |
| 17:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T232122 Remove use of eventlogging-service (duration: 01m 03s) |
[production] |
| 17:33 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Re-sync for safety after scap errored with a broken pipe (duration: 01m 03s) |
[production] |