|
2019-09-11
ยง
|
| 11:59 |
<hashar> |
Restarting Gerrit due to deadlock in the account cache # T224448 |
[production] |
| 11:57 |
<bblack> |
applying GRE MTU -> MSS fixup to cobalt and gerrit2001 - T218184 |
[production] |
| 11:41 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:40 |
<ladsgroup@deploy1001> |
Synchronized php-1.34.0-wmf.21/maintenance/getReplicaServer.php: SWAT: [[gerrit:535217|maintenance/getReplicaServer.php: Remove reference to long-deleted config var (T232268)]] (duration: 01m 04s) |
[production] |
| 11:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:535615|Disable AMC Outreach modal (T231436)]] (duration: 01m 04s) |
[production] |
| 11:15 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:535815|Set item terms on write both up to Q10mio (T225055)]] (duration: 01m 03s) |
[production] |
| 11:10 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:535567|TR: set WikibaseTaintedReferencesEnabled true on labs wikidatawiki (T232191)]] (duration: 01m 03s) |
[production] |
| 10:57 |
<mobrovac> |
drop the wiktionary definition keyspace - T231361 |
[production] |
| 10:23 |
<moritzm> |
removed roentgenium/tureis in Ganeti T224559 |
[production] |
| 10:18 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:17 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:01 |
<jynus> |
stopping and upgrading db1074 |
[production] |
| 09:56 |
<jynus> |
upgrading mariadb client libary on mariadb root clients |
[production] |
| 09:46 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 50% - T219150 (duration: 01m 03s) |
[production] |
| 09:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a (duration: 12m 15s) |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 (duration: 13m 18s) |
[production] |
| 09:19 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 |
[production] |
| 09:16 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 (duration: 03m 59s) |
[production] |
| 09:13 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 |
[production] |
| 09:11 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 (duration: 03m 24s) |
[production] |
| 09:08 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 |
[production] |
| 08:36 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 (duration: 02m 13s) |
[production] |
| 08:34 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition (duration: 00m 34s) |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition |
[production] |
| 08:22 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 (duration: 02m 45s) |
[production] |
| 08:19 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 |
[production] |
| 08:13 |
<elukey> |
add thirdparty/amd-rocm271 to buster-wikimedia and update it with ROCm 2.7.1 packages |
[production] |
| 08:09 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:07 |
<elukey> |
execute reprepro clearvanished on install1002 to clear buster-wikimedia|thirdparty/amd-rocm27 (not used anymore) |
[production] |
| 08:07 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P9088 and previous config saved to /var/cache/conftool/dbconfig/20190911-080450-marostegui.json |
[production] |
| 07:52 |
<moritzm> |
reimaging restbase-dev1005 to Stretch T224554 |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9087 and previous config saved to /var/cache/conftool/dbconfig/20190911-075139-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9086 and previous config saved to /var/cache/conftool/dbconfig/20190911-073335-marostegui.json |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9085 and previous config saved to /var/cache/conftool/dbconfig/20190911-072344-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9084 and previous config saved to /var/cache/conftool/dbconfig/20190911-071450-marostegui.json |
[production] |
| 07:07 |
<marostegui> |
Stop MySQL on db1122 to reboot for a kernel upgrade T230785 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json |
[production] |
| 07:00 |
<hashar> |
Restarting Gerrit - T224448 |
[production] |
| 06:58 |
<hashar> |
Restarting Gerrit |
[production] |
| 06:45 |
<marostegui> |
Drop unused database puppet on m1 - T231539 |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9082 and previous config saved to /var/cache/conftool/dbconfig/20190911-061924-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9081 and previous config saved to /var/cache/conftool/dbconfig/20190911-061659-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2048, will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P9080 and previous config saved to /var/cache/conftool/dbconfig/20190911-054855-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2112 to s1 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P9079 and previous config saved to /var/cache/conftool/dbconfig/20190911-054753-marostegui.json |
[production] |
| 05:29 |
<marostegui> |
Switchover s1 codfw master db2048 -> db2112 T230106 |
[production] |