|
2019-09-11
ยง
|
| 15:48 |
<bblack> |
lvs2006 - remove eventbus.svc.codfw.wmnet service, restart pybal, etc |
[production] |
| 15:03 |
<bblack> |
downtimed dns-discovery confd health checks for eventbus - T232122 |
[production] |
| 13:13 |
<hashar@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.22 (duration: 01m 02s) |
[production] |
| 13:12 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.22 |
[production] |
| 12:48 |
<moritzm> |
upgrade labpuppetmaster* to use facter 3 / puppet 5 |
[production] |
| 12:40 |
<moritzm> |
removing now obsolete puppet/puppetdb packages from labpuppetmaster* T171188 |
[production] |
| 12:40 |
<moritzm> |
removing now puppet/puppetdb packages from labpuppetmaster* T171188 |
[production] |
| 11:59 |
<hashar> |
Restarting Gerrit due to deadlock in the account cache # T224448 |
[production] |
| 11:57 |
<bblack> |
applying GRE MTU -> MSS fixup to cobalt and gerrit2001 - T218184 |
[production] |
| 11:41 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:40 |
<ladsgroup@deploy1001> |
Synchronized php-1.34.0-wmf.21/maintenance/getReplicaServer.php: SWAT: [[gerrit:535217|maintenance/getReplicaServer.php: Remove reference to long-deleted config var (T232268)]] (duration: 01m 04s) |
[production] |
| 11:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:535615|Disable AMC Outreach modal (T231436)]] (duration: 01m 04s) |
[production] |
| 11:15 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:535815|Set item terms on write both up to Q10mio (T225055)]] (duration: 01m 03s) |
[production] |
| 11:10 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:535567|TR: set WikibaseTaintedReferencesEnabled true on labs wikidatawiki (T232191)]] (duration: 01m 03s) |
[production] |
| 10:57 |
<mobrovac> |
drop the wiktionary definition keyspace - T231361 |
[production] |
| 10:23 |
<moritzm> |
removed roentgenium/tureis in Ganeti T224559 |
[production] |
| 10:18 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:17 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 10:01 |
<jynus> |
stopping and upgrading db1074 |
[production] |
| 09:56 |
<jynus> |
upgrading mariadb client libary on mariadb root clients |
[production] |
| 09:46 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 50% - T219150 (duration: 01m 03s) |
[production] |
| 09:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a (duration: 12m 15s) |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a |
[production] |
| 09:32 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 (duration: 13m 18s) |
[production] |
| 09:19 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 |
[production] |
| 09:16 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 (duration: 03m 59s) |
[production] |
| 09:13 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 |
[production] |
| 09:11 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 (duration: 03m 24s) |
[production] |
| 09:08 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 |
[production] |
| 08:36 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 (duration: 02m 13s) |
[production] |
| 08:34 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition (duration: 00m 34s) |
[production] |
| 08:24 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition |
[production] |
| 08:22 |
<mobrovac@deploy1001> |
Finished deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 (duration: 02m 45s) |
[production] |
| 08:19 |
<mobrovac@deploy1001> |
Started deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 |
[production] |
| 08:13 |
<elukey> |
add thirdparty/amd-rocm271 to buster-wikimedia and update it with ROCm 2.7.1 packages |
[production] |
| 08:09 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:07 |
<elukey> |
execute reprepro clearvanished on install1002 to clear buster-wikimedia|thirdparty/amd-rocm27 (not used anymore) |
[production] |
| 08:07 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P9088 and previous config saved to /var/cache/conftool/dbconfig/20190911-080450-marostegui.json |
[production] |
| 07:52 |
<moritzm> |
reimaging restbase-dev1005 to Stretch T224554 |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9087 and previous config saved to /var/cache/conftool/dbconfig/20190911-075139-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9086 and previous config saved to /var/cache/conftool/dbconfig/20190911-073335-marostegui.json |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9085 and previous config saved to /var/cache/conftool/dbconfig/20190911-072344-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9084 and previous config saved to /var/cache/conftool/dbconfig/20190911-071450-marostegui.json |
[production] |
| 07:07 |
<marostegui> |
Stop MySQL on db1122 to reboot for a kernel upgrade T230785 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json |
[production] |
| 07:00 |
<hashar> |
Restarting Gerrit - T224448 |
[production] |