|
2022-09-21
§
|
| 20:20 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:833463|cirrus: Limit shard count to 1 in deployment-prep (T316711)]] |
[production] |
| 20:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:17 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:833837|Enable DiscussionTools visual enhancements as beta on en/dewiki (T315625)]] (duration: 05m 31s) |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:12 |
<samtar@deploy1002> |
samtar and kemayo: Backport for [[gerrit:833837|Enable DiscussionTools visual enhancements as beta on en/dewiki (T315625)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 20:11 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:833837|Enable DiscussionTools visual enhancements as beta on en/dewiki (T315625)]] |
[production] |
| 20:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:09 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]] (duration: 05m 16s) |
[production] |
| 20:04 |
<samtar@deploy1002> |
samtar and zabe: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:04 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]] |
[production] |
| 19:33 |
<nokafor@deploy1002> |
Finished deploy [airflow-dags/analytics@ce20ecd]: (no justification provided) (duration: 00m 10s) |
[production] |
| 19:33 |
<nokafor@deploy1002> |
Started deploy [airflow-dags/analytics@ce20ecd]: (no justification provided) |
[production] |
| 19:25 |
<joal> |
Kill oozie daily cassandra loading jobs as we move them to airflow |
[analytics] |
| 19:18 |
<ottomata> |
kill aarora process 30421 run_embedding_training.sh on stat1005 |
[analytics] |
| 19:13 |
<joal> |
Deployed refinery for HQL patch (Njideka) |
[analytics] |
| 19:11 |
<ottomata> |
kill aarora process 14584 on stat1005 - using 2500% cpu |
[analytics] |
| 19:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:04 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: b8b2ebd3933cb891b62bb6aea01b2342c017cec8: Growth: Switch pilot wikis to structured mentor list (T310905) (duration: 03m 59s) |
[production] |
| 19:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:01 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 19:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 19:00 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 18:55 |
<nokafor@deploy1002> |
Finished deploy [analytics/refinery@91d0cf8] (thin): Regular analytics weekly train THIN [analytics/refinery@91d0cf8] (duration: 00m 08s) |
[production] |
| 18:55 |
<nokafor@deploy1002> |
Started deploy [analytics/refinery@91d0cf8] (thin): Regular analytics weekly train THIN [analytics/refinery@91d0cf8] |
[production] |
| 18:49 |
<TheresNoTime> |
cherry-picked [[gerrit:833839]] to deployment-puppetmaster04, testing T317417 |
[releng] |
| 18:44 |
<nokafor@deploy1002> |
Finished deploy [analytics/refinery@91d0cf8]: Regular analytics weekly train [analytics/refinery@91d0cf8] (duration: 05m 40s) |
[production] |
| 18:38 |
<nokafor@deploy1002> |
Started deploy [analytics/refinery@91d0cf8]: Regular analytics weekly train [analytics/refinery@91d0cf8] |
[production] |
| 18:19 |
<zabe> |
install mariadb 10.6 via role::mariadb::beta on deployment-db10 # T318126 |
[releng] |
| 17:55 |
<zabe> |
create volume db10 and attach to deployment-db10 # T318126 |
[releng] |
| 17:54 |
<zabe> |
create deployment-db10 as g3.cores8.ram16.disk20 # T318126 |
[releng] |
| 14:56 |
<Emperor> |
set thanos ring replicas to 3.75 T311690 |
[production] |
| 14:50 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/db-labs.php: Config: [[gerrit:833783|Pool deployment-db09, depool deployment-db08 (T318126)]] (Beta-only, exchange one replica for another) [*actually* sync it this time since I forgot to git rebase before the last sync 🤦] (duration: 03m 41s) |
[production] |
| 14:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:46 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 14:46 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 14:44 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/db-labs.php: Config: [[gerrit:833783|Pool deployment-db09, depool deployment-db08 (T318126)]] (Beta-only, exchange one replica for another) (duration: 03m 48s) |
[production] |
| 14:21 |
<zabe> |
deployment-db09: restart mariadb # T318126 |
[releng] |
| 14:00 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:59 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:59 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:59 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:58 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:57 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/db-labs.php: Config: [[gerrit:833776|Add back deployment-db08 (T318126)]] (Beta-only, restore old replica) (duration: 03m 48s) |
[production] |
| 13:55 |
<TheresNoTime> |
modified deployment-prep "prometheus" security group - port 80, T315699 |
[releng] |