|
2020-11-11
§
|
| 08:20 |
<bstorm> |
loading dump on the replica T266587 |
[clouddb-services] |
| 06:06 |
<bstorm> |
dump completed, transferring to replica to start things up again T266587 |
[clouddb-services] |
| 02:18 |
<ryankemper> |
(WDQS deploy completed) |
[production] |
| 00:48 |
<ryankemper> |
Restarting `wdqs-categories` one host at a time across all wdqs production instances: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'` |
[production] |
| 00:47 |
<ryankemper> |
Restarted `wdqs-categories` across wdqs test hosts: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 00:47 |
<ryankemper> |
Restarted `wdqs-updater` simultaneously across all wdqs hosts: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 00:47 |
<ryankemper> |
[wdqs deploy] following deploy, example query succeeds on `query.wikidata.org`, proceeding to post deploy steps |
[production] |
| 00:46 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@03219df]: 0.3.55 (duration: 11m 24s) |
[production] |
| 00:46 |
<ryankemper> |
T222669 [Elasticsearch reindex] Began long-running reindex of cirrus elasticsearch for `codfw`, `eqiad`, and `cloudelastic`. 3 tmux sessions on `ryankemper@mwmaint1002`: `reindex_eqiad`, `reindex_codfw`, `reindex_cloudelastic` |
[production] |
| 00:38 |
<ryankemper> |
Following deploy to canary `wdqs1003`, automated tests are passing as is a manual test of an example query. Proceeding... |
[production] |
| 00:34 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@03219df]: 0.3.55 |
[production] |
| 00:32 |
<ryankemper> |
About to begin wdqs deploy; before-deploy tests on canary `wdqs1003` are passing |
[production] |
| 00:09 |
<eileen> |
civicrm revision changed from d0cd7f6dbb to e5d12cc46c, config revision is e2d133eff4 |
[production] |
|
2020-11-10
§
|
| 23:19 |
<dpifke> |
Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/640226 and https://gerrit.wikimedia.org/r/c/performance/coal/+/640227 in beta; should only affect deployment-webperf11. |
[releng] |
| 22:14 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:14 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:13 |
<thcipriani> |
restarting php7.2-fpm on deployment-jobrunner03 and deployment-parsoid11 |
[releng] |
| 22:08 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:08 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:05 |
<mholloway-shell@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 21:59 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 21:58 |
<jgleeson> |
update civicrm revision changed from c36a5cc1b1 to d0cd7f6dbb |
[production] |
| 21:57 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 21:55 |
<mholloway-shell@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 21:47 |
<ebernhardson> |
unban elastic1050 from eqiad search psi cluster |
[production] |
| 21:28 |
<cstone> |
civicrm revision changed from b1342c4129 to c36a5cc1b1 |
[production] |
| 21:24 |
<brennen@deploy1001> |
sync-file aborted: Testing: README.md sync-file with ssh -n for T223287 (duration: 00m 37s) |
[production] |
| 21:23 |
<brennen> |
testing some scap operations, modified to use ssh -n for debugging T223287 |
[production] |
| 21:11 |
<ebernhardson> |
ban elastic1050 from eqiad psi cluster due to excessive load |
[production] |
| 21:02 |
<brennen@deploy1001> |
Finished scap: Backport: [[gerrit:640487|language: Honor $wgTranslateNumerals, even if PHP does digit translation(T267614)]] and [[gerrit:640488|Downgrade the severity of the non-numeric argument to formatNum warnings (T267370, T267587)]] (duration: 34m 46s) |
[production] |
| 20:27 |
<brennen@deploy1001> |
Started scap: Backport: [[gerrit:640487|language: Honor $wgTranslateNumerals, even if PHP does digit translation(T267614)]] and [[gerrit:640488|Downgrade the severity of the non-numeric argument to formatNum warnings (T267370, T267587)]] |
[production] |
| 20:16 |
<chicocvenancio> |
restart hub to apply move to sqlite. T267667 |
[paws] |
| 20:10 |
<brennen@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:640254|Turn on formatnum logging (T267587, T267370)]] (duration: 01m 02s) |
[production] |
| 19:45 |
<andrewbogott> |
rebooting tools-sgeexec-0950; OOM |
[tools] |
| 19:42 |
<bstorm> |
safelisted "argocd" namespace with namespaceSelector for registry-admission controller |
[toolsbeta] |
| 19:32 |
<joal> |
Deploy wikistats2 v2.8.2 |
[analytics] |
| 19:06 |
<hknust> |
holger mwmaint1002 Stop T219279 |
[production] |
| 18:49 |
<legoktm> |
associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it |
[toolsbeta] |
| 18:40 |
<apergos> |
deployment-prep: switched deployment-prep to libicu63, see T264991 |
[releng] |
| 18:31 |
<hknust> |
holger mwmaint1002 Start T219279 |
[production] |
| 18:27 |
<legoktm> |
creating toolsbeta-docker-imagebuilder-01 (T267616) |
[toolsbeta] |
| 18:16 |
<joal> |
Releasing refinery-source v0,0,139 to archiva |
[analytics] |
| 17:57 |
<effie> |
pool mw1263 mw1264 |
[production] |
| 17:31 |
<effie> |
briefly depool mw1263 and mw1264 |
[production] |
| 17:30 |
<jynus> |
about to shutdown db1139 for hw maintenance T261405 |
[production] |
| 17:18 |
<dcaro> |
launching instance toolsbeta-test-k8s-etcd-4 (T267140) |
[toolsbeta] |
| 17:15 |
<dcaro> |
removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) (T267140) |
[toolsbeta] |
| 17:13 |
<dwisehaupt> |
upping thank you mail flow through frmx's to 30% of the total runs |
[production] |
| 16:41 |
<arturo> |
set paws in sqlite mode because T266587 (kubectl --namespace prod edit configmap hub-config) |
[paws] |
| 16:37 |
<arturo> |
icinga downtime toolschecker for 2h becasue toolsdb maintenance (T266587) |
[admin] |