|
2019-11-15
§
|
| 06:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1113:3315 for schema change and temporary pool db1082 into vslow,dump', diff saved to https://phabricator.wikimedia.org/P9643 and previous config saved to /var/cache/conftool/dbconfig/20191115-060807-marostegui.json |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2088:3311 for compression', diff saved to https://phabricator.wikimedia.org/P9642 and previous config saved to /var/cache/conftool/dbconfig/20191115-060425-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3312 db1082 after schema changes', diff saved to https://phabricator.wikimedia.org/P9641 and previous config saved to /var/cache/conftool/dbconfig/20191115-060300-marostegui.json |
[production] |
| 05:57 |
<marostegui> |
Run maintain-views for s5 on labsdb1009, labsdb1010, labsdb1012 (pending labsdb1011 as it is still running the schema change) T233135 |
[production] |
| 05:07 |
<vgutierrez> |
Move cp3064 from nginx to ats-tls - T231627 |
[production] |
| 04:38 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@2ad7b1a]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 04:38 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@2ad7b1a]: Deploy design/style-guide: |
[production] |
| 04:17 |
<vgutierrez> |
Move cp3062 from nginx to ats-tls - T231627 |
[production] |
| 04:00 |
<vgutierrez> |
Move cp3060 from nginx to ats-tls - T231627 |
[production] |
| 01:35 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/Rest/Handler/CompareHandler.php: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 53s) |
[production] |
| 01:33 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/Rest/coreRoutes.json: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 52s) |
[production] |
| 01:32 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/parser/Parser.php: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 54s) |
[production] |
|
2019-11-14
§
|
| 23:03 |
<mutante> |
restarting gerrit to ncrease defaultThreadPoolSize to 2 |
[production] |
| 22:29 |
<eileen> |
civicrm revision changed from a3714003ff to ae9b3819cd, config revision is 6adc66a20b |
[production] |
| 21:32 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@150f9af]: Updating Parsoid to 74203415 (duration: 08m 21s) |
[production] |
| 21:24 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@150f9af]: Updating Parsoid to 74203415 |
[production] |
| 21:14 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 20:06 |
<cdanis@cumin2001> |
dbctl commit (dc=all): 'remove now-defunct wikitech section T233236', diff saved to https://phabricator.wikimedia.org/P9639 and previous config saved to /var/cache/conftool/dbconfig/20191114-200649-cdanis.json |
[production] |
| 20:04 |
<gehel> |
reloading data on wdqs1004 from wdqs1007 to catch up on lag faster - T238229 |
[production] |
| 19:57 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 19:33 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:31 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:20 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:49 |
<catrope@deploy1001> |
Synchronized wmf-config/: Use s10/s11 dblists for wikitechs (for real this time) (T233236) (duration: 00m 52s) |
[production] |
| 18:37 |
<catrope@deploy1001> |
Synchronized dblists/: Use s10/s11 dblists for wikitechs (T233236) (duration: 00m 51s) |
[production] |
| 18:35 |
<catrope@deploy1001> |
Synchronized dblists/: Add s10/s11 dblists for wikitechs (T233236) (duration: 00m 52s) |
[production] |
| 18:34 |
<mutante> |
scandium - restart php7.2-fpm |
[production] |
| 18:31 |
<mutante> |
phabricator (phab1003, prod server) - upgrade PHP version to 7.2.24 (T237239) |
[production] |
| 18:17 |
<cdanis@cumin2001> |
dbctl commit (dc=all): 'alias wikitech section to new s10 section T233236', diff saved to https://phabricator.wikimedia.org/P9638 and previous config saved to /var/cache/conftool/dbconfig/20191114-181732-cdanis.json |
[production] |
| 17:46 |
<robh> |
running dell epsa tool on cp3056 per T236497 |
[production] |
| 17:35 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 17:35 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 17:35 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 17:22 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 17:22 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 17:22 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 17:22 |
<ejegg> |
updated payments-wiki from bd907656fb to 30579d34d8 |
[production] |
| 17:17 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 17:17 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 17:17 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 16:09 |
<mutante> |
phab2001 - upgrading PHP version to 7.2.24 (T237239) |
[production] |
| 16:06 |
<mutante> |
scandium - upgrading PHP version to 7.2.24 (fyi, @subbu T228069) (T237239) |
[production] |
| 16:04 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikibase: [[gerrit:550828|Put a layer of APC cache on top of reading wb_terms in SqlEntityInfoBuilder]] (T231011 T229407 T236681), Try II (duration: 00m 56s) |
[production] |
| 14:54 |
<ema> |
pool cp3060 with ATS backend T227432 |
[production] |
| 14:53 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Fix bug when when looking up entity for an unknown ID (duration: 00m 53s) |
[production] |
| 14:48 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set MCR migration stage to NEW on group1 for T198312 (duration: 00m 53s) |
[production] |
| 14:27 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:24 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:01 |
<ema> |
depool cp3060 and reimage as text_ats T227432 |
[production] |
| 13:37 |
<ladsgroup@deploy1001> |
scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |