|
2019-11-15
§
|
| 11:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1113:3315 into vslow,dump after schema change', diff saved to https://phabricator.wikimedia.org/P9645 and previous config saved to /var/cache/conftool/dbconfig/20191115-112520-marostegui.json |
[production] |
| 11:19 |
<marostegui> |
Reboot dbproxy2002 |
[production] |
| 11:15 |
<marostegui> |
Reboot dbproxy2004 |
[production] |
| 11:12 |
<marostegui> |
Reboot dbproxy2001 |
[production] |
| 10:45 |
<marostegui> |
Run maintain-views for s5 on labsdb1011 T233135 |
[production] |
| 10:38 |
<moritzm> |
installing ghostscript security updates |
[production] |
| 10:37 |
<mobrovac> |
restbase - truncated parsoidphp data tables - T229015 |
[production] |
| 10:36 |
<ema> |
pool cp3062 with ATS backend T227432 |
[production] |
| 10:24 |
<godog> |
roll-restart logstash to apply configuration change |
[production] |
| 10:19 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:15 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:50 |
<ema> |
depool cp3062 and reimage as text_ats T227432 |
[production] |
| 09:47 |
<vgutierrez> |
Use a synthetic warning for 1% of TLSv1/TLS1v.1 pageviews - T238038 |
[production] |
| 09:18 |
<vgutierrez> |
Move cp1079 from nginx to ats-tls - T231627 |
[production] |
| 09:13 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) |
[production] |
| 09:02 |
<vgutierrez> |
Move cp1077 from nginx to ats-tls - T231627 |
[production] |
| 08:42 |
<vgutierrez> |
Move cp2006 from nginx to ats-tls - T231627 |
[production] |
| 08:30 |
<vgutierrez> |
Move cp2004 from nginx to ats-tls - T231627 |
[production] |
| 06:41 |
<marostegui> |
Stop MySQL on db2065 to clone db2134 (this will trigger an haproxy irc alert) - T238183 |
[production] |
| 06:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1113:3315 for schema change and temporary pool db1082 into vslow,dump', diff saved to https://phabricator.wikimedia.org/P9643 and previous config saved to /var/cache/conftool/dbconfig/20191115-060807-marostegui.json |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2088:3311 for compression', diff saved to https://phabricator.wikimedia.org/P9642 and previous config saved to /var/cache/conftool/dbconfig/20191115-060425-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3312 db1082 after schema changes', diff saved to https://phabricator.wikimedia.org/P9641 and previous config saved to /var/cache/conftool/dbconfig/20191115-060300-marostegui.json |
[production] |
| 05:57 |
<marostegui> |
Run maintain-views for s5 on labsdb1009, labsdb1010, labsdb1012 (pending labsdb1011 as it is still running the schema change) T233135 |
[production] |
| 05:07 |
<vgutierrez> |
Move cp3064 from nginx to ats-tls - T231627 |
[production] |
| 04:38 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@2ad7b1a]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 04:38 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@2ad7b1a]: Deploy design/style-guide: |
[production] |
| 04:17 |
<vgutierrez> |
Move cp3062 from nginx to ats-tls - T231627 |
[production] |
| 04:00 |
<vgutierrez> |
Move cp3060 from nginx to ats-tls - T231627 |
[production] |
| 01:35 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/Rest/Handler/CompareHandler.php: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 53s) |
[production] |
| 01:33 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/Rest/coreRoutes.json: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 52s) |
[production] |
| 01:32 |
<tstarling@deploy1001> |
Synchronized php-1.35.0-wmf.5/includes/parser/Parser.php: deploying REST compare section feature because iOS team need it for a beta release due very soon (duration: 00m 54s) |
[production] |
|
2019-11-14
§
|
| 23:03 |
<mutante> |
restarting gerrit to ncrease defaultThreadPoolSize to 2 |
[production] |
| 22:29 |
<eileen> |
civicrm revision changed from a3714003ff to ae9b3819cd, config revision is 6adc66a20b |
[production] |
| 21:32 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@150f9af]: Updating Parsoid to 74203415 (duration: 08m 21s) |
[production] |
| 21:24 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@150f9af]: Updating Parsoid to 74203415 |
[production] |
| 21:14 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 20:06 |
<cdanis@cumin2001> |
dbctl commit (dc=all): 'remove now-defunct wikitech section T233236', diff saved to https://phabricator.wikimedia.org/P9639 and previous config saved to /var/cache/conftool/dbconfig/20191114-200649-cdanis.json |
[production] |
| 20:04 |
<gehel> |
reloading data on wdqs1004 from wdqs1007 to catch up on lag faster - T238229 |
[production] |
| 19:57 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 19:33 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:31 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:20 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:49 |
<catrope@deploy1001> |
Synchronized wmf-config/: Use s10/s11 dblists for wikitechs (for real this time) (T233236) (duration: 00m 52s) |
[production] |
| 18:37 |
<catrope@deploy1001> |
Synchronized dblists/: Use s10/s11 dblists for wikitechs (T233236) (duration: 00m 51s) |
[production] |
| 18:35 |
<catrope@deploy1001> |
Synchronized dblists/: Add s10/s11 dblists for wikitechs (T233236) (duration: 00m 52s) |
[production] |
| 18:34 |
<mutante> |
scandium - restart php7.2-fpm |
[production] |
| 18:31 |
<mutante> |
phabricator (phab1003, prod server) - upgrade PHP version to 7.2.24 (T237239) |
[production] |
| 18:17 |
<cdanis@cumin2001> |
dbctl commit (dc=all): 'alias wikitech section to new s10 section T233236', diff saved to https://phabricator.wikimedia.org/P9638 and previous config saved to /var/cache/conftool/dbconfig/20191114-181732-cdanis.json |
[production] |
| 17:46 |
<robh> |
running dell epsa tool on cp3056 per T236497 |
[production] |