|
2019-11-19
ยง
|
| 20:17 |
<gehel> |
completed reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826 |
[production] |
| 20:14 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 20:10 |
<XioNoX> |
homer push on mgmt routers |
[production] |
| 20:09 |
<mutante> |
phab1003 after merging gerrit:551910 puppet now also stopped the actual aphlict service and removed the systemd unit file. had to manually run 'systemctl reset-failed' though to clean systemd status and avoid icinga alert (T238593) |
[production] |
| 20:07 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 19:20 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:18 |
<dzahn@cumin1001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 19:18 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 19:08 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) (duration: 06m 49s) |
[production] |
| 19:01 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@6e6bd42]: Prevent expensive content transforms from blocking the event loop (T229286) |
[production] |
| 19:00 |
<elukey> |
regenerate TLS cert for yarn.wikimedia.org (containing SANs for all analytics UIs) to add datasets.w.o SAN (site was failing due to ATS not being able to contact thorium) |
[production] |
| 18:59 |
<rlazarus> |
restarted php7.2-fpm on wtp2001, wtp2002 |
[production] |
| 18:56 |
<rlazarus> |
restarted php7.2-fpm on wtp1025, wtp1026 |
[production] |
| 18:35 |
<catrope@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/VisualEditor/: Unbreak instrumentation of init events (duration: 00m 53s) |
[production] |
| 18:34 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 (duration: 02m 04s) |
[production] |
| 18:32 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@6e7cffd]: Updating Parsoid to 1a1105a7 |
[production] |
| 18:30 |
<mutante> |
icinga config - manually added team-dcops, started icinga |
[production] |
| 18:20 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag, hook) [[gerrit:551858]] (duration: 00m 53s) |
[production] |
| 18:12 |
<RoanKattouw> |
That was eowiktionary, not eowikisource |
[production] |
| 18:09 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure default search namespaces for eowikisource (T237792) (duration: 00m 52s) |
[production] |
| 17:43 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag, maint script) [[gerrit:551857]] (duration: 00m 52s) |
[production] |
| 17:39 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 17:11 |
<addshore@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Wikidata.org: T221774 - Wikidata.org extension (queryservice maxlag) [[gerrit:551855]] [[gerrit:551856]] (duration: 00m 54s) |
[production] |
| 17:02 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@d73818a]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 17:02 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@d73818a]: Deploy design/style-guide: |
[production] |
| 16:58 |
<ema> |
pool cp2007 with ATS backend T227432 |
[production] |
| 16:30 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:28 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<moritzm> |
installing glib2.0 security updates |
[production] |
| 16:21 |
<mutante> |
phab1003 - puppet restarts aphlict service even with "phabricator_aphlict_enabled: false" in Hiera. But it does properly remove the proxy config lines from apache. so service is running but not used. (T238593) |
[production] |
| 16:17 |
<mutante> |
phab1003 - systemctl stop aphlict (proxy config in apache is disabled as well as disabled in ATS) (T238593) |
[production] |
| 16:15 |
<gehel> |
reloading data from wdqs1007 to wdqs1004 - after failed test of merging updater - T212826 |
[production] |
| 16:14 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 16:10 |
<ema> |
depool cp2007 and reimage as text_ats T227432 |
[production] |
| 16:09 |
<ema> |
pool cp2006 with ATS backend T227432 |
[production] |
| 15:59 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@564b2c6]: New Parsoid/PHP config structure (duration: 02m 11s) |
[production] |
| 15:57 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@564b2c6]: New Parsoid/PHP config structure |
[production] |
| 15:37 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:34 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:27 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@5e7f759]: Switch test.wp and test2.wp to Parsoid/PHP - T229015 (duration: 14m 22s) |
[production] |
| 15:15 |
<ema> |
depool cp2006 and reimage as text_ats T227432 |
[production] |
| 15:13 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@5e7f759]: Switch test.wp and test2.wp to Parsoid/PHP - T229015 |
[production] |
| 15:09 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@5e7f759] (dev-cluster): Switch test.wp and test2.wp to Parsoid/PHP (duration: 02m 58s) |
[production] |
| 15:07 |
<ema> |
pool cp2004 with ATS backend T227432 |
[production] |
| 15:06 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@5e7f759] (dev-cluster): Switch test.wp and test2.wp to Parsoid/PHP |
[production] |
| 14:38 |
<ema@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:36 |
<ema@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:34 |
<gehel> |
restarting blazegraph with additional logging on wdqs1004 - T231411 |
[production] |
| 14:18 |
<ema> |
depool cp2004 and reimage as text_ats T227432 |
[production] |
| 14:13 |
<ema> |
pool cp2001 with ATS backend T227432 |
[production] |