|
2016-09-13
§
|
| 17:58 |
<mutante> |
wtp2019 - powercycled, back up without the error, services started |
[production] |
| 17:55 |
<mutante> |
wtp2019 Uncorrectable Memory Error |
[production] |
| 17:55 |
<mutante> |
wtp2019 - down sinc a couple days. console says "Alert! System fatal error during previous boot" |
[production] |
| 17:53 |
<volans> |
reimaging mw2198 that failed early today |
[production] |
| 17:36 |
<demon@tin> |
Started scap: testwiki to wmf.19 + l10n bootstrap (try 2) |
[production] |
| 17:24 |
<demon@tin> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4282891950" --threads=4 --lang en --quiet' returned non-zero exit status 1 (duration: 09m 23s) |
[production] |
| 17:15 |
<demon@tin> |
Started scap: testwiki to wmf.19 + l10n bootstrap |
[production] |
| 16:59 |
<jynus> |
beta dbs back in rw mode |
[production] |
| 16:54 |
<mobrovac> |
restbase deploy end of d10d759 |
[production] |
| 16:53 |
<demon@tin> |
Synchronized multiversion/updateWikiversions: unbreak myself (duration: 00m 48s) |
[production] |
| 16:41 |
<mobrovac> |
restbase deploy start of d10d759 |
[production] |
| 16:25 |
<mobrovac> |
change-prop deploying d701a69 |
[production] |
| 16:24 |
<godog> |
dump hhvm backtrace on mw1162 and restart hhvm, apache gets connection refused |
[production] |
| 16:17 |
<urandom> |
T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet) |
[production] |
| 16:06 |
<jynus> |
power resetting db1082 |
[production] |
| 16:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 01m 00s) |
[production] |
| 15:46 |
<gehel> |
test_* indices remvoed on relforge cluster, cluster restarted |
[production] |
| 15:44 |
<jynus> |
setting deployment-db1 and deployment-db1 mysqls in read only mode |
[production] |
| 15:40 |
<gehel> |
shutting down relforge cluster for indices cleanup |
[production] |
| 02:46 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Sep 13 02:46:54 UTC 2016 (duration 7m 12s) |
[production] |
| 02:39 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 41s) |
[production] |
| 01:58 |
<aaron@tin> |
Synchronized wmf-config/CommonSettings.php: Lower $wgMaxUserDBWriteDuration to 3 (duration: 00m 47s) |
[production] |
| 01:27 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.18/extensions/GlobalUsage: 1843a856c7c648a310a1983d53139e5b79ea6585 (duration: 00m 48s) |
[production] |
| 01:24 |
<bblack> |
cache_upload: reverting codfw to file storage |
[production] |
| 01:15 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 49s) |
[production] |
| 01:14 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: (no message) (duration: 00m 52s) |
[production] |
| 00:50 |
<bblack> |
cache_upload: reverting eqiad to file storage |
[production] |
| 00:50 |
<bblack> |
reverting eqiad to file storage |
[production] |
| 00:17 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 48s) |
[production] |
| 00:16 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: 0419831bbae66bda1e4dca8d690b607b493b2f5f (duration: 00m 47s) |
[production] |
|
2016-09-12
§
|
| 23:40 |
<bblack> |
switch cache_upload codfw to -sdeprecated_persistent... |
[production] |
| 23:40 |
<dereckson@tin> |
Synchronized wmf-config/throttle.php: Women in Science throttle rules (T145115 and T145253) (duration: 00m 47s) |
[production] |
| 23:33 |
<urandom> |
T144826: Restarting Cassandra on restbase2004-b.codfw.wmnet (scrub complete, re-joining cluster) |
[production] |
| 23:13 |
<bblack> |
switch cache_upload eqiad to -sdeprecated_persistent... |
[production] |
| 21:54 |
<eileen> |
from 3f01d93237c2b8af4fbda5e629a4a59c77dab3c1 to f381bd1c15e30e4d47fe372e128143baee6a7c7a |
[production] |
| 21:17 |
<arlolra> |
For completeness, "back" in my last log is a mistake. I scap deployed the wrong --rev, but that was ultimately the version we wanted deployed anyways, so no harm no foul. (T145460) |
[production] |
| 20:40 |
<arlolra> |
Parsoid back on 7c43009c |
[production] |
| 20:32 |
<arlolra> |
Parsoid deploy failed, rolling back |
[production] |
| 20:28 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-b.eqiad.wmnet |
[production] |
| 20:28 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-b.eqiad.wmnet |
[production] |
| 20:28 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-b.eqiad.wmnet |
[production] |
| 20:21 |
<mobrovac> |
change-prop deploying 86a60b3 |
[production] |
| 20:13 |
<arlolra> |
starting Parsoid deploy |
[production] |
| 19:52 |
<demon@tin> |
Synchronized multiversion/getMWVersion: for dumps <3 (duration: 00m 46s) |
[production] |
| 19:44 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-c.eqiad.wmnet |
[production] |
| 19:44 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-c.eqiad.wmnet |
[production] |
| 19:44 |
<urandom> |
!log T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-c.eqiad.wmnet |
[production] |
| 19:42 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-c.eqiad.wmnet |
[production] |
| 19:42 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-c.eqiad.wmnet |
[production] |
| 19:42 |
<urandom> |
T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-c.eqiad.wmnet |
[production] |