|
2017-09-12
§
|
| 09:24 |
<moritzm> |
upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts) |
[production] |
| 08:38 |
<gehel> |
deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before) |
[production] |
| 08:34 |
<gehel> |
deploying elasticsearch plugins on relforge - T175159 |
[production] |
| 08:28 |
<moritzm> |
upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts) |
[production] |
| 07:35 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s) |
[production] |
| 07:15 |
<moritzm> |
upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts) |
[production] |
| 02:29 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s) |
[production] |
| 02:22 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s) |
[production] |
| 02:10 |
<bblack> |
cp1050 - backend restart, mailbox lag |
[production] |
| 02:08 |
<bblack> |
cp1099 - backend restart, mailbox lagh |
[production] |
| 00:48 |
<awight@tin> |
Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s) |
[production] |
| 00:48 |
<awight@tin> |
Started deploy [ores/deploy@42c5663]: Cause ORES service restart |
[production] |
|
2017-09-11
§
|
| 23:06 |
<mutante> |
jenkins-bot says: java.lang.OutofMemoryError: PermGen space |
[production] |
| 22:30 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4026.* |
[production] |
| 22:30 |
<dereckson@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case ([[Gerrit:377281]], no-op in prod) (duration: 00m 45s) |
[production] |
| 21:45 |
<awight@tin> |
Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s) |
[production] |
| 21:41 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4025.* |
[production] |
| 21:31 |
<awight@tin> |
Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster |
[production] |
| 21:03 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4023.* |
[production] |
| 20:13 |
<demon@tin> |
Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s) |
[production] |
| 19:11 |
<chasemp> |
adjust nodepool rate to 10 |
[production] |
| 19:08 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s) |
[production] |
| 19:01 |
<mutante> |
restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space |
[production] |
| 18:52 |
<mutante> |
restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service |
[production] |
| 18:40 |
<niharika29@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s) |
[production] |
| 18:33 |
<robh> |
firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475 |
[production] |
| 18:29 |
<chasemp> |
bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically) |
[production] |
| 18:29 |
<robh> |
firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475 |
[production] |
| 18:24 |
<robh> |
updating firmware on scs-a1-codfw and scs-c1-codfw |
[production] |
| 18:10 |
<robh> |
scs-ulsfo has successfully updated firmware T174475 |
[production] |
| 18:03 |
<robh> |
upgrading firmware on scs-ulsfo |
[production] |
| 17:09 |
<gehel@tin> |
Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s) |
[production] |
| 17:07 |
<gehel> |
wdqs weekly deployment (logging and throttling fixes) |
[production] |
| 17:06 |
<gehel@tin> |
Started deploy [wdqs/wdqs@177d20a]: (no justification provided) |
[production] |
| 15:20 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4022.* |
[production] |
| 15:20 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4028.* |
[production] |
| 14:40 |
<godog> |
roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997 |
[production] |
| 14:24 |
<hashar> |
tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed |
[production] |
| 14:23 |
<hashar> |
tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace |
[production] |
| 14:05 |
<hashar> |
European SWAT cancelled due to deploy infrastructure issue |
[production] |
| 13:55 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s) |
[production] |
| 13:54 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s) |
[production] |
| 13:50 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s) |
[production] |
| 13:43 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s) |
[production] |
| 13:40 |
<hashar@tin> |
Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s) |
[production] |
| 11:43 |
<moritzm> |
installing file/libmagic security updates (stretch-specific, does not affect older distros) |
[production] |
| 10:30 |
<jynus> |
restarting mysql @ labsdb1001 |
[production] |
| 09:40 |
<godog> |
roll-restart thumbor to apply changes for gif max animated area - T173580 |
[production] |
| 09:39 |
<mobrovac@tin> |
Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s) |
[production] |
| 09:39 |
<mobrovac@tin> |
Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 |
[production] |