|
2017-06-14
§
|
| 13:41 |
<zfilipin@tin> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:358958|Remove unneeded ContentTranslationTargetNamespace (T167865)]] (duration: 00m 44s) |
[production] |
| 13:35 |
<zfilipin@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 12:24 |
<hashar> |
gerrit: marked mediawiki/skins/Donate has read-only ( https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/skins/Donate ) - T124519 |
[releng] |
| 12:24 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Switchover pc2004 to db2072 (duration: 00m 43s) |
[production] |
| 12:13 |
<akosiaris> |
upload apertium-spa-ita_0.2.0~r78826-1+wmf to apt.wikimedia.org/jessie-wikimedia/main |
[production] |
| 12:13 |
<akosiaris> |
upload apertium-fra-cat_1.2.0~r78602-1+wmf to apt.wikimedia.org/jessie-wikimedia/main |
[production] |
| 11:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Switchover pc1004 to db1096 (duration: 00m 54s) |
[production] |
| 11:34 |
<jynus> |
about to deploy performance-impacting change on the parsercache persistent storage T167567 |
[production] |
| 11:19 |
<marostegui> |
Deploy alter table s4 - labsdb1011 - T166206 |
[production] |
| 10:04 |
<Sebastian-WMSE> |
Deploy latest from Git master: e641fcb (T165540), dfee00f (T148623) |
[wikispeech] |
| 09:46 |
<marostegui> |
Rename table titlekey before dropping it on enwiki - db1089 - T164949 |
[production] |
| 09:18 |
<godog> |
delete files older than 365d from 'servers' graphite hierarchy |
[production] |
| 07:59 |
<marostegui> |
Drop table updates on s3 - T139342 |
[production] |
| 07:32 |
<moritzm> |
installing zziplib security updates on jessie |
[production] |
| 07:04 |
<elukey> |
restart pdfrender on scb200[2,4] (xpra race condition) |
[production] |
| 07:03 |
<elukey> |
restart pdfrender on scb1004 (xpra race condition) |
[production] |
| 06:32 |
<moritzm> |
installing remaining libtasn security updates |
[production] |
| 03:14 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Jun 14 03:14:28 UTC 2017 (duration 6m 56s) |
[production] |
| 03:07 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.5) (duration: 14m 52s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 07m 58s) |
[production] |
| 01:48 |
<mutante> |
netmon1002 - chown rancid:rancid /var/lib/rancid ; touch /var/lib/rancid/.gitconfig, let rancid write to config, then git config --global user.email and user.name as the rancid user | fix permissions on .git/objects files, let rancid user own them all | re-commit .gitingore change | SSH_AUTH_SOCK=/run/keyholder/proxy.sock /usr/lib/rancid/bin/rancid-run as user "rancid" runs clean, |
[production] |
| 01:20 |
<mutante> |
netmon1002 - copied missing router.db, routers.all/.down/.up over from netmon1001 to /var/lib/rancid/core. routers.db is an untracked file, the others are in .gitignore. this is all like on netmon1001 as well. adding routers.db to .gitignore file on both, like the other router* files already were (T159756) |
[production] |
| 01:00 |
<mutante> |
netmon1002 - locally "git clone /var/lib/rancid/GIT/core" into /var/lib/rancid (i rsynced that but it's a bare repository without a work tree. work tree is /var/lib/rancid/core (after this) (T159756) |
[production] |
| 00:44 |
<mutante> |
naos: disarm keyholder and armed it again to proof i didn't break anything on jessie by fixing keyholder on stretch with gerrit:358884 |
[production] |
| 00:39 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: extdist update (duration: 00m 44s) |
[production] |
| 00:09 |
<aaron@tin> |
Synchronized wmf-config/InitialiseSettings.php: Capture messages on 'autoloader' debug log channel (duration: 00m 44s) |
[production] |
|
2017-06-13
§
|
| 23:29 |
<RainbowSprinkles> |
gerrit: upgrading on master 2.13.4-13-gc0c5cc4742 -> 2.13.8-1-g7c438d37a2 (been running on slave for a week) |
[production] |
| 23:13 |
<mutante> |
contint1001 - started zuul using the old init script |
[production] |
| 23:04 |
<mutante> |
netmon1001/1002: rsynced /var/lib/rancid/CVS and /var/lib/rancid/GIT from 1001 to 1002 for rancid migration (T159756) |
[production] |
| 23:04 |
<demon@tin> |
Synchronized php-1.30.0-wmf.4/extensions/OpenStackManager: Re-adding deleted special page (duration: 00m 45s) |
[production] |
| 22:06 |
<ejegg> |
updated fundraising tools from f2522cdabf1741a60b7b60ac8f7ead7afd50b054 to 585f546aa0c092ccd938a9a01b4bc3eb7662804d |
[production] |
| 22:05 |
<hashar> |
Zuul resarted manually from a terminal on contint1001. It does not have any statsd configuration so we will miss metrics for a bit till it is restarted properly. |
[releng] |
| 21:59 |
<gwicke> |
restarted pdfrender on scb1003; was spinning on CPU & using 15G of memory (!) |
[production] |
| 21:58 |
<gwicke> |
restarted pdfrender on scb1002 and scb1004; was spinning on CPU |
[production] |
| 21:56 |
<hashar> |
Zuul back, running in an interactive terminal. |
[production] |
| 21:46 |
<mutante> |
netmon1002 - was able to "keyholder arm" after stretch install after applying https://gerrit.wikimedia.org/r/358884 as hotfix |
[production] |
| 21:30 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@9a86d4c]: (no justification provided) (duration: 01m 06s) |
[production] |
| 21:29 |
<mobrovac@tin> |
Started deploy [restbase/deploy@9a86d4c]: (no justification provided) |
[production] |
| 21:13 |
<hashar> |
Gracefully restarting Zuul |
[production] |
| 21:13 |
<hashar> |
Gracefully restarting Zuul |
[releng] |
| 21:11 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements (duration: 01m 08s) |
[production] |
| 21:10 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements |
[production] |
| 21:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: Fix bad method name (duration: 00m 44s) |
[production] |
| 20:37 |
<hashar> |
Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4) |
[releng] |
| 20:37 |
<hashar> |
Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4) |
[production] |
| 20:31 |
<hashar> |
Nodepool: deleted a bunch of Trusty instances. It scheduled lot of them that are taking slots in the pool. Better have jessie nodes to be spawned instead since there is high demand for them |
[releng] |
| 20:19 |
<hashar> |
deployment-prep: added Polishdeveloper to the "importer" global group. https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/Polishdeveloper - T167823 |
[releng] |
| 20:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/includes/api/ApiParse.php: T167826 (duration: 00m 44s) |
[production] |
| 20:00 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@4c1cdd0]: (no justification provided) (duration: 04m 29s) |
[production] |
| 19:56 |
<mobrovac@tin> |
Started deploy [restbase/deploy@4c1cdd0]: (no justification provided) |
[production] |