|
2017-06-14
§
|
| 13:35 |
<zfilipin@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 12:24 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Switchover pc2004 to db2072 (duration: 00m 43s) |
[production] |
| 12:13 |
<akosiaris> |
upload apertium-spa-ita_0.2.0~r78826-1+wmf to apt.wikimedia.org/jessie-wikimedia/main |
[production] |
| 12:13 |
<akosiaris> |
upload apertium-fra-cat_1.2.0~r78602-1+wmf to apt.wikimedia.org/jessie-wikimedia/main |
[production] |
| 11:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Switchover pc1004 to db1096 (duration: 00m 54s) |
[production] |
| 11:34 |
<jynus> |
about to deploy performance-impacting change on the parsercache persistent storage T167567 |
[production] |
| 11:19 |
<marostegui> |
Deploy alter table s4 - labsdb1011 - T166206 |
[production] |
| 09:46 |
<marostegui> |
Rename table titlekey before dropping it on enwiki - db1089 - T164949 |
[production] |
| 09:18 |
<godog> |
delete files older than 365d from 'servers' graphite hierarchy |
[production] |
| 07:59 |
<marostegui> |
Drop table updates on s3 - T139342 |
[production] |
| 07:32 |
<moritzm> |
installing zziplib security updates on jessie |
[production] |
| 07:04 |
<elukey> |
restart pdfrender on scb200[2,4] (xpra race condition) |
[production] |
| 07:03 |
<elukey> |
restart pdfrender on scb1004 (xpra race condition) |
[production] |
| 06:32 |
<moritzm> |
installing remaining libtasn security updates |
[production] |
| 03:14 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Jun 14 03:14:28 UTC 2017 (duration 6m 56s) |
[production] |
| 03:07 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.5) (duration: 14m 52s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 07m 58s) |
[production] |
| 01:48 |
<mutante> |
netmon1002 - chown rancid:rancid /var/lib/rancid ; touch /var/lib/rancid/.gitconfig, let rancid write to config, then git config --global user.email and user.name as the rancid user | fix permissions on .git/objects files, let rancid user own them all | re-commit .gitingore change | SSH_AUTH_SOCK=/run/keyholder/proxy.sock /usr/lib/rancid/bin/rancid-run as user "rancid" runs clean, |
[production] |
| 01:20 |
<mutante> |
netmon1002 - copied missing router.db, routers.all/.down/.up over from netmon1001 to /var/lib/rancid/core. routers.db is an untracked file, the others are in .gitignore. this is all like on netmon1001 as well. adding routers.db to .gitignore file on both, like the other router* files already were (T159756) |
[production] |
| 01:00 |
<mutante> |
netmon1002 - locally "git clone /var/lib/rancid/GIT/core" into /var/lib/rancid (i rsynced that but it's a bare repository without a work tree. work tree is /var/lib/rancid/core (after this) (T159756) |
[production] |
| 00:44 |
<mutante> |
naos: disarm keyholder and armed it again to proof i didn't break anything on jessie by fixing keyholder on stretch with gerrit:358884 |
[production] |
| 00:39 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: extdist update (duration: 00m 44s) |
[production] |
| 00:09 |
<aaron@tin> |
Synchronized wmf-config/InitialiseSettings.php: Capture messages on 'autoloader' debug log channel (duration: 00m 44s) |
[production] |
|
2017-06-13
§
|
| 23:29 |
<RainbowSprinkles> |
gerrit: upgrading on master 2.13.4-13-gc0c5cc4742 -> 2.13.8-1-g7c438d37a2 (been running on slave for a week) |
[production] |
| 23:13 |
<mutante> |
contint1001 - started zuul using the old init script |
[production] |
| 23:04 |
<mutante> |
netmon1001/1002: rsynced /var/lib/rancid/CVS and /var/lib/rancid/GIT from 1001 to 1002 for rancid migration (T159756) |
[production] |
| 23:04 |
<demon@tin> |
Synchronized php-1.30.0-wmf.4/extensions/OpenStackManager: Re-adding deleted special page (duration: 00m 45s) |
[production] |
| 22:06 |
<ejegg> |
updated fundraising tools from f2522cdabf1741a60b7b60ac8f7ead7afd50b054 to 585f546aa0c092ccd938a9a01b4bc3eb7662804d |
[production] |
| 21:59 |
<gwicke> |
restarted pdfrender on scb1003; was spinning on CPU & using 15G of memory (!) |
[production] |
| 21:58 |
<gwicke> |
restarted pdfrender on scb1002 and scb1004; was spinning on CPU |
[production] |
| 21:56 |
<hashar> |
Zuul back, running in an interactive terminal. |
[production] |
| 21:46 |
<mutante> |
netmon1002 - was able to "keyholder arm" after stretch install after applying https://gerrit.wikimedia.org/r/358884 as hotfix |
[production] |
| 21:30 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@9a86d4c]: (no justification provided) (duration: 01m 06s) |
[production] |
| 21:29 |
<mobrovac@tin> |
Started deploy [restbase/deploy@9a86d4c]: (no justification provided) |
[production] |
| 21:13 |
<hashar> |
Gracefully restarting Zuul |
[production] |
| 21:11 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements (duration: 01m 08s) |
[production] |
| 21:10 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements |
[production] |
| 21:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: Fix bad method name (duration: 00m 44s) |
[production] |
| 20:37 |
<hashar> |
Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4) |
[production] |
| 20:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/includes/api/ApiParse.php: T167826 (duration: 00m 44s) |
[production] |
| 20:00 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@4c1cdd0]: (no justification provided) (duration: 04m 29s) |
[production] |
| 19:56 |
<mobrovac@tin> |
Started deploy [restbase/deploy@4c1cdd0]: (no justification provided) |
[production] |
| 19:37 |
<Amir1> |
restarting ores-related services in scb1001 (T167819) |
[production] |
| 19:24 |
<mutante> |
scb1001 - killed process 10971 (pdfrendering/electron) |
[production] |
| 19:24 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/extensions/CategoryTree/CategoryPageSubclass.php: Fix up variable visibility (duration: 00m 44s) |
[production] |
| 19:12 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5 |
[production] |
| 19:09 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@9a86d4c]: (no justification provided) (duration: 07m 33s) |
[production] |
| 19:08 |
<mutante> |
netmon1002 - reinstallled with stretch, revoked puppet cert, salt key, signing new cert, accepting new key, initial puppet run (T159756) |
[production] |
| 19:01 |
<mobrovac@tin> |
Started deploy [restbase/deploy@9a86d4c]: (no justification provided) |
[production] |
| 18:56 |
<mutante> |
reinstalling netmon1002 with stretch - scheduled icinga downtime |
[production] |