|
2017-06-14
§
|
| 06:32 |
<moritzm> |
installing remaining libtasn security updates |
[production] |
| 03:14 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Jun 14 03:14:28 UTC 2017 (duration 6m 56s) |
[production] |
| 03:07 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.5) (duration: 14m 52s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 07m 58s) |
[production] |
| 01:48 |
<mutante> |
netmon1002 - chown rancid:rancid /var/lib/rancid ; touch /var/lib/rancid/.gitconfig, let rancid write to config, then git config --global user.email and user.name as the rancid user | fix permissions on .git/objects files, let rancid user own them all | re-commit .gitingore change | SSH_AUTH_SOCK=/run/keyholder/proxy.sock /usr/lib/rancid/bin/rancid-run as user "rancid" runs clean, |
[production] |
| 01:20 |
<mutante> |
netmon1002 - copied missing router.db, routers.all/.down/.up over from netmon1001 to /var/lib/rancid/core. routers.db is an untracked file, the others are in .gitignore. this is all like on netmon1001 as well. adding routers.db to .gitignore file on both, like the other router* files already were (T159756) |
[production] |
| 01:00 |
<mutante> |
netmon1002 - locally "git clone /var/lib/rancid/GIT/core" into /var/lib/rancid (i rsynced that but it's a bare repository without a work tree. work tree is /var/lib/rancid/core (after this) (T159756) |
[production] |
| 00:44 |
<mutante> |
naos: disarm keyholder and armed it again to proof i didn't break anything on jessie by fixing keyholder on stretch with gerrit:358884 |
[production] |
| 00:39 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: extdist update (duration: 00m 44s) |
[production] |
| 00:09 |
<aaron@tin> |
Synchronized wmf-config/InitialiseSettings.php: Capture messages on 'autoloader' debug log channel (duration: 00m 44s) |
[production] |
|
2017-06-13
§
|
| 23:29 |
<RainbowSprinkles> |
gerrit: upgrading on master 2.13.4-13-gc0c5cc4742 -> 2.13.8-1-g7c438d37a2 (been running on slave for a week) |
[production] |
| 23:13 |
<mutante> |
contint1001 - started zuul using the old init script |
[production] |
| 23:04 |
<mutante> |
netmon1001/1002: rsynced /var/lib/rancid/CVS and /var/lib/rancid/GIT from 1001 to 1002 for rancid migration (T159756) |
[production] |
| 23:04 |
<demon@tin> |
Synchronized php-1.30.0-wmf.4/extensions/OpenStackManager: Re-adding deleted special page (duration: 00m 45s) |
[production] |
| 22:06 |
<ejegg> |
updated fundraising tools from f2522cdabf1741a60b7b60ac8f7ead7afd50b054 to 585f546aa0c092ccd938a9a01b4bc3eb7662804d |
[production] |
| 22:05 |
<hashar> |
Zuul resarted manually from a terminal on contint1001. It does not have any statsd configuration so we will miss metrics for a bit till it is restarted properly. |
[releng] |
| 21:59 |
<gwicke> |
restarted pdfrender on scb1003; was spinning on CPU & using 15G of memory (!) |
[production] |
| 21:58 |
<gwicke> |
restarted pdfrender on scb1002 and scb1004; was spinning on CPU |
[production] |
| 21:56 |
<hashar> |
Zuul back, running in an interactive terminal. |
[production] |
| 21:46 |
<mutante> |
netmon1002 - was able to "keyholder arm" after stretch install after applying https://gerrit.wikimedia.org/r/358884 as hotfix |
[production] |
| 21:30 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@9a86d4c]: (no justification provided) (duration: 01m 06s) |
[production] |
| 21:29 |
<mobrovac@tin> |
Started deploy [restbase/deploy@9a86d4c]: (no justification provided) |
[production] |
| 21:13 |
<hashar> |
Gracefully restarting Zuul |
[production] |
| 21:13 |
<hashar> |
Gracefully restarting Zuul |
[releng] |
| 21:11 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements (duration: 01m 08s) |
[production] |
| 21:10 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@4ba3c59]: Rate-limiter enhancements |
[production] |
| 21:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: Fix bad method name (duration: 00m 44s) |
[production] |
| 20:37 |
<hashar> |
Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4) |
[releng] |
| 20:37 |
<hashar> |
Restarting Nodepool. apparently confused in pool tracking and spawning to many Trusty nodes (7 instead of 4) |
[production] |
| 20:31 |
<hashar> |
Nodepool: deleted a bunch of Trusty instances. It scheduled lot of them that are taking slots in the pool. Better have jessie nodes to be spawned instead since there is high demand for them |
[releng] |
| 20:19 |
<hashar> |
deployment-prep: added Polishdeveloper to the "importer" global group. https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUserRights/Polishdeveloper - T167823 |
[releng] |
| 20:02 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/includes/api/ApiParse.php: T167826 (duration: 00m 44s) |
[production] |
| 20:00 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@4c1cdd0]: (no justification provided) (duration: 04m 29s) |
[production] |
| 19:56 |
<mobrovac@tin> |
Started deploy [restbase/deploy@4c1cdd0]: (no justification provided) |
[production] |
| 19:37 |
<Amir1> |
restarting ores-related services in scb1001 (T167819) |
[production] |
| 19:24 |
<mutante> |
scb1001 - killed process 10971 (pdfrendering/electron) |
[production] |
| 19:24 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/extensions/CategoryTree/CategoryPageSubclass.php: Fix up variable visibility (duration: 00m 44s) |
[production] |
| 19:12 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5 |
[production] |
| 19:09 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@9a86d4c]: (no justification provided) (duration: 07m 33s) |
[production] |
| 19:08 |
<mutante> |
netmon1002 - reinstallled with stretch, revoked puppet cert, salt key, signing new cert, accepting new key, initial puppet run (T159756) |
[production] |
| 19:01 |
<mobrovac@tin> |
Started deploy [restbase/deploy@9a86d4c]: (no justification provided) |
[production] |
| 18:56 |
<mutante> |
reinstalling netmon1002 with stretch - scheduled icinga downtime |
[production] |
| 18:54 |
<legoktm> |
starting to delete all rows from linter tables on large wikis - T167758 |
[production] |
| 18:48 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@4c1cdd0]: (no justification provided) (duration: 04m 36s) |
[production] |
| 18:47 |
<andrewbogott> |
root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter" |
[deployment-prep] |
| 18:47 |
<andrewbogott> |
root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter" |
[releng] |
| 18:46 |
<andrewbogott> |
using salt to "apt-get -y install facter" on all deployment-prep instances |
[releng] |
| 18:43 |
<mobrovac@tin> |
Started deploy [restbase/deploy@4c1cdd0]: (no justification provided) |
[production] |
| 18:39 |
<mobrovac@tin> |
Started deploy [restbase/deploy@4c1cdd0]: (no justification provided) |
[production] |
| 18:38 |
<andrewbogott> |
restarting apache2 on deployment-puppetmaster02 |
[releng] |