|
2014-03-31
§
|
| 15:56 |
<hashar_> |
Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight |
[releng] |
| 15:10 |
<hashar_> |
Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ |
[releng] |
| 14:20 |
<hashar> |
deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish |
[releng] |
| 14:05 |
<hashar> |
shutdowning database and apache boxes for now. |
[releng] |
| 14:03 |
<hashar> |
shutdowning varnishes instances in pmtpa |
[releng] |
| 13:56 |
<hashar> |
Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 |
[releng] |
| 13:52 |
<hashar> |
upload varnish cache working :-] |
[releng] |
| 13:47 |
<hashar> |
applying role::cache::upload to role-cache-upload02 |
[releng] |
| 13:37 |
<hashar> |
migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master |
[releng] |
| 13:22 |
<hashar> |
Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" |
[releng] |
| 11:30 |
<hashar> |
Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) |
[releng] |
|
2014-03-27
§
|
| 15:23 |
<hashar> |
role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( |
[releng] |
| 15:21 |
<hashar> |
applying role::beta::natfix on deployment-bastion.eqiad |
[releng] |
| 14:58 |
<hashar> |
fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} |
[releng] |
| 13:58 |
<hashar> |
rebased puppetmaster git repository, reapplied ottomata live hacks. |
[releng] |
| 12:55 |
<hashar> |
mediawiki l10n cache being rebuild!!! |
[releng] |
| 12:54 |
<hashar> |
Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts |
[releng] |
| 11:29 |
<hashar> |
MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console |
[releng] |
| 11:11 |
<hashar> |
deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad |
[releng] |
| 10:54 |
<hashar> |
Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad |
[releng] |
|
2014-03-26
§
|
| 19:05 |
<bd808> |
Added ottomata as a project member and admin |
[releng] |
| 15:46 |
<springle> |
deployment-db1 data loaded |
[releng] |
| 14:45 |
<bd808> |
created proxy https://logstash-beta.wmflabs.org for logstash instance |
[releng] |
| 14:17 |
<hashar> |
fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649 21.5M Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} |
[releng] |
| 14:05 |
<hashar> |
udp2log functional on eqiad beta cluster \\O/ |
[releng] |
| 13:55 |
<hashar> |
stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) |
[releng] |
| 13:52 |
<hashar> |
dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull |
[releng] |
| 13:14 |
<hashar> |
Dropping enwikivoyage and dewikivoyage databases from sql02. Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ |
[releng] |
| 07:53 |
<springle> |
installed mariadb via puppet on deployment-db1. no data yet |
[releng] |
|
2014-03-24
§
|
| 23:16 |
<marktraceur> |
Touching all the MMV scripts because they're not getting invalidated or something |
[releng] |
| 23:10 |
<hashar> |
l10n cache got broken due to a PHP fatal error I introduced. It is back up now. Found out via https://integration.wikimedia.org/dashboard/ |
[releng] |
| 23:09 |
<hashar> |
upgraded all pmtpa varnishes, ran puppet on all of them. all set! |
[releng] |
| 22:57 |
<hashar> |
restarting deployment-cache-upload04 , apparently stalled\t |
[releng] |
| 22:48 |
<hashar> |
upgrading varnish on all pmtpa caches. |
[releng] |
| 22:47 |
<hashar> |
apt-get upgrade varnish on deployment-cache-bits03 |
[releng] |
| 22:45 |
<marktraceur> |
attempted restart of varnish on betalabs; seems to have failed, trying again |
[releng] |
| 22:42 |
<hashar> |
made marktraceur a project admin and granted sudo rights |
[releng] |
| 22:39 |
<marktraceur> |
Restarting betalabs varnish to workaround https://bugzilla.wikimedia.org/show_bug.cgi?id=63034 |
[releng] |
| 17:25 |
<bd808> |
Converted deployment-db1.eqiad.wmflabs to use local puppet & salt masters |
[releng] |
| 17:06 |
<bd808> |
Changed rules in sql security group to use CIDR 10.0.0.0/8. |
[releng] |
| 17:05 |
<bd808> |
Changed rules in search security group to use CIDR 10.0.0.0/8. |
[releng] |