2051-2100 of 3357 results (14ms)
2014-03-31 §
20:00 <hashar> stopped parsoid . It is killing the application servers [releng]
19:53 <hashar> restarting both apaches [releng]
19:21 <hashar> restarting job service on jobrunner01 to apply {{gerrit|122436}} [releng]
19:20 <hashar> Unbreak puppetmaster on deployment-salt.eqiad.wmflabs [releng]
19:01 <hashar> puppet master is broken :( [releng]
17:39 <hashar> lowering # of jobs spawned by the jobrunner {{gerrit|122436}} [releng]
16:00 <bd808> Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 [releng]
15:58 <bd808> Updated kibana on deployment-logstash1 to e317bc6 [releng]
15:56 <hashar_> Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight [releng]
15:10 <hashar_> Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ [releng]
14:20 <hashar> deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish [releng]
14:05 <hashar> shutdowning database and apache boxes for now. [releng]
14:03 <hashar> shutdowning varnishes instances in pmtpa [releng]
13:56 <hashar> Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 [releng]
13:52 <hashar> upload varnish cache working :-] [releng]
13:47 <hashar> applying role::cache::upload to role-cache-upload02 [releng]
13:37 <hashar> migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master [releng]
13:22 <hashar> Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" [releng]
11:30 <hashar> Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) [releng]
2014-03-28 §
16:18 <hashar> rebased puppet on deployment-salt [releng]
15:39 <hashar> Last log made to wrong project [releng]
15:39 <hashar> deleting instance ntegration-selenium-driver no more needed. browsertests jobs should now be runnable on integration-slave1001 and integration-slave1002 (in eqiad) [releng]
10:54 <hashar> deleting instance integration-debian-builder . That is breaking all debian-glue jobs. Will revisit later next week to get pbuilder/cowbuilder set up on the other eqiad slaves [releng]
08:48 <hashar> deleting integration-slave-pbuilder. Unneeded (i need a coffee) [releng]
08:43 <hashar> Created integration-slave-pbuilder on eqiad to replace pmtpa instance integration-debian-builder [releng]
00:23 <bd808> `sudo chmod -R a+rwx /data/project/upload7`; We need to get this file permissions thing figured out [releng]
2014-03-27 §
15:23 <hashar> role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( [releng]
15:21 <hashar> applying role::beta::natfix on deployment-bastion.eqiad [releng]
14:58 <hashar> fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} [releng]
13:58 <hashar> rebased puppetmaster git repository, reapplied ottomata live hacks. [releng]
12:55 <hashar> mediawiki l10n cache being rebuild!!! [releng]
12:54 <hashar> Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts [releng]
11:29 <hashar> MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console [releng]
11:11 <hashar> deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad [releng]
10:54 <hashar> Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad [releng]
2014-03-26 §
19:05 <bd808> Added ottomata as a project member and admin [releng]
15:46 <springle> deployment-db1 data loaded [releng]
14:45 <bd808> created proxy https://logstash-beta.wmflabs.org for logstash instance [releng]
14:17 <hashar> fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649 21.5M Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} [releng]
14:05 <hashar> udp2log functional on eqiad beta cluster \\O/ [releng]
13:55 <hashar> stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) [releng]
13:52 <hashar> dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull [releng]
13:14 <hashar> Dropping enwikivoyage and dewikivoyage databases from sql02. Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ [releng]
07:53 <springle> installed mariadb via puppet on deployment-db1. no data yet [releng]
2014-03-25 §
19:43 <hashar> created jenkins slave deployment-bastion.eqiad [releng]
17:17 <hashar> Created and validated job that updates Parsoid on the EQIAD beta cluster \\O/ [releng]
2014-03-24 §
23:16 <marktraceur> Touching all the MMV scripts because they're not getting invalidated or something [releng]
23:10 <hashar> l10n cache got broken due to a PHP fatal error I introduced. It is back up now. Found out via https://integration.wikimedia.org/dashboard/ [releng]
23:09 <hashar> upgraded all pmtpa varnishes, ran puppet on all of them. all set! [releng]
22:57 <hashar> restarting deployment-cache-upload04 , apparently stalled\t [releng]