801-850 of 2124 results (17ms)
2014-04-03 §
15:33 <bd808> Restarted logstash on deploymnet-logstash1; Stuck in a bad state due to jvm oom logged at 2014-04-03T12:03:43Z [releng]
2014-04-02 §
17:54 <manybubbles> done installing plugins on Elasticsearch in beta [releng]
14:10 <hashar> Fixed database updating job https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ . It was not running on the proper node. [releng]
12:50 <hashar> restarted parsoid daemon on deployment-parsoid04.eqiad.wmflabs. It also now log to /data/project/parsoid/parsoid.log [releng]
12:36 <hashar> Manually deleting parsoid user/group on deployment-parsoid04. Will use the LDAP uid/gid instead. [releng]
2014-04-01 §
21:38 <hashar> Removed the Zuul triggers that updated beta cluster in PMTPA {{gerrit|123100}}. [releng]
19:49 <bd808> Converted deployment-graphite.eqiad.wmflabs to use local puppet & salt masters [releng]
19:20 <bd808> Deleting and re-creating deployment-graphite because I forgot to add the web security group [releng]
15:57 <andrewbogott> shutting down all pmtpa instances [releng]
14:32 <manybubbles> completed upgrade to Elasticsearch 1.1.0 and fixed deployment-elastic04. [releng]
13:32 <hashar> Thumbs access more or less fixed [releng]
13:31 <hashar> deployment-upload is rejecting connection on port 80. Applying role::beta::uploadservice from {{gerrit|122786}} [releng]
13:30 <manybubbles> upgrading labs Elasticsearch to 1.1.0 [releng]
08:31 <hashar> MediaWiki config paths tweaks for Math {{bug|63331}} and Captchas {{bug|63342}} [releng]
00:32 <bd808> Converted deployment-graphite to use local puppet & salt masters [releng]
2014-03-31 §
21:02 <hashar> Making Parsoid daemon to write its logs to /data/project/parsoid/parsoid.log {{gerrit|122561}} [releng]
20:17 <hashar> restarted parsoid daemon [releng]
20:00 <hashar> stopped parsoid . It is killing the application servers [releng]
19:53 <hashar> restarting both apaches [releng]
19:21 <hashar> restarting job service on jobrunner01 to apply {{gerrit|122436}} [releng]
19:20 <hashar> Unbreak puppetmaster on deployment-salt.eqiad.wmflabs [releng]
19:01 <hashar> puppet master is broken :( [releng]
17:39 <hashar> lowering # of jobs spawned by the jobrunner {{gerrit|122436}} [releng]
16:00 <bd808> Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 [releng]
15:58 <bd808> Updated kibana on deployment-logstash1 to e317bc6 [releng]
15:56 <hashar_> Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight [releng]
15:10 <hashar_> Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ [releng]
14:20 <hashar> deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish [releng]
14:05 <hashar> shutdowning database and apache boxes for now. [releng]
14:03 <hashar> shutdowning varnishes instances in pmtpa [releng]
13:56 <hashar> Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 [releng]
13:52 <hashar> upload varnish cache working :-] [releng]
13:47 <hashar> applying role::cache::upload to role-cache-upload02 [releng]
13:37 <hashar> migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master [releng]
13:22 <hashar> Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" [releng]
11:30 <hashar> Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) [releng]
2014-03-28 §
16:18 <hashar> rebased puppet on deployment-salt [releng]
15:39 <hashar> Last log made to wrong project [releng]
15:39 <hashar> deleting instance ntegration-selenium-driver no more needed. browsertests jobs should now be runnable on integration-slave1001 and integration-slave1002 (in eqiad) [releng]
10:54 <hashar> deleting instance integration-debian-builder . That is breaking all debian-glue jobs. Will revisit later next week to get pbuilder/cowbuilder set up on the other eqiad slaves [releng]
08:48 <hashar> deleting integration-slave-pbuilder. Unneeded (i need a coffee) [releng]
08:43 <hashar> Created integration-slave-pbuilder on eqiad to replace pmtpa instance integration-debian-builder [releng]
00:23 <bd808> `sudo chmod -R a+rwx /data/project/upload7`; We need to get this file permissions thing figured out [releng]
2014-03-27 §
15:23 <hashar> role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( [releng]
15:21 <hashar> applying role::beta::natfix on deployment-bastion.eqiad [releng]
14:58 <hashar> fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} [releng]
13:58 <hashar> rebased puppetmaster git repository, reapplied ottomata live hacks. [releng]
12:55 <hashar> mediawiki l10n cache being rebuild!!! [releng]
12:54 <hashar> Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts [releng]
11:29 <hashar> MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console [releng]