1301-1350 of 2639 results (13ms)
2014-04-10 §
21:13 <bd808> Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ to work on scap setup [releng]
14:52 <hashar> Adding Tobias Gritschacher to the project so he can look at udp2log / apache logs whenever needed :-] [releng]
2014-04-09 §
23:04 <bd808> Re-enabled puppet on deployment-apache02 and forced a puppet run [releng]
21:39 <bd808> Cherry-picked I8f77e0c into puppet and forced puppet run on deployment-bastion [releng]
2014-04-08 §
17:53 <manybubbles> rebuilding simplewiki's search index optimized for the new highlighter to check the size difference [releng]
05:34 <Ryan_Lane> upgraded libssl on all nodes, restarted affected ssl servers [releng]
05:03 <Ryan_Lane> upgraded libssl on all salt accessible nodes [releng]
2014-04-05 §
11:19 <hashar> Attempting to reenable SSL support with {{gerrit|124057}} [releng]
2014-04-04 §
21:39 <bd808> Restarted logstash; it stopped processing events again at 2014-04-04T19:56:46Z [releng]
17:31 <bd808> Forced puppet run on deployment-cache-text02 [releng]
17:29 <bd808> Manually fixed puppet config on deployment-cache-text02 (the cert html error problem) [releng]
17:22 <bd808> Rebooting deployment-cache-bits01 [releng]
17:21 <bd808> Forced puppet run on deployment-cache-bits01 [releng]
16:15 <manybubbles> Performing a rolling restart of Elasticsearch nodes to pick up a new plugin [releng]
2014-04-03 §
17:32 <bd808> Fixed certname in /etc/puppet/puppet.conf manually on deployment-bastion so puppet would run again. [releng]
15:33 <bd808> Restarted logstash on deploymnet-logstash1; Stuck in a bad state due to jvm oom logged at 2014-04-03T12:03:43Z [releng]
2014-04-02 §
17:54 <manybubbles> done installing plugins on Elasticsearch in beta [releng]
14:10 <hashar> Fixed database updating job https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ . It was not running on the proper node. [releng]
12:50 <hashar> restarted parsoid daemon on deployment-parsoid04.eqiad.wmflabs. It also now log to /data/project/parsoid/parsoid.log [releng]
12:36 <hashar> Manually deleting parsoid user/group on deployment-parsoid04. Will use the LDAP uid/gid instead. [releng]
2014-04-01 §
21:38 <hashar> Removed the Zuul triggers that updated beta cluster in PMTPA {{gerrit|123100}}. [releng]
19:49 <bd808> Converted deployment-graphite.eqiad.wmflabs to use local puppet & salt masters [releng]
19:20 <bd808> Deleting and re-creating deployment-graphite because I forgot to add the web security group [releng]
15:57 <andrewbogott> shutting down all pmtpa instances [releng]
14:32 <manybubbles> completed upgrade to Elasticsearch 1.1.0 and fixed deployment-elastic04. [releng]
13:32 <hashar> Thumbs access more or less fixed [releng]
13:31 <hashar> deployment-upload is rejecting connection on port 80. Applying role::beta::uploadservice from {{gerrit|122786}} [releng]
13:30 <manybubbles> upgrading labs Elasticsearch to 1.1.0 [releng]
08:31 <hashar> MediaWiki config paths tweaks for Math {{bug|63331}} and Captchas {{bug|63342}} [releng]
00:32 <bd808> Converted deployment-graphite to use local puppet & salt masters [releng]
2014-03-31 §
21:02 <hashar> Making Parsoid daemon to write its logs to /data/project/parsoid/parsoid.log {{gerrit|122561}} [releng]
20:17 <hashar> restarted parsoid daemon [releng]
20:00 <hashar> stopped parsoid . It is killing the application servers [releng]
19:53 <hashar> restarting both apaches [releng]
19:21 <hashar> restarting job service on jobrunner01 to apply {{gerrit|122436}} [releng]
19:20 <hashar> Unbreak puppetmaster on deployment-salt.eqiad.wmflabs [releng]
19:01 <hashar> puppet master is broken :( [releng]
17:39 <hashar> lowering # of jobs spawned by the jobrunner {{gerrit|122436}} [releng]
16:00 <bd808> Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 [releng]
15:58 <bd808> Updated kibana on deployment-logstash1 to e317bc6 [releng]
15:56 <hashar_> Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight [releng]
15:10 <hashar_> Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ [releng]
14:20 <hashar> deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish [releng]
14:05 <hashar> shutdowning database and apache boxes for now. [releng]
14:03 <hashar> shutdowning varnishes instances in pmtpa [releng]
13:56 <hashar> Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 [releng]
13:52 <hashar> upload varnish cache working :-] [releng]
13:47 <hashar> applying role::cache::upload to role-cache-upload02 [releng]
13:37 <hashar> migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master [releng]
13:22 <hashar> Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" [releng]