351-400 of 1698 results (8ms)
2014-04-16 §
13:10 <manybubbles> rolling restart of Elasticsearch nodes in beta to make super sure it picked up new plugins [releng]
09:33 <hashar> rebased puppetmaster [releng]
2014-04-15 §
20:02 <manybubbles> restarting elasticsearch in beta to pick up a plugin update - no downtime should occur [releng]
14:24 <hashar> rebased puppetmaster [releng]
2014-04-11 §
17:41 <bd808> Tried to enable role::protoproxy::ssl::beta on deployment-cache-text02 but it failed to apply because /etc/ssl/certs/star.wmflabs.org.pem and /etc/ssl/private/star.wmflabs.org.key don't match. [releng]
03:59 <bd808> sudo apt-get install mysql-client on deployment-bastion [releng]
03:54 <bd808> Added legoktm as a project member [releng]
00:02 <bd808> Enabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ [releng]
2014-04-10 §
21:35 <bd808> Running scap on deployment-bastion for the first time in eqiad [releng]
21:13 <bd808> Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ to work on scap setup [releng]
14:52 <hashar> Adding Tobias Gritschacher to the project so he can look at udp2log / apache logs whenever needed :-] [releng]
2014-04-09 §
23:04 <bd808> Re-enabled puppet on deployment-apache02 and forced a puppet run [releng]
21:39 <bd808> Cherry-picked I8f77e0c into puppet and forced puppet run on deployment-bastion [releng]
2014-04-08 §
17:53 <manybubbles> rebuilding simplewiki's search index optimized for the new highlighter to check the size difference [releng]
05:34 <Ryan_Lane> upgraded libssl on all nodes, restarted affected ssl servers [releng]
05:03 <Ryan_Lane> upgraded libssl on all salt accessible nodes [releng]
2014-04-05 §
11:19 <hashar> Attempting to reenable SSL support with {{gerrit|124057}} [releng]
2014-04-04 §
21:39 <bd808> Restarted logstash; it stopped processing events again at 2014-04-04T19:56:46Z [releng]
17:31 <bd808> Forced puppet run on deployment-cache-text02 [releng]
17:29 <bd808> Manually fixed puppet config on deployment-cache-text02 (the cert html error problem) [releng]
17:22 <bd808> Rebooting deployment-cache-bits01 [releng]
17:21 <bd808> Forced puppet run on deployment-cache-bits01 [releng]
16:15 <manybubbles> Performing a rolling restart of Elasticsearch nodes to pick up a new plugin [releng]
2014-04-03 §
17:32 <bd808> Fixed certname in /etc/puppet/puppet.conf manually on deployment-bastion so puppet would run again. [releng]
15:33 <bd808> Restarted logstash on deploymnet-logstash1; Stuck in a bad state due to jvm oom logged at 2014-04-03T12:03:43Z [releng]
2014-04-02 §
17:54 <manybubbles> done installing plugins on Elasticsearch in beta [releng]
14:10 <hashar> Fixed database updating job https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ . It was not running on the proper node. [releng]
12:50 <hashar> restarted parsoid daemon on deployment-parsoid04.eqiad.wmflabs. It also now log to /data/project/parsoid/parsoid.log [releng]
12:36 <hashar> Manually deleting parsoid user/group on deployment-parsoid04. Will use the LDAP uid/gid instead. [releng]
2014-04-01 §
21:38 <hashar> Removed the Zuul triggers that updated beta cluster in PMTPA {{gerrit|123100}}. [releng]
19:49 <bd808> Converted deployment-graphite.eqiad.wmflabs to use local puppet & salt masters [releng]
19:20 <bd808> Deleting and re-creating deployment-graphite because I forgot to add the web security group [releng]
15:57 <andrewbogott> shutting down all pmtpa instances [releng]
14:32 <manybubbles> completed upgrade to Elasticsearch 1.1.0 and fixed deployment-elastic04. [releng]
13:32 <hashar> Thumbs access more or less fixed [releng]
13:31 <hashar> deployment-upload is rejecting connection on port 80. Applying role::beta::uploadservice from {{gerrit|122786}} [releng]
13:30 <manybubbles> upgrading labs Elasticsearch to 1.1.0 [releng]
08:31 <hashar> MediaWiki config paths tweaks for Math {{bug|63331}} and Captchas {{bug|63342}} [releng]
00:32 <bd808> Converted deployment-graphite to use local puppet & salt masters [releng]
2014-03-31 §
21:02 <hashar> Making Parsoid daemon to write its logs to /data/project/parsoid/parsoid.log {{gerrit|122561}} [releng]
20:17 <hashar> restarted parsoid daemon [releng]
20:00 <hashar> stopped parsoid . It is killing the application servers [releng]
19:53 <hashar> restarting both apaches [releng]
19:21 <hashar> restarting job service on jobrunner01 to apply {{gerrit|122436}} [releng]
19:20 <hashar> Unbreak puppetmaster on deployment-salt.eqiad.wmflabs [releng]
19:01 <hashar> puppet master is broken :( [releng]
17:39 <hashar> lowering # of jobs spawned by the jobrunner {{gerrit|122436}} [releng]
16:00 <bd808> Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 [releng]
15:58 <bd808> Updated kibana on deployment-logstash1 to e317bc6 [releng]
15:56 <hashar_> Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight [releng]