|
2014-03-31
§
|
| 20:00 |
<hashar> |
stopped parsoid . It is killing the application servers |
[releng] |
| 19:53 |
<hashar> |
restarting both apaches |
[releng] |
| 19:31 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 19:21 |
<hashar> |
restarting job service on jobrunner01 to apply {{gerrit|122436}} |
[releng] |
| 19:20 |
<hashar> |
Unbreak puppetmaster on deployment-salt.eqiad.wmflabs |
[releng] |
| 19:01 |
<hashar> |
puppet master is broken :( |
[releng] |
| 17:39 |
<hashar> |
lowering # of jobs spawned by the jobrunner {{gerrit|122436}} |
[releng] |
| 16:37 |
<aaron> |
synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadpdf" pool counter config' |
[production] |
| 16:11 |
<bd808> |
Upgraded kibana on logstash cluster to e317bc663495d0172339a4d4ace9c2a580ceed45 |
[production] |
| 16:08 |
<mutante> |
DNS update - remove tampa search pools (a second time, was duplicate, heh) |
[production] |
| 16:00 |
<bd808> |
Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 |
[releng] |
| 15:58 |
<bd808> |
Updated kibana on deployment-logstash1 to e317bc6 |
[releng] |
| 15:56 |
<hashar_> |
Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight |
[releng] |
| 15:33 |
<Jeff_Green> |
reenable fundraising services, reenable silicon icinga monitoring |
[production] |
| 15:10 |
<hashar_> |
Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ |
[releng] |
| 14:28 |
<bd808> |
Started logstash on logstash1001 |
[production] |
| 14:23 |
<bd808> |
Elasticsearch upgraded to 1.0.1 on logstash100[123] |
[production] |
| 14:20 |
<hashar> |
deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish |
[releng] |
| 14:18 |
<hashar> |
beta cluster DNS entries migrated to point to the EQIAD datacenter. Keeping pmtpa instances around for a couple days |
[production] |
| 14:16 |
<bd808> |
Stopped elasticsearch on logstash100[123] |
[production] |
| 14:13 |
<bd808> |
Stopped logstash on logstash1001 |
[production] |
| 14:05 |
<hashar> |
shutdowning database and apache boxes for now. |
[releng] |
| 14:03 |
<hashar> |
shutdowning varnishes instances in pmtpa |
[releng] |
| 13:56 |
<hashar> |
Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 |
[releng] |
| 13:52 |
<hashar> |
upload varnish cache working :-] |
[releng] |
| 13:48 |
<Jeff_Green> |
icinga alerts disabled for silicon |
[production] |
| 13:47 |
<hashar> |
applying role::cache::upload to role-cache-upload02 |
[releng] |
| 13:37 |
<hashar> |
migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master |
[releng] |
| 13:34 |
<Jeff_Green> |
fundraising-system full-stop for hardware repairs on queue server |
[production] |
| 13:22 |
<hashar> |
Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" |
[releng] |
| 11:57 |
<hashar> |
Jenkins: updating jslint jobs to run a PHP based json linter, will bails out whenever json files are invalid. {{bug|58279}} {{gerrit|113958}} |
[production] |
| 11:30 |
<hashar> |
Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) |
[releng] |
| 10:48 |
<hashar> |
Jenkins fixed up browsertests jobs. Bundler could not compile gems on the eqiad slaves {{gerrit|122346}} |
[production] |
| 09:46 |
<hashar> |
Jenkins: applied hasBrowserTests label on both labs slave. Unblocks the browser tests which were still tied to a deleted instance {{gerrit|122341}} |
[production] |
| 07:23 |
<faidon> |
synchronized php-1.23wmf20/extensions/SpamBlacklist/SpamBlacklist.php 'revert I694860b - SpamBlacklist' |
[production] |
| 07:22 |
<faidon> |
synchronized php-1.23wmf19/extensions/SpamBlacklist/SpamBlacklist.php 'revert I694860b - SpamBlacklist' |
[production] |
| 05:47 |
<faidon> |
synchronized php-1.23wmf19/extensions/SpamBlacklist/SpamBlacklist.php 'local hack to test memcached traffic increase theory' |
[production] |
| 03:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Mar 31 03:08:39 UTC 2014 (duration 8m 38s) |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.23wmf20) at 2014-03-31 02:30:26+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf19) at 2014-03-31 02:13:38+00:00 |
[production] |