|
2014-03-31
§
|
| 23:30 |
<springle> |
synchronized wmf-config/db-eqiad.php 's7 depool db1034 for upgrade' |
[production] |
| 23:22 |
<ori> |
End of SWAT deploy. |
[production] |
| 23:21 |
<ori> |
synchronized php-1.23wmf19/extensions 'I6f0f1b18d: Update MobileFrontend, PageImages and TextExtracts for bug 63248' |
[production] |
| 23:16 |
<ori> |
synchronized php-1.23wmf20/extensions 'I6f0f1b18d: Update MobileFrontend, PageImages and TextExtracts for bug 63248' |
[production] |
| 23:05 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I532f8ee7c: Add "upload" custom debug group to $wgDebugLogGroups' |
[production] |
| 23:05 |
<ori> |
updated /a/common to {{Gerrit|I532f8ee7c}}: Add 'upload' custom debug group to wgDebugLogGroups |
[production] |
| 23:03 |
<ori> |
Starting SWAT deploy window. |
[production] |
| 21:02 |
<hashar> |
Making Parsoid daemon to write its logs to /data/project/parsoid/parsoid.log {{gerrit|122561}} |
[releng] |
| 20:36 |
<awight> |
updated crm from 5151d97044ce2fff7299cee8920ee9bd3dbbdf54 to e726e4213f1d65cae7276f91379eb1b229b19134 |
[production] |
| 20:17 |
<hashar> |
restarted parsoid daemon |
[releng] |
| 20:00 |
<hashar> |
stopped parsoid . It is killing the application servers |
[releng] |
| 19:53 |
<hashar> |
restarting both apaches |
[releng] |
| 19:31 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 19:21 |
<hashar> |
restarting job service on jobrunner01 to apply {{gerrit|122436}} |
[releng] |
| 19:20 |
<hashar> |
Unbreak puppetmaster on deployment-salt.eqiad.wmflabs |
[releng] |
| 19:01 |
<hashar> |
puppet master is broken :( |
[releng] |
| 17:39 |
<hashar> |
lowering # of jobs spawned by the jobrunner {{gerrit|122436}} |
[releng] |
| 16:37 |
<aaron> |
synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadpdf" pool counter config' |
[production] |
| 16:11 |
<bd808> |
Upgraded kibana on logstash cluster to e317bc663495d0172339a4d4ace9c2a580ceed45 |
[production] |
| 16:08 |
<mutante> |
DNS update - remove tampa search pools (a second time, was duplicate, heh) |
[production] |
| 16:00 |
<bd808> |
Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 |
[releng] |
| 15:58 |
<bd808> |
Updated kibana on deployment-logstash1 to e317bc6 |
[releng] |
| 15:56 |
<hashar_> |
Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight |
[releng] |
| 15:33 |
<Jeff_Green> |
reenable fundraising services, reenable silicon icinga monitoring |
[production] |
| 15:10 |
<hashar_> |
Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ |
[releng] |
| 14:28 |
<bd808> |
Started logstash on logstash1001 |
[production] |
| 14:23 |
<bd808> |
Elasticsearch upgraded to 1.0.1 on logstash100[123] |
[production] |
| 14:20 |
<hashar> |
deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish |
[releng] |
| 14:18 |
<hashar> |
beta cluster DNS entries migrated to point to the EQIAD datacenter. Keeping pmtpa instances around for a couple days |
[production] |
| 14:16 |
<bd808> |
Stopped elasticsearch on logstash100[123] |
[production] |
| 14:13 |
<bd808> |
Stopped logstash on logstash1001 |
[production] |
| 14:05 |
<hashar> |
shutdowning database and apache boxes for now. |
[releng] |
| 14:03 |
<hashar> |
shutdowning varnishes instances in pmtpa |
[releng] |
| 13:56 |
<hashar> |
Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 |
[releng] |
| 13:52 |
<hashar> |
upload varnish cache working :-] |
[releng] |
| 13:48 |
<Jeff_Green> |
icinga alerts disabled for silicon |
[production] |
| 13:47 |
<hashar> |
applying role::cache::upload to role-cache-upload02 |
[releng] |
| 13:37 |
<hashar> |
migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master |
[releng] |
| 13:34 |
<Jeff_Green> |
fundraising-system full-stop for hardware repairs on queue server |
[production] |
| 13:22 |
<hashar> |
Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" |
[releng] |
| 11:57 |
<hashar> |
Jenkins: updating jslint jobs to run a PHP based json linter, will bails out whenever json files are invalid. {{bug|58279}} {{gerrit|113958}} |
[production] |
| 11:30 |
<hashar> |
Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) |
[releng] |
| 10:48 |
<hashar> |
Jenkins fixed up browsertests jobs. Bundler could not compile gems on the eqiad slaves {{gerrit|122346}} |
[production] |