|
2014-04-01
§
|
| 18:19 |
<mutante> |
ms10 - disable puppet, revoke puppet cert,salt key,icinga.. |
[production] |
| 18:03 |
<ori> |
synchronized php-1.23wmf20/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name' |
[production] |
| 18:02 |
<ori> |
synchronized php-1.23wmf19/includes/profiler/ProfilerSimple.php 'Iad91f1d12: Send profiled items under the correct name' |
[production] |
| 17:34 |
<mutante> |
logging to eqiad wikitech after Andrew switched over |
[production] |
| 16:05 |
<andrewbogott> |
switching wikitech to read-only, migrating to eqiad |
[production] |
| 15:57 |
<andrewbogott> |
shutting down all pmtpa instances |
[releng] |
| 15:06 |
<reedy> |
updated /a/common to {{Gerrit|If3ca3d486}}: beta: adjust $wgCaptchaDirectory |
[production] |
| 15:01 |
<hashar> |
Gerrit super slow again :-( |
[production] |
| 14:46 |
<mutante> |
added oblivion to root-auth-keys |
[production] |
| 14:32 |
<manybubbles> |
completed upgrade to Elasticsearch 1.1.0 and fixed deployment-elastic04. |
[releng] |
| 14:17 |
<mutante> |
welcome new shell user oblivion |
[production] |
| 14:04 |
<hashar> |
Gerrit flushed a few caches related to user accounts / LDAP |
[production] |
| 13:43 |
<mutante> |
adding oblivion to ops and wmf LDAP groups |
[production] |
| 13:32 |
<hashar> |
Thumbs access more or less fixed |
[releng] |
| 13:31 |
<hashar> |
deployment-upload is rejecting connection on port 80. Applying role::beta::uploadservice from {{gerrit|122786}} |
[releng] |
| 13:30 |
<manybubbles> |
upgrading labs Elasticsearch to 1.1.0 |
[releng] |
| 08:44 |
<mutante> |
solr1/2 - revoke puppet certs |
[production] |
| 08:43 |
<mutante> |
solr3 - delete salt key, puppet cert |
[production] |
| 08:31 |
<hashar> |
MediaWiki config paths tweaks for Math {{bug|63331}} and Captchas {{bug|63342}} |
[releng] |
| 03:11 |
<springle> |
synchronized wmf-config/db-eqiad.php 's7 db1034 full steam' |
[production] |
| 02:02 |
<LocalisationUpdate> |
failed: git pull of extensions failed |
[production] |
| 01:58 |
<springle> |
killed research queries on db1047. email me |
[production] |
| 01:35 |
<springle> |
restarted sanitarium s3 instance for additional private wikis |
[production] |
| 00:59 |
<springle> |
synchronized wmf-config/db-eqiad.php 's7 repool db1034, warm up' |
[production] |
| 00:32 |
<bd808> |
Converted deployment-graphite to use local puppet & salt masters |
[releng] |
|
2014-03-31
§
|
| 23:30 |
<springle> |
synchronized wmf-config/db-eqiad.php 's7 depool db1034 for upgrade' |
[production] |
| 23:22 |
<ori> |
End of SWAT deploy. |
[production] |
| 23:21 |
<ori> |
synchronized php-1.23wmf19/extensions 'I6f0f1b18d: Update MobileFrontend, PageImages and TextExtracts for bug 63248' |
[production] |
| 23:16 |
<ori> |
synchronized php-1.23wmf20/extensions 'I6f0f1b18d: Update MobileFrontend, PageImages and TextExtracts for bug 63248' |
[production] |
| 23:05 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I532f8ee7c: Add "upload" custom debug group to $wgDebugLogGroups' |
[production] |
| 23:05 |
<ori> |
updated /a/common to {{Gerrit|I532f8ee7c}}: Add 'upload' custom debug group to wgDebugLogGroups |
[production] |
| 23:03 |
<ori> |
Starting SWAT deploy window. |
[production] |
| 21:02 |
<hashar> |
Making Parsoid daemon to write its logs to /data/project/parsoid/parsoid.log {{gerrit|122561}} |
[releng] |
| 20:36 |
<awight> |
updated crm from 5151d97044ce2fff7299cee8920ee9bd3dbbdf54 to e726e4213f1d65cae7276f91379eb1b229b19134 |
[production] |
| 20:17 |
<hashar> |
restarted parsoid daemon |
[releng] |
| 20:00 |
<hashar> |
stopped parsoid . It is killing the application servers |
[releng] |
| 19:53 |
<hashar> |
restarting both apaches |
[releng] |
| 19:31 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 19:21 |
<hashar> |
restarting job service on jobrunner01 to apply {{gerrit|122436}} |
[releng] |
| 19:20 |
<hashar> |
Unbreak puppetmaster on deployment-salt.eqiad.wmflabs |
[releng] |
| 19:01 |
<hashar> |
puppet master is broken :( |
[releng] |
| 17:39 |
<hashar> |
lowering # of jobs spawned by the jobrunner {{gerrit|122436}} |
[releng] |
| 16:37 |
<aaron> |
synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadpdf" pool counter config' |
[production] |
| 16:11 |
<bd808> |
Upgraded kibana on logstash cluster to e317bc663495d0172339a4d4ace9c2a580ceed45 |
[production] |
| 16:08 |
<mutante> |
DNS update - remove tampa search pools (a second time, was duplicate, heh) |
[production] |
| 16:00 |
<bd808> |
Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 |
[releng] |
| 15:58 |
<bd808> |
Updated kibana on deployment-logstash1 to e317bc6 |
[releng] |
| 15:56 |
<hashar_> |
Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight |
[releng] |
| 15:33 |
<Jeff_Green> |
reenable fundraising services, reenable silicon icinga monitoring |
[production] |
| 15:10 |
<hashar_> |
Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ |
[releng] |