|
2010-06-03
§
|
| 09:04 |
<Tim> |
cleaning COSS on sq45, resynced its configuration, will start squid when done |
[production] |
| 08:58 |
<Tim> |
kernel reports degraded RAID on sq33, sq34, sq35, sq37, sq38, sq40 |
[production] |
| 08:39 |
<Tim> |
checked all serial consoles, all nonresponsive, rebooted all |
[production] |
| 08:23 |
<Tim> |
sq33, sq34, sq35, sq37, sq38, sq40, sq45 have been down for 16-28 days, apparently for no good reason, can't find any log or DT entries. Will try restarts. |
[production] |
| 07:56 |
<Tim> |
added new squids to nagios |
[production] |
| 06:36 |
<Tim> |
cleaning cache directories on sq56 to avoid resurrection of expired content |
[production] |
| 06:35 |
<Tim> |
adding monitoring for rather important service IPs: upload.esams and text.esams |
[production] |
| 06:22 |
<Tim> |
sq56 not responding to ping or serial console (for 4 days), nothing in racadm getsel, rebooting |
[production] |
| 06:07 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling ClickTracking due to CR r58099' |
[production] |
| 05:24 |
<Tim> |
started apache on srv216, was stopped for some reason |
[production] |
| 03:57 |
<Fred> |
shutting down mailman on list for a few minutes while exim and spamd catch up |
[production] |
| 01:42 |
<Tim> |
adding forward and reverse DNS for mobile.tesla.usability.wikimedia.org, 208.80.152.245 |
[production] |
|
2010-06-02
§
|
| 23:48 |
<Fred> |
re-imaging mobile2 again ;p |
[production] |
| 21:12 |
<robh> |
ran sync-common-all |
[production] |
| 20:57 |
<Rob> |
srv134 has temp errors and multiple bad fans, out of warranty, decommissioning and removing from nagios/lvs/dsh |
[production] |
| 20:52 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '23756 - Enable "Rollbacker" group on itwiki' |
[production] |
| 20:44 |
<Rob> |
how many ops does it take to bring srv281 back online? one + a vounteer to point out he did it wrong. |
[production] |
| 20:42 |
<robh> |
ran sync-common-all |
[production] |
| 20:36 |
<robh> |
ran sync-common-all |
[production] |
| 20:26 |
<jeluf> |
synchronizing Wikimedia installation... Revision: 66620 |
[production] |
| 20:15 |
<Rob> |
srv281 back online and apache files updated |
[production] |
| 20:08 |
<Rob> |
lomaria is down, because it was throwing crap around and generally broken, so its shutdown now. |
[production] |
| 19:59 |
<Rob> |
had to swap power cable on msw-b5-sdtpa |
[production] |
| 14:02 |
<Rob> |
stay off db28, fixing its firmware. |
[production] |
| 13:44 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
| 13:39 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
| 13:38 |
<Tim> |
ArchRunner on lily was stuck in a busy loop of some kind, had to restart it with kill -9 |
[production] |
| 13:36 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
| 13:35 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
| 13:32 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php |
[production] |
| 13:32 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php |
[production] |
| 13:31 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.i18n.php |
[production] |
| 13:31 |
<andrew> |
synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php |
[production] |
| 06:49 |
<tstarling> |
synchronized php-1.5/extensions/CodeReview/backend/Subversion.php 'r67213' |
[production] |
| 06:45 |
<Tim> |
updated codereview-proxy on isidore to r67211 |
[production] |
| 06:13 |
<Tim> |
attempting to set up CodeReview for pywikipedia |
[production] |
| 05:48 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '23705 - Change namespaces in ja.wikipedia' |
[production] |
| 05:33 |
<jeluf> |
synchronized php-1.5/wmf-config/flaggedrevs.php '23742 - make category namespace flaggable on en.wikinews' |
[production] |
| 05:29 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '23572 - Imagemap cannot accept dynamic links.' |
[production] |
|
2010-06-01
§
|
| 23:10 |
<Fred> |
mobile2 re-installed and puppetized |
[production] |
| 22:44 |
<Fred> |
re-imaging mobile2 for total lucid puppetization. |
[production] |
| 20:17 |
<Rob> |
mobile1 decided no more ssh server, its reinstalled and defined in puppet so it is ok now. |
[production] |
| 19:42 |
<catrope> |
synchronized php-1.5/languages/messages/MessagesCkb_arab.php 'r67198' |
[production] |
| 14:26 |
<catrope> |
synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php 'r67183' |
[production] |
| 14:10 |
<domas> |
livehacked the CategoryTreeFunctions.php to fix a fatal error :) |
[production] |
| 14:09 |
<midom> |
synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php |
[production] |
| 12:02 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 66620 |
[production] |
| 12:01 |
<Tim> |
updating CategoryTree to r67181 |
[production] |
| 05:56 |
<midom> |
synchronized php-1.5/includes/db/Database.php |
[production] |
| 05:52 |
<midom> |
synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php |
[production] |