|
2010-10-28
§
|
| 21:24 |
<RobH> |
sq69 is borked, powercycling |
[production] |
| 17:51 |
<Ryan_Lane> |
running checksetup.pl on kaulen for bugzilla |
[production] |
| 17:50 |
<Ryan_Lane> |
running mysqlcheck --autorepair on bugzilla database on db9 for the bug_fulltext table |
[production] |
| 15:23 |
<atglenn> |
reenabled logging for fundraising on locke |
[production] |
| 14:50 |
<atglenn> |
I see a lot of lot of ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) after reboot of db9... not awake enough to try to look at it; services seem to be running ok |
[production] |
| 14:46 |
<atglenn> |
powercycled db9, it was unreachable by ssh, ganglia showed load and wait_cpu through the roof |
[production] |
| 14:46 |
<RoanKattouw> |
db9 back after having been powercycled by Ariel |
[production] |
| 14:18 |
<RoanKattouw> |
db9 down. Responds to ping but doesn't respond to anything else |
[production] |
|
2010-10-26
§
|
| 22:34 |
<RobH> |
removed borked tenwiki deployments entirely from cluster, time to start over |
[production] |
| 18:33 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'readding ms1' |
[production] |
| 18:04 |
<RobH> |
dropping out all the tenwikipedia stuff again, internet hates this wiki |
[production] |
| 18:00 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos some more' |
[production] |
| 17:59 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos' |
[production] |
| 17:30 |
<RoanKattouw> |
Removed 10wikipedia from all.dblist and ran sync-dblist |
[production] |
| 17:08 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'sync cuz i said so' |
[production] |
| 17:04 |
<RobH> |
kicking srv183 around |
[production] |
| 16:56 |
<robh> |
synchronized php-1.5/wmf-config/mc.php 'srv183 fell over' |
[production] |
| 16:51 |
<robh> |
ran sync-common-all |
[production] |
| 16:26 |
<RobH> |
sync-docroot run, nothing blew up, yay |
[production] |
| 14:34 |
<apergos> |
powercycled mobile3. hey in case anyone ever needs to know this again, mobile3 is a dell r410 with the new drac 6. |
[production] |
| 12:47 |
<hcatlin> |
mobile3 is not responding to anything. needs restart. attempting to contact ops. |
[production] |
|
2010-10-25
§
|
| 21:50 |
<Ryan_Lane> |
pdns on ns0 not responding, restarting |
[production] |
| 21:47 |
<Ryan_Lane> |
fixing test-payments.tesla entry in dns |
[production] |
| 19:09 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25604 - Project namespace creation on frwiktionary' |
[production] |
| 19:05 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25627 - Please create various namespaces for fawikinews' |
[production] |
| 18:54 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '22283 - Add "Author" namespace to Telugu Wikisource' |
[production] |
| 18:47 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25635 - Please set the Crimean Tatar Wikipedia logo' |
[production] |
| 18:44 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25636 - Abuse filter in Persian wikinews' |
[production] |
| 18:42 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '25576 - Install AbuseFilter on simplewiki' |
[production] |
| 18:34 |
<jeluf> |
ran sync-common-all |
[production] |
| 18:32 |
<jeluf> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'bug 25634 - Please install flagged revisions on fawikinews' |
[production] |
| 18:30 |
<jeluf> |
synchronized flaggedrevs.dblist |
[production] |
| 16:09 |
<apergos> |
after restart of mysqld on db9, it now accepts requests from singer, blogs appear to be back |
[production] |
| 15:41 |
<apergos> |
powercycled singer, it was unresponsive to ssh, huge cpu spikes |
[production] |
|
2010-10-24
§
|
| 20:44 |
<mark> |
Shutting down mayflower for decommissioning |
[production] |
| 20:40 |
<mark> |
Changed root password on lily, iris, ragweed, mayflower and sage |
[production] |
| 20:37 |
<mark> |
Changed root password on browne and srv2 |
[production] |
| 19:31 |
<RobH> |
installed python-openssl on spence for testing |
[production] |
| 19:11 |
<tomaszf> |
removing downtime with init script on loudon |
[production] |
| 18:21 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 18:20 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 17:44 |
<Ryan_Lane> |
reinstalling ersch for openstack testing |
[production] |
| 17:29 |
<mark> |
Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes |
[production] |
| 17:21 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |