|
2009-12-04
§
|
| 16:14 |
<Rob> |
srv245 down and unresponsive, rebooting |
[production] |
| 16:12 |
<Rob> |
sq43's replacement disk is also bad (talk about bad luck), placing rma with dell. system will remain powered down for now. |
[production] |
| 15:55 |
<Rob> |
sq43 isn't seeing a replaced disk, rebooting and troubleshooting |
[production] |
| 15:33 |
<domas> |
'arcconf setcache 1 logicaldrive 0 roff ' - disabling any read caching on db11-db30 RAIDs |
[production] |
| 15:13 |
<Rob> |
after tinkering with it with domas, it appears rebuild is indeed automatic. db21 rebuilding raid array |
[production] |
| 15:09 |
<Rob> |
db21 bad disk swapped out, rebuild should be automatic |
[production] |
| 14:57 |
<Rob> |
sq14 back up, rebuilding its cache |
[production] |
| 14:54 |
<Rob> |
sq13 primary disk dead, out of warranty |
[production] |
| 14:53 |
<Rob> |
swapping sdc in sq13 and sq14 to bring sq14 back online |
[production] |
| 14:53 |
<Rob> |
sq14 disk sdc dead, out of warranty. |
[production] |
| 05:18 |
<Tim> |
on fenari: running all pending renameUser jobs from enwiki |
[production] |
| 03:37 |
<Tim> |
Around 03:12, accidentally renamed enwiki's job table and so renamed it back a second later. This caused all slaves to stop due to a replication bug. Fixed now. |
[production] |
| 03:25 |
<Tim> |
testing fixJobQueueExplosion.php on commonswiki |
[production] |
| 02:46 |
<Tim> |
srv156 not responding to ssh, trying reboot |
[production] |
| 01:13 |
<Tim> |
restarting job runners |
[production] |
| 01:13 |
<tstarling> |
synchronized php-1.5/includes/HTMLCacheUpdate.php 'patching out all category backlink updates, major bug causing job queue to stall' |
[production] |
| 00:12 |
<Tim> |
granted access to root@fenari on all servers in the mysql node group |
[production] |
|
2009-12-03
§
|
| 23:46 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Allow bcrats to add and remove new arbcom group on nlwiki' |
[production] |
| 23:40 |
<RoanKattouw> |
Synced InitiatiseSettings.php: allow bcrats to add and remove new arbcom group on nlwiki |
[production] |
| 22:49 |
<RoanKattouw> |
Importing 365 images into Commons as User:GeographBot, requested by Multichill |
[production] |
| 22:39 |
<RoanKattouw> |
Synced InitialiseSettings.php for bug 21238: self-removal of flood flag on plwiki |
[production] |
| 22:33 |
<RoanKattouw> |
Synced InitialiseSettings.php for bugs 20775 and 21719. sync-file is stalling on what seems to be an unresponsive server |
[production] |
| 21:35 |
<RoanKattouw> |
Running namespaceDupes on usabilitywiki for bug 21753 |
[production] |
| 21:35 |
<RoanKattouw> |
catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21753 Fix Multimedia talk NS on usabilitywiki' |
[production] |
| 04:20 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'fixing conversion rate bugs' |
[production] |
|
2009-12-02
§
|
| 23:28 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'reenabling db18 and db25, also, attempting to overwrite stale db.php copies' |
[production] |
| 23:25 |
<Fred> |
massaged mc.php to retrieve working spare, and remove broken memcached nodes. all is now good in the land of memcache |
[production] |
| 22:13 |
<mark> |
Recovered torrus from deadlock |
[production] |
| 21:00 |
<Fred> |
rebooted srv194 (hung) |
[production] |
| 20:48 |
<Rob> |
removed bayle and khaldun from dsh, both are in rack running wipe with network pulled |
[production] |
| 20:38 |
<Fred> |
bart removed from nagios (well that sounds funny) |
[production] |
| 20:36 |
<Rob> |
khaldun is down forever! decomissioned and running wipe in rack with the network pulled |
[production] |
| 20:35 |
<Rob> |
isidore rebooted by accident due to power cable issues |
[production] |
| 20:21 |
<Rob> |
srv136 crashed with temp warnings, going to decommission it, rebooting to wipe and remove network |
[production] |
| 20:15 |
<Rob> |
bart decommissioned, unracked, wipe running on testbench with usbcdrom |
[production] |
| 19:49 |
<Rob> |
decommissioned, unracked srv66, srv51, srv81, srv118 (previously removed from pybal) |
[production] |
| 19:39 |
<Rob> |
decommissioned srv130, unracked |
[production] |
| 19:20 |
<Rob> |
srv122 decommissioned, wiped, unracked |
[production] |
| 18:19 |
<Rob> |
ms7/ms8 racked in sdtpa a2, network wired, dns setup, racktables updated, & LOM online |
[production] |
| 18:18 |
<Rob> |
serial connection to ps1-a4-sdtpa returned to normal |
[production] |
| 18:05 |
<Rob> |
ps1-a4-sdtpa temp losing its serial connection, stealing adapter to setup ms7/8 |
[production] |
| 18:04 |
<Rob> |
added ms7/ms8 to dns for wmnet and mgmt nics |
[production] |
| 16:20 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r59665' |
[production] |
| 16:19 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'Deploy r59665' |
[production] |
| 15:40 |
<Rob> |
rebooted the following per domas request: srv101 srv105 srv112 srv117 srv138 srv183 srv89 srv84 srv89 srv91 srv96 srv98 srv99 |
[production] |
| 15:02 |
<mark> |
Shutting down mint for installation of a wifi card |
[production] |
| 14:35 |
<Andrew> |
Scapping to update LiquidThreads alpha |
[production] |
| 14:34 |
<Andrew> |
Updates of LiquidThreads alpha to trunk state in progress |
[production] |
| 13:46 |
<RoanKattouw> |
Remove all LU cache files and rebuild from scratch to mitigate problems with root-owned cache files |
[production] |
| 13:41 |
<RoanKattouw> |
Purging LocalisationUpdate hashes and running the update script to ensure we didn't miss anything in the Nov 19 - Dec 1 blackout |
[production] |