|
2009-12-02
§
|
| 23:28 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'reenabling db18 and db25, also, attempting to overwrite stale db.php copies' |
[production] |
| 23:25 |
<Fred> |
massaged mc.php to retrieve working spare, and remove broken memcached nodes. all is now good in the land of memcache |
[production] |
| 22:13 |
<mark> |
Recovered torrus from deadlock |
[production] |
| 21:00 |
<Fred> |
rebooted srv194 (hung) |
[production] |
| 20:48 |
<Rob> |
removed bayle and khaldun from dsh, both are in rack running wipe with network pulled |
[production] |
| 20:38 |
<Fred> |
bart removed from nagios (well that sounds funny) |
[production] |
| 20:36 |
<Rob> |
khaldun is down forever! decomissioned and running wipe in rack with the network pulled |
[production] |
| 20:35 |
<Rob> |
isidore rebooted by accident due to power cable issues |
[production] |
| 20:21 |
<Rob> |
srv136 crashed with temp warnings, going to decommission it, rebooting to wipe and remove network |
[production] |
| 20:15 |
<Rob> |
bart decommissioned, unracked, wipe running on testbench with usbcdrom |
[production] |
| 19:49 |
<Rob> |
decommissioned, unracked srv66, srv51, srv81, srv118 (previously removed from pybal) |
[production] |
| 19:39 |
<Rob> |
decommissioned srv130, unracked |
[production] |
| 19:20 |
<Rob> |
srv122 decommissioned, wiped, unracked |
[production] |
| 18:19 |
<Rob> |
ms7/ms8 racked in sdtpa a2, network wired, dns setup, racktables updated, & LOM online |
[production] |
| 18:18 |
<Rob> |
serial connection to ps1-a4-sdtpa returned to normal |
[production] |
| 18:05 |
<Rob> |
ps1-a4-sdtpa temp losing its serial connection, stealing adapter to setup ms7/8 |
[production] |
| 18:04 |
<Rob> |
added ms7/ms8 to dns for wmnet and mgmt nics |
[production] |
| 16:20 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r59665' |
[production] |
| 16:19 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'Deploy r59665' |
[production] |
| 15:40 |
<Rob> |
rebooted the following per domas request: srv101 srv105 srv112 srv117 srv138 srv183 srv89 srv84 srv89 srv91 srv96 srv98 srv99 |
[production] |
| 15:02 |
<mark> |
Shutting down mint for installation of a wifi card |
[production] |
| 14:35 |
<Andrew> |
Scapping to update LiquidThreads alpha |
[production] |
| 14:34 |
<Andrew> |
Updates of LiquidThreads alpha to trunk state in progress |
[production] |
| 13:46 |
<RoanKattouw> |
Remove all LU cache files and rebuild from scratch to mitigate problems with root-owned cache files |
[production] |
| 13:41 |
<RoanKattouw> |
Purging LocalisationUpdate hashes and running the update script to ensure we didn't miss anything in the Nov 19 - Dec 1 blackout |
[production] |
| 05:28 |
<Tim> |
edited authz on mediawiki svn to deny write access to root |
[production] |
| 01:17 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'new banner for cn' |
[production] |
| 01:04 |
<Tim> |
restarted apache on srv211 and srv127 |
[production] |
|
2009-11-30
§
|
| 22:54 |
<Fred> |
rebooting pascal since it once again went down. |
[production] |
| 21:21 |
<RoanKattouw> |
Fred moved l10nupdate cronjob on hume to /etc/cron.d/l10nupdate , now runs as brion instead of root |
[production] |
| 21:21 |
<RoanKattouw> |
Unlocked /h/w/l10n/trunk/extensions , had been locked since Nov 19th blocking L10nUpdates to extensions |
[production] |
| 20:06 |
<rainman-sr> |
set search limit to 51 per Roan |
[production] |
| 20:06 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.php 'Raise style version for r59598, r59599' |
[production] |
| 20:05 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59599 for real this time' |
[production] |
| 20:04 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59599' |
[production] |
| 20:00 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59598' |
[production] |
| 19:55 |
<rainman-sr> |
converting searchidx1 back to indexer, things seem to be back to normal now |
[production] |
| 19:46 |
<atglenn> |
enabled ExternalPages on test.wikipedia (fundraising extension) after adding it by hand to extension-list and ExtensionMessages.php |
[production] |
| 19:30 |
<rainman-sr> |
doing a delayed restart of all searchers |
[production] |
| 19:10 |
<mark> |
Setup puppet to enforce desired JVM alternatives on the search servers |
[production] |
| 17:00 |
<rainman-sr> |
all search servers (except search11) should be switched back from 64 bit java (java-6-openjdk) to 32 bit java (ia32-java-6-sun) |
[production] |
| 16:33 |
<rainman-sr> |
temporarely stop indexing and put searchidx1 in en.wp rotation, will back out when peak times are over |
[production] |
| 15:28 |
<mark> |
Puppet removed syslog-ng on nfs1/2 as well. Restored. |
[production] |
| 14:49 |
<mark> |
Replacing stock syslogd with rsyslog globally across the cluster (with puppet). rsyslog is the default in Ubuntu 9.10 onwards |
[production] |
| 10:35 |
<mark> |
Using streber for karmic install testing |
[production] |