|
2009-11-03
§
|
| 23:47 |
<tstarling> |
synchronized wmf-deployment/cache/trusted-xff.cdb |
[production] |
| 23:23 |
<Tim> |
deploying trusted XFF update from r58506 |
[production] |
| 23:12 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'removing db28' |
[production] |
| 23:10 |
<Tim> |
restarted udp2log on locke, sampled-1000.log was empty |
[production] |
| 19:32 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r58494' |
[production] |
| 16:12 |
<mark> |
Recovered torrus from its deadlocked state |
[production] |
| 15:24 |
<mark> |
Setup APT repository and preseeding repository on brewster, using puppet |
[production] |
| 15:21 |
<mark> |
Moved apt.wikimedia.org DNS CNAME from khaldun to brewster |
[production] |
| 15:17 |
<mark> |
Shutdown BGP session to AS 16150; no connectivity to wikitech.wikimedia.org |
[production] |
| 15:14 |
<mark> |
Added --timeout=30 to the wget invocations in the post-commit script on svn.wikimedia.org |
[production] |
| 14:17 |
<Andrew> |
Scapping |
[production] |
| 14:09 |
<fvassard> |
synchronized php-1.5/wmf-config/mc.php 'swapped dead memcache from srv217 to srv156' |
[production] |
| 14:06 |
<Andrew> |
Will scap to update LiquidThreads in a few minutes |
[production] |
|
2009-11-01
§
|
| 19:41 |
<rainman-sr> |
started search indexer on searchidx1 |
[production] |
| 19:22 |
<mark> |
Renamed db6 to locke in DNS and racktables |
[production] |
| 19:04 |
<mark> |
Powercycled frozen server searchidx1 |
[production] |
| 18:59 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changed proxy host for file-by-url testing on test.wikipedia.org from khaldun to brewster' |
[production] |
| 18:59 |
<mark> |
Stopped oprofile on srv218 and removed all samples; its disk was full |
[production] |
| 18:54 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
| 18:53 |
<mark> |
Setup Squid proxy on brewster, and setup selective proxying for security.ubuntu.com through it on all hosts, using puppet |
[production] |
| 17:44 |
<mark> |
Set up tftp server on brewster (using puppet) |
[production] |
| 16:54 |
<mark> |
Added ubuntu.wikimedia.org DNS entry |
[production] |
| 16:45 |
<mark> |
Set up Ubuntu mirror on brewster using puppet |
[production] |
| 16:45 |
<mark> |
Fixed Domas's IPC cleanup cron job to not run if there are no stale ipc semaphores |
[production] |
| 12:48 |
<river> |
allocated vlan 301 to the toolserver, 10.23.1.0/24 (we're out of public IPs) |
[production] |
|
2009-10-30
§
|
| 23:11 |
<RoanKattouw> |
Rebuilt l10ncache on srv194, seems to have fixed it. Presumably caused by an incomplete run of a sync script caused by permissions errors (see private-l) |
[production] |
| 23:00 |
<Fred> |
depooled srv194 while testing / fixing |
[production] |
| 22:08 |
<Fred> |
actually stopped apache on srv194 since the restart did not help. Investigating.... |
[production] |
| 22:07 |
<Fred> |
restarted apache on srv194 since it was throwing a lot of ""is not a valid magic thingie for xxxx" |
[production] |
| 20:05 |
<Fred> |
scaped as root to resolve cache problems on some boxes. |
[production] |
| 20:02 |
<fvassard> |
ran sync-common-all |
[production] |
| 19:50 |
<RoanKattouw> |
Scapping in the hopes of resolving the MagicWordsArray exception; looks like it could be a problem with localisation cache |
[production] |
| 19:37 |
<catrope> |
synchronized php-1.5/includes/MagicWord.php 'Live hack for debugging: make MagicWordArray::parseMatch parameter not found error more verbose' |
[production] |
| 18:36 |
<catrope> |
synchronized php-1.5/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Deploy r58359 (LocalisationUpdate fix for languages with a '-' in their name)' |
[production] |
| 18:33 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Per bug 19865, enable the enhanced toolbar by default on enwikinews and disable the OptIn extension' |
[production] |
| 16:39 |
<RoanKattouw> |
sync-common-all threw lots of errors like srv175: rsync: failed to set times on "/usr/local/apache/common-local/wmf-deployment/includes/.svn": Operation not permitted (1) |
[production] |
| 16:38 |
<catrope> |
ran sync-common-all |
[production] |
| 16:36 |
<RoanKattouw> |
Scapping so srv194 will receive updates |
[production] |
| 15:34 |
<apergos> |
cleared out some files in /tmp and ran apt-get clean on srv194 to get back about 1gb space on / |
[production] |
| 14:08 |
<RoanKattouw> |
@wikimediatech feed on identi.ca has been broken for the past 3 days |
[production] |
| 14:01 |
<RoanKattouw> |
srv194 has full disk, causing syncs to fail; throws PHP exception "MagicWordArray::parseMatch: parameter not found" |
[production] |
| 11:47 |
<rainman-sr> |
searchidx1 ssh dead, shows 100% I/O, disk dead? |
[production] |