|
2010-03-03
§
|
| 06:06 |
<apergos> |
replication off again tonight, zfs nicely borked on ms8, zfs destroy snapshot X gives message "cannot destroy X, dataset already exists" . time for sun contract and call them |
[production] |
| 05:53 |
<apergos> |
gmetrics turned off again on ms8, running zpool scrub export |
[production] |
| 05:40 |
<apergos> |
reset SYs on ms8 worked (slowly). gmetrics reenabled. (and I thought morebots was just being a bad bot) |
[production] |
| 05:39 |
<apergos> |
( :-P ) starting catchup for the last day's lost replication ms7 -> ms8 |
[production] |
| 05:22 |
<apergos> |
reboot failed, bootadm hung probably waiting on zfs. stop SYS claims to work but subsequent show SYS shows power state on. |
[production] |
| 05:14 |
<apergos> |
rebootiing ms8 to clear zfs deadlock |
[production] |
|
2010-03-02
§
|
| 23:32 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'r63191' |
[production] |
| 23:31 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.php 'r63191' |
[production] |
| 23:31 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Tweak OptIn settings' |
[production] |
| 22:36 |
<RoanKattouw> |
Rebuilding localisation cache |
[production] |
| 20:15 |
<Tim> |
on hume: started recompressTracked.php for all wikis, to fix any newly-tracked pointers from fixBug20757.php and to complete enwiki |
[production] |
| 03:48 |
<atglenn> |
leaving replication and gmetrics turned off on ms8 for thenight, will go back to looking at scat and mdb output tomorrow. |
[production] |
| 02:27 |
<atglenn> |
turning off gmetric on ms8, the dfs are piling up behind a deadlocked zfs list |
[production] |
| 02:13 |
<atglenn> |
turn of ms7 replication to ms8 temporarily to work out a zfs deadlock issue on ms8 |
[production] |
|
2010-02-26
§
|
| 23:40 |
<Tim> |
disabled the temporary squid purge script, I'm assuming it's done its job now |
[production] |
| 19:43 |
<catrope> |
synchronized php-1.5/includes/api/ApiBase.php 'r63014, r63015' |
[production] |
| 18:24 |
<Fred> |
bugzilla now residing on Kaulen... Updating server roles. |
[production] |
| 18:07 |
<Fred> |
switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most) |
[production] |
| 17:28 |
<midom> |
synchronized php-1.5/includes/LocalisationCache.php 'profiling' |
[production] |
| 17:26 |
<midom> |
synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook' |
[production] |
| 17:23 |
<midom> |
synchronized php-1.5/includes/StubObject.php 'tracehooks' |
[production] |
| 17:21 |
<midom> |
synchronized php-1.5/includes/StubObject.php 'tracehooks' |
[production] |
| 14:38 |
<RoanKattouw> |
logmsgbot is missing in action AGAIN, starting it as me on fenari |
[production] |
| 14:37 |
<RoanKattouw> |
Synced r63008 |
[production] |
| 12:17 |
<domas> |
*sigh*, recovered locke after OOM, added few entries to limits.conf |
[production] |
|
2010-02-25
§
|
| 23:48 |
<Tim> |
running the new fixBug20757.php on all wikis |
[production] |
| 21:48 |
<RoanKattouw> |
Removed empty line from the end of all.dblist, broke account creation ^^ |
[production] |
| 20:18 |
<RoanKattouw> |
Removed those closed wikis again, db lists back to normal |
[production] |
| 20:08 |
<RoanKattouw> |
Temp adding some closed wikis to all.dblist to be able to dump them (again) |
[production] |
| 19:01 |
<Rob> |
removed old servers will, bart, etc from DNS |
[production] |
| 18:32 |
<Rob> |
pushing more dns changes for LOM on new servers. |
[production] |
| 00:59 |
<Tim> |
made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639 |
[production] |
| 00:32 |
<Tim> |
installed ack-grep on fenari |
[production] |
|
2010-02-24
§
|
| 20:48 |
<Rob> |
IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console. |
[production] |
| 16:10 |
<Rob> |
had formey and alsted ip info mixed up in dns, updating dns |
[production] |
| 13:13 |
<RoanKattouw> |
Adding optin_survey table to wikis that didn't have it |
[production] |
| 01:38 |
<atglenn> |
ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen) |
[production] |
| 01:20 |
<domas> |
that was fun, let's do it again! |
[production] |
| 01:17 |
<mark> |
Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches |
[production] |
| 00:59 |
<Rob> |
added austin to the mortals user category for shell access on puppet, updating singer's account list |
[production] |
| 00:30 |
<mark> |
Fixing wikipedia.be vhost ServerAlias setting in redirects.conf |
[production] |
| 00:14 |
<domas> |
redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff |
[production] |