|
2010-02-26
§
|
| 18:24 |
<Fred> |
bugzilla now residing on Kaulen... Updating server roles. |
[production] |
| 18:07 |
<Fred> |
switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most) |
[production] |
| 17:28 |
<midom> |
synchronized php-1.5/includes/LocalisationCache.php 'profiling' |
[production] |
| 17:26 |
<midom> |
synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook' |
[production] |
| 17:23 |
<midom> |
synchronized php-1.5/includes/StubObject.php 'tracehooks' |
[production] |
| 17:21 |
<midom> |
synchronized php-1.5/includes/StubObject.php 'tracehooks' |
[production] |
| 14:38 |
<RoanKattouw> |
logmsgbot is missing in action AGAIN, starting it as me on fenari |
[production] |
| 14:37 |
<RoanKattouw> |
Synced r63008 |
[production] |
| 12:17 |
<domas> |
*sigh*, recovered locke after OOM, added few entries to limits.conf |
[production] |
|
2010-02-25
§
|
| 23:48 |
<Tim> |
running the new fixBug20757.php on all wikis |
[production] |
| 21:48 |
<RoanKattouw> |
Removed empty line from the end of all.dblist, broke account creation ^^ |
[production] |
| 20:18 |
<RoanKattouw> |
Removed those closed wikis again, db lists back to normal |
[production] |
| 20:08 |
<RoanKattouw> |
Temp adding some closed wikis to all.dblist to be able to dump them (again) |
[production] |
| 19:01 |
<Rob> |
removed old servers will, bart, etc from DNS |
[production] |
| 18:32 |
<Rob> |
pushing more dns changes for LOM on new servers. |
[production] |
| 00:59 |
<Tim> |
made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639 |
[production] |
| 00:32 |
<Tim> |
installed ack-grep on fenari |
[production] |
|
2010-02-24
§
|
| 20:48 |
<Rob> |
IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console. |
[production] |
| 16:10 |
<Rob> |
had formey and alsted ip info mixed up in dns, updating dns |
[production] |
| 13:13 |
<RoanKattouw> |
Adding optin_survey table to wikis that didn't have it |
[production] |
| 01:38 |
<atglenn> |
ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen) |
[production] |
| 01:20 |
<domas> |
that was fun, let's do it again! |
[production] |
| 01:17 |
<mark> |
Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches |
[production] |
| 00:59 |
<Rob> |
added austin to the mortals user category for shell access on puppet, updating singer's account list |
[production] |
| 00:30 |
<mark> |
Fixing wikipedia.be vhost ServerAlias setting in redirects.conf |
[production] |
| 00:14 |
<domas> |
redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff |
[production] |
|
2010-02-23
§
|
| 23:41 |
<domas> |
stopped nginx on en.m until we resolve special:random issue |
[production] |
| 20:53 |
<root> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22628 - Logo of Chamorro Wikipedia' |
[production] |
| 17:23 |
<Rob> |
drac setup on new servers ersch, formey, alsted, and kaulen. still need port allocation. |
[production] |
| 16:47 |
<Rob> |
forgot to add mgmt, updating dns |
[production] |
| 16:43 |
<Rob> |
updating dns for new servers ersch, formey, alsted, & kaulen |
[production] |
| 15:37 |
<Rob> |
FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress |
[production] |
| 12:50 |
<RoanKattouw> |
Cleaned up my dblist voodoo, everything should be back to normal now |
[production] |
| 12:41 |
<RoanKattouw> |
Fixed sync-dblist, was broken |
[production] |
| 12:31 |
<RoanKattouw> |
Syncing pmtpa.dblist anyway, seems I have to |
[production] |
| 12:24 |
<RoanKattouw> |
Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this |
[production] |
|
2010-02-22
§
|
| 23:40 |
<Tim> |
restarted replication on ms2 |
[production] |
| 23:39 |
<Tim> |
ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1 |
[production] |
| 21:09 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert' |
[production] |
| 20:30 |
<root> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary' |
[production] |
| 20:09 |
<Rob> |
finished the setup of srv217 (just needed puppet fixed and files updated) |
[production] |
| 20:02 |
<root> |
ran sync-common-all |
[production] |
| 20:01 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia' |
[production] |
| 19:44 |
<root> |
ran sync-common-all |
[production] |
| 18:10 |
<Fred> |
ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic" |
[production] |
| 07:53 |
<Tim> |
rebooted srv206 via drac |
[production] |
| 07:48 |
<tstarling> |
synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down' |
[production] |
| 06:19 |
<Tim> |
increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms* |
[production] |
| 05:54 |
<Tim> |
started ms2 -> ms1 copy using nc+tarpipe |
[production] |
| 05:30 |
<Tim> |
shutting down mysql on ms2 for copy to ms1 |
[production] |