|
2010-02-24
§
|
| 20:48 |
<Rob> |
IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console. |
[production] |
| 16:10 |
<Rob> |
had formey and alsted ip info mixed up in dns, updating dns |
[production] |
| 13:13 |
<RoanKattouw> |
Adding optin_survey table to wikis that didn't have it |
[production] |
| 01:38 |
<atglenn> |
ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen) |
[production] |
| 01:20 |
<domas> |
that was fun, let's do it again! |
[production] |
| 01:17 |
<mark> |
Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches |
[production] |
| 00:59 |
<Rob> |
added austin to the mortals user category for shell access on puppet, updating singer's account list |
[production] |
| 00:30 |
<mark> |
Fixing wikipedia.be vhost ServerAlias setting in redirects.conf |
[production] |
| 00:14 |
<domas> |
redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff |
[production] |
|
2010-02-23
§
|
| 23:41 |
<domas> |
stopped nginx on en.m until we resolve special:random issue |
[production] |
| 20:53 |
<root> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22628 - Logo of Chamorro Wikipedia' |
[production] |
| 17:23 |
<Rob> |
drac setup on new servers ersch, formey, alsted, and kaulen. still need port allocation. |
[production] |
| 16:47 |
<Rob> |
forgot to add mgmt, updating dns |
[production] |
| 16:43 |
<Rob> |
updating dns for new servers ersch, formey, alsted, & kaulen |
[production] |
| 15:37 |
<Rob> |
FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress |
[production] |
| 12:50 |
<RoanKattouw> |
Cleaned up my dblist voodoo, everything should be back to normal now |
[production] |
| 12:41 |
<RoanKattouw> |
Fixed sync-dblist, was broken |
[production] |
| 12:31 |
<RoanKattouw> |
Syncing pmtpa.dblist anyway, seems I have to |
[production] |
| 12:24 |
<RoanKattouw> |
Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this |
[production] |
|
2010-02-22
§
|
| 23:40 |
<Tim> |
restarted replication on ms2 |
[production] |
| 23:39 |
<Tim> |
ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1 |
[production] |
| 21:09 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert' |
[production] |
| 20:30 |
<root> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary' |
[production] |
| 20:09 |
<Rob> |
finished the setup of srv217 (just needed puppet fixed and files updated) |
[production] |
| 20:02 |
<root> |
ran sync-common-all |
[production] |
| 20:01 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia' |
[production] |
| 19:44 |
<root> |
ran sync-common-all |
[production] |
| 18:10 |
<Fred> |
ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic" |
[production] |
| 07:53 |
<Tim> |
rebooted srv206 via drac |
[production] |
| 07:48 |
<tstarling> |
synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down' |
[production] |
| 06:19 |
<Tim> |
increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms* |
[production] |
| 05:54 |
<Tim> |
started ms2 -> ms1 copy using nc+tarpipe |
[production] |
| 05:30 |
<Tim> |
shutting down mysql on ms2 for copy to ms1 |
[production] |
| 05:09 |
<tstarling> |
synchronized php-1.5/wmf-config/db.php 'depooling ms2 for copy to ms1' |
[production] |