|
2009-02-02
§
|
| 16:18 |
<brion-sick> |
large spike in text backend service times |
[production] |
| 16:15 |
<brion-sick> |
secure.wikimedia.org is returning 503 Service Temporarily Unavailable |
[production] |
| 08:11 |
<Tim> |
removing ancient static HTML dump from srv31 |
[production] |
| 08:05 |
<Tim> |
removed cluster13 and cluster14 from db.php, will watch exception.log for attempted connections |
[production] |
| 08:02 |
<Tim> |
removed srv130 from LVS and the apaches node group, not accessible by ssh but still serving pages |
[production] |
| 07:56 |
<Tim> |
find /home/wikipedia/logs -size 0 -delete |
[production] |
| 07:43 |
<Tim> |
re-added db22 to s1 rotation, no explanation for its removal in server admin log |
[production] |
| 06:39 |
<Tim> |
dropped the otrs_test database |
[production] |
| 06:38 |
<Tim> |
moved the OTRS database from otrs_real back to otrs. Updated exim4 config on mchenry |
[production] |
| 04:23 |
<Tim> |
db10's relay log was corrupted, did a flush slave/change master |
[production] |
| 01:11 |
<Tim> |
started mysqld on db23, doing recovery |
[production] |
| 00:59 |
<Tim> |
rebooted db23 |
[production] |
| 00:56 |
<Tim> |
db23 down, depooled |
[production] |
| 00:05 |
<Tim> |
adjusted innodb configuration on db10, restarted, starting replication |
[production] |
|
2009-01-31
§
|
| 18:17 |
<mark> |
Following reports of OTRS rapidly deleting old tickets/emails every ~ 10 minutes, I disabled (set to invalid) all GenericAgent jobs pending investigation |
[production] |
| 15:43 |
<mark> |
Set <tt>local_from_check = false</tt> in exim.conf on williams, to prevent Sender headers from being added (annoying for Outlook users) |
[production] |
| 07:11 |
<Tim> |
converting OTRS database to proper UTF-8 (instead of UTF-8 in latin1 fields) using ~/fix-schema.php |
[production] |
| 01:30 |
<brion> |
updating eswikibooks logo [[bugzilla:17078]] |
[production] |
| 00:55 |
<brion> |
setting mswikibooks logo [[bugzilla:17263]] |
[production] |
| 00:53 |
<brion> |
copied wikimedia favicon to blog.wikimedia.org [[bugzilla:17171]] |
[production] |
| 00:51 |
<domas> |
lomaria needs reinstall, db24 and db30 are live in s2 duty |
[production] |
|
2009-01-30
§
|
| 17:54 |
<domas> |
*giggle*, booted up lomaria with SMP kernel |
[production] |
| 17:43 |
<domas> |
lomaria kernel detects just one CPU (out of four) |
[production] |
| 17:26 |
<domas> |
converted lomaria into dewiki-only server |
[production] |
| 12:52 |
<mark> |
Installed ganglia on williams |
[production] |
| 11:50 |
<mark> |
Letting OTRS mail through to williams on mchenry |
[production] |
| 10:50 |
<Tim> |
running upgrade of OTRS DB |
[production] |
| 10:44 |
<mark> |
Removed all OTRS test copies in the queue of williams |
[production] |
| 10:42 |
<mark> |
Deferring all OTRS mail on the queue of mchenry |
[production] |
| 10:30 |
<mark> |
Put in a quick hack to forward misrouted OTRS mails from williams to bart |
[production] |
| 08:52 |
<Tim> |
sent upgrade warning email to all OTRS agents |
[production] |
| 02:36 |
<Tim> |
set up SSL on williams and switched ticket.wikimedia.org DNS to point to there |
[production] |
| 02:21 |
<brion> |
set up new SSL cert for ticket.wikimedia.org; tim's poking at installing it |
[production] |
| 02:19 |
<brion> |
updated password on tridge *cough* |
[production] |
| 01:43 |
<brion> |
syncing update to Drafts with IE 7 fix (r46571 and style ver update) |
[production] |
| 00:16 |
<brion> |
live-merging r46570 -- fixes to DB access in revisiondelete |
[production] |
|
2009-01-29
§
|
| 22:55 |
<mark> |
Did s/knams/esams/ on the selective AAAA answer config of ns0/ns1/ns2.wikimedia.org |
[production] |
| 22:48 |
<mark> |
While messages are held in the queue on williams, use "mailq" to view the queue, and "exim -M <messageid>" to let an individual message through for testing |
[production] |
| 22:44 |
<mark> |
SpamAssassin training from the OTRS Junk queue not yet setup |
[production] |
| 22:43 |
<mark> |
'''Note:''' Exim on williams queries for mail addresses from the ''live'' OTRS database, not the test database |
[production] |
| 22:42 |
<mark> |
Completed OTRS mail setup on williams. wikitech documentation updated in [[OTRS]] and [[Mail]]. OTRS mail is still copied to williams, and then held on the queue. |
[production] |
| 22:00 |
<mark> |
Added db10 as secondary DB to query for Exim on mchenry |
[production] |
| 21:59 |
<mark> |
Granted SELECT privileges on otrs.system_address to exim@williams on db9/db10 |
[production] |
| 21:58 |
<brion> |
enabling revision & log suppression for oversighters |
[production] |
| 21:12 |
<brion> |
live-merging r46429 change to Special:Contributions -- stub marking fix |
[production] |