|
2009-01-13
§
|
| 11:38 |
<Tim> |
reducing innodb_buffer_pool_size on db19, db21, db22, db29 |
[production] |
| 09:15 |
<Tim> |
restarting mysqld on db23 again |
[production] |
| 09:09 |
<Tim> |
restarting mysqld on db18 again |
[production] |
| 07:08 |
<Tim> |
removed db23 from rotation, since I'm bringing it up soon and it will be lagged |
[production] |
| 07:02 |
<Tim> |
shutting down mysqld on db18 for further mem usage tweak |
[production] |
| 06:53 |
<Tim> |
fixed broken /etc/fstab on db23 via serial console |
[production] |
| 06:42 |
<Tim> |
restarting db23 |
[production] |
| 00:08 |
<Tim> |
repooling db18, has caught up |
[production] |
|
2009-01-12
§
|
| 21:50 |
<brion> |
testing a scap after touching MessagesWuu.php to see if that clears borked serialized btis |
[production] |
| 21:22 |
<RobH> |
erzurumi installed |
[production] |
| 17:55 |
<brion> |
temporarily stopped apache on srv78, srv118 |
[production] |
| 17:54 |
<brion> |
srv78 doesn't have upload5 mounted |
[production] |
| 17:54 |
<brion> |
srv118 doesn't have upload5 mounted |
[production] |
| 17:46 |
<RobH> |
fixed some settings for flaggedrevs in https://bugzilla.wikimedia.org/show_bug.cgi?id=14648 |
[production] |
| 17:31 |
<RobH> |
per brion commented out db18 in db.php cuz its making other crap lag too much |
[production] |
| 17:26 |
<RobH> |
updated flaggedrevs.php for https://bugzilla.wikimedia.org/show_bug.cgi?id=16365 |
[production] |
| 17:23 |
<RobH> |
updated apache config on yongle for wap => mobile forwarding oversight per https://bugzilla.wikimedia.org/show_bug.cgi?id=16692 |
[production] |
| 17:05 |
<brion> |
db18 is backlogged 191k seconds. depooling it if it's still in; complaints of hella lag |
[production] |
| 15:32 |
<Tim> |
restarted mysqld on db18 with reduced memory usage, repooled |
[production] |
| 14:12 |
<Tim> |
rebooting db18 |
[production] |
| 13:20 |
<Tim> |
depooled db18 (is down) |
[production] |
|
2009-01-10
§
|
| 16:08 |
<domas> |
rotated 300g sampled-1000.log ;-) |
[production] |
| 07:09 |
<river> |
applied current OS patches to ms2 and rebooted |
[production] |
| 01:21 |
<Tim> |
restarted apache on srv95,srv114,srv37,srv49 |
[production] |
| 01:19 |
<Tim> |
cleaned up disk space on db1. Still looks suspiciously like the master... |
[production] |
| 00:33 |
<brion> |
redirecting old bylaws.pdf to wiki page bylaws on wikimediafoundation.org (foundation.conf update) |
[production] |
| 00:13 |
<brion> |
reconfigured exim on wikitech to hopefully actually send mail out. whether it reaches anything, we'll see |
[production] |
| 00:12 |
<tomaszf> |
turned off fundraising banners |
[production] |
| 00:08 |
<brion> |
installed a mail server on wikitech server, hopefully |
[production] |
|
2009-01-08
§
|
| 22:08 |
<brion> |
putting db12 back in service, caught up |
[production] |
| 21:42 |
<RobH> |
changed the ip address for the management interfaces on sq31-sq50 |
[production] |
| 21:30 |
<RobH> |
updated dns with the squids and srv mangement info for pmtpa |
[production] |
| 21:16 |
<brion> |
taking load off db12 while it updates |
[production] |
| 21:15 |
<brion> |
killing stuck query threads on db12 (lagged 13k seconds) |
[production] |
| 20:23 |
<RobH> |
updated dns removing a large number of decommissioned servers from records. |
[production] |
| 20:08 |
<RobH> |
pushed updates to dns for mangement ip allocations, changed mangement ips of search8-search12 |
[production] |
| 19:43 |
<RobH> |
changed the mangement ip addresses of db5-db10 to fit into current ip scheme |
[production] |
| 18:20 |
<RobH> |
updated dns for the management name resolution of db11-db30 |
[production] |
| 18:11 |
<RobH> |
ms5 has lom access enabled and is ready for testing. (Only one ethernet connection in lieu of the typical 3 on the thumper/thors) |
[production] |
| 15:50 |
<RobH> |
srv118 reinstalled |
[production] |
| 15:46 |
<RobH> |
srv136 is borked. Even after reinstall, it will run for a few minutes, then lock hard. Going to RMA it. |
[production] |
| 15:38 |
<RobH> |
reinstalled srv136 and srv118 cuz they were pissing me off (a valid reinstallation reason if there ever was one.) |
[production] |