2010-09-21
§
|
04:42 |
<Tim> |
on streber: removed python-pysnmp2, conflicted with python-pysnmp4 and apparently wasn't being used |
[production] |
04:28 |
<Tim> |
upgraded packages on stafford, streber. python-pysnmp-common on streber is broken |
[production] |
04:14 |
<Tim> |
upgraded packages on singer |
[production] |
04:07 |
<Tim> |
upgrading packages on spence |
[production] |
04:03 |
<Tim> |
upgrading packages on sockpuppet |
[production] |
03:53 |
<laner> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
03:53 |
<Tim> |
scheduled reboot of singer at 05:00 using atd |
[production] |
03:50 |
<tstarling> |
synchronized php-1.5/wmf-config/secure.php 'secure.wikimedia.org scheduled reboot notice' |
[production] |
03:46 |
<laner> |
synchronizing Wikimedia installation... Revision: 72920 |
[production] |
03:22 |
<Ryan_Lane> |
updated the kernel on oldusability.wikimedia.org linode server, and rebooted |
[production] |
02:55 |
<Tim> |
upgrading kernel on wikitech linode to latest 2.6.32, will reboot |
[production] |
02:45 |
<laner> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'removing debug info' |
[production] |
02:44 |
<laner> |
synchronized php-1.5/extensions/CentralNotice/SpecialBannerListLoader.php 'removing debug info' |
[production] |
02:43 |
<laner> |
synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info' |
[production] |
02:39 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialCentralNotice.php 'removing debug info' |
[production] |
02:13 |
<tfinc> |
synchronizing Wikimedia installation... Revision: 72920 |
[production] |
02:12 |
<Tim> |
rebooting locke and sanger for kernel upgrade |
[production] |
02:09 |
<Tim> |
rebooting hume for kernel upgrade |
[production] |
01:33 |
<tfinc> |
ran sync-common-all |
[production] |
01:12 |
<Tim-away> |
fixed syntax error in InitialiseSettings.php, which caused the site to go down |
[production] |
2010-09-20
§
|
22:23 |
<Ryan_Lane> |
finished kernel upgrades on tesla VMs |
[production] |
20:54 |
<Ryan_Lane> |
moved mysql data from /var/lib/mysql to /a/sqldata on prototype |
[production] |
18:33 |
<Ryan_Lane> |
Correction: previous ufw logs were for eiximenis |
[production] |
18:33 |
<Ryan_Lane> |
added ufw application configuration for etherpad. Added allow rules for OpenSSH and Etherpad in ufw config. Enabled ufw. |
[production] |
18:32 |
<Ryan_Lane> |
added ufw prerouting rules for 80->9000 for etherpad. |
[production] |
18:32 |
<Ryan_Lane> |
restarted etherpad on eiximenis |
[production] |
17:46 |
<Ryan_Lane> |
upgrading kernel on all tesla VMs |
[production] |
16:30 |
<Ryan_Lane> |
running database recovery on torrus on streber, and restarting the service |
[production] |
14:29 |
<Tim> |
rebooting mchenry for kernel upgrade |
[production] |
13:39 |
<Tim> |
rebooting ersch, erzurumi, formey and kaulen for new kernel |
[production] |
12:34 |
<Tim> |
rebooting brewster and eiximenis for new kernel |
[production] |
12:25 |
<Tim> |
updated browne /etc/motd |
[production] |
08:49 |
<Tim> |
rebooting alsted for new kernel |
[production] |
07:35 |
<Tim> |
rebooting fenari to get new kernel |
[production] |
07:03 |
<Tim> |
upgrading everything on fenari |
[production] |
2010-09-17
§
|
20:56 |
<RobH> |
db7 & db8 no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding on both |
[production] |
20:35 |
<RobH> |
locke no longer under warranty, replaced failed 73gb 10k with 146gb 10k drive on site. raid rebuilding, closing rt#114 |
[production] |
19:36 |
<RobH> |
db28 coming down for fanboard replacement (not hot swappable since its the controller) |
[production] |
19:30 |
<RobH> |
all SP back online for search1-search12 reporting error free. |
[production] |
19:25 |
<RobH> |
resetting the SP on search1-search12 to make them forget the spare powersupply they really do not have. |
[production] |
19:16 |
<RobH> |
full power reset cleared drac errors, noted for followup, system back online. |
[production] |
19:07 |
<RobH> |
srv206 has all kinds of errors, working on it, ignore nagios flaps. |
[production] |
18:49 |
<RobH> |
db34 online and ready to be setup into database deployments. |
[production] |
00:15 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Fixing placement of VariablePage extension' |
[production] |