|
2010-12-13
§
|
| 23:47 |
<apergos> |
dataset1 is up, ssh accessible, with the data partition mounted read only, rsync on tridge in a screen session dong rsync as root of /data/xmldatadumps from dataset1 to /data/XMLdumps on tridge. |
[production] |
| 21:37 |
<Ryan_Lane> |
restarting pdns on ns1 |
[production] |
| 21:36 |
<Ryan_Lane> |
changing IP address for payments.wikimedia.org in DNS |
[production] |
| 20:54 |
<Ryan_Lane> |
restarting pybal on lvs4 |
[production] |
| 20:33 |
<Ryan_Lane> |
intalling wikimedia-lvs-realserver on payments3 and payments4 |
[production] |
| 20:31 |
<Ryan_Lane> |
adding payments configuration to pybal on lvs4 |
[production] |
| 20:11 |
<Ryan_Lane> |
moving payments4 into public subnet and restarting network services |
[production] |
| 20:10 |
<Ryan_Lane> |
moving payments4 into squid vlan |
[production] |
| 20:09 |
<mark> |
Failed over text.esams and bits.esams traffic to amslvs3 using an increased BGP metric for amslvs1 routes on csw1-esams |
[production] |
| 20:03 |
<mark> |
Configured amslvs3 with option conn_tab_size=20 for module ip_vs |
[production] |
| 20:01 |
<mark> |
Rebooting amslvs3 again to test |
[production] |
| 19:53 |
<mark> |
Rebooting amslvs3 |
[production] |
| 19:51 |
<mark> |
Running apt-get upgrade on amslvs3 |
[production] |
| 19:50 |
<mark> |
Added "kernel-ppa" PPA to amslvs3, installing linux 2.6.36 |
[production] |
| 18:34 |
<Ryan_Lane> |
renaming payments3 and payments4 in puppet |
[production] |
| 18:27 |
<Ryan_Lane> |
restarting network and services on payments3 |
[production] |
| 18:00 |
<Ryan_Lane> |
moving payments3 and payments4 to a public subnet |
[production] |
| 17:59 |
<Ryan_Lane> |
disabling payments3 and payments4 from haproxy config on loudon |
[production] |
| 12:42 |
<mark> |
Revived down servers |
[production] |
|
2010-12-12
§
|
| 22:01 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'removed dead settings' |
[production] |
| 18:47 |
<apergos> |
bayes unreachable by mgmt or ping, needs power cycled somehow |
[production] |
| 18:47 |
<apergos> |
bringing snapshot1 back up (not intended, but it was sitting there in the bios, I was actually looking for the bayes mgmt console as its current ip for that is nonresponsive... seems like snapshot1 got the old one though. |
[production] |
| 15:55 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Belated style version bump for last sync' |
[production] |
| 15:38 |
<catrope> |
synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js 'r78249' |
[production] |
| 14:26 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'Reorganized autopromote settings to be more sane' |
[production] |
| 05:52 |
<Ryan_Lane> |
installing php5-uuid on nova-controller.tesla |
[production] |
|
2010-12-10
§
|
| 23:29 |
<Ryan_Lane> |
adding generic::geoip to owa* |
[production] |
| 20:24 |
<RobH> |
updating dns with emery entry |
[production] |
| 20:20 |
<RobH> |
snapshot1 powering back up, emery being pulled for temp storage use |
[production] |
| 18:54 |
<RobH> |
snapshot1 going offline to add capacity |
[production] |
| 13:40 |
<mark> |
Deployed new version exim 4.69-2ubuntu0.1wm2 with a str_format vuln fixed from exim 4.70 (lily, mchenry, sanger, grosley, williams) |
[production] |
| 13:40 |
<mark> |
Ran apt-get upgrade on sockpuppet |
[production] |
| 13:40 |
<mark> |
Ran apt-get upgrade on lily |
[production] |
| 05:57 |
<Ryan_Lane> |
cleaning /tmp on pdf1-3 |
[production] |
| 05:45 |
<Ryan_Lane> |
rebooting pdf3 |
[production] |
| 05:38 |
<Ryan_Lane> |
running mw-serve-ctl in cleanup mode on pdf3 |
[production] |
| 02:10 |
<Ryan_Lane> |
installing ldap scripts into /usr/local/sbin on nova-controller.tesla |
[production] |
| 02:03 |
<Ryan_Lane> |
installing ldap-utils on nova-controller.tesla |
[production] |
| 01:06 |
<Ryan_Lane> |
installing php5-curl on nova-controller.tesla |
[production] |
| 00:27 |
<Ryan_Lane> |
installing php5-memcache on nova-controller.tesla |
[production] |
| 00:27 |
<Ryan_Lane> |
installing php-apc and memcached on nova-controller.tesla |
[production] |
| 00:18 |
<mark> |
Deployed slightly modified exim4-daemon-heavy 4.69-2ubuntu0.1wm1 package on mchenry, sanger, lily, grosley |
[production] |