|
2010-12-15
§
|
| 22:26 |
<Ryan_Lane> |
restarted slave on db10 |
[production] |
| 22:25 |
<Ryan_Lane> |
killed mysqlcheck process on db10, as it was eating all available cpu |
[production] |
| 22:09 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26222 - Set $wgLogo to local logo on rowiki' |
[production] |
| 21:55 |
<Ryan_Lane> |
restarted slave on db10 |
[production] |
| 21:16 |
<Ryan_Lane> |
restarting mysqld on db10 |
[production] |
| 21:16 |
<Ryan_Lane> |
enabling long query log on db10 |
[production] |
| 20:43 |
<Ryan_Lane> |
stoped and started slave on db10 |
[production] |
| 19:39 |
<RobH> |
pushing dns change to switch dumps to dataset1 from singer (download and all its various incarnations are all pointing at dumps |
[production] |
| 19:17 |
<RobH> |
amaranth accidental reboot due to power removal |
[production] |
| 18:50 |
<RobH> |
msw-b1-sdtpa power cycled due to power cable being moved within the rack |
[production] |
| 18:41 |
<RobH> |
storage3 powering down to have its power distro fixed in rack |
[production] |
| 18:30 |
<RobH> |
bringing down db41/db42, the power in rack B1-sdtpa is not balanced properly. |
[production] |
| 17:52 |
<demon> |
synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'r78443' |
[production] |
| 17:45 |
<demon> |
synchronized php-1.5/extensions/CodeReview/backend/CodeRevision.php 'Debugging pywikipedia failure' |
[production] |
| 09:48 |
<domas> |
upgraded stuff on secure to fix ssl warnings |
[production] |
| 02:37 |
<phuzion> |
Wrote configs for powerdns with ldap backend and restarted powerdns. |
[production] |
| 02:34 |
<Ryan_Lane> |
restarting opendj on nova-controller.tesla |
[production] |
| 02:34 |
<Ryan_Lane> |
adding the dnsdomain2 schema to opendj on nova-controller.tesla |
[production] |
| 02:13 |
<phuzion> |
Instaled pdns-server and pdns-backend-ldap on nova-controller.tesla |
[production] |
| 01:15 |
<awjrichards> |
synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.js 'updating to r78421 so fundraising stats page defaults to 2010 fundraiser' |
[production] |
| 01:15 |
<awjrichards> |
synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 'updating to r78421 so fundraising stats page defaults to 2010 fundraiser' |
[production] |
| 00:09 |
<awjrichards> |
synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.js 'Revering to r64689 due to broken JS' |
[production] |
|
2010-12-13
§
|
| 23:47 |
<apergos> |
dataset1 is up, ssh accessible, with the data partition mounted read only, rsync on tridge in a screen session dong rsync as root of /data/xmldatadumps from dataset1 to /data/XMLdumps on tridge. |
[production] |
| 21:37 |
<Ryan_Lane> |
restarting pdns on ns1 |
[production] |
| 21:36 |
<Ryan_Lane> |
changing IP address for payments.wikimedia.org in DNS |
[production] |
| 20:54 |
<Ryan_Lane> |
restarting pybal on lvs4 |
[production] |
| 20:33 |
<Ryan_Lane> |
intalling wikimedia-lvs-realserver on payments3 and payments4 |
[production] |
| 20:31 |
<Ryan_Lane> |
adding payments configuration to pybal on lvs4 |
[production] |
| 20:11 |
<Ryan_Lane> |
moving payments4 into public subnet and restarting network services |
[production] |
| 20:10 |
<Ryan_Lane> |
moving payments4 into squid vlan |
[production] |
| 20:09 |
<mark> |
Failed over text.esams and bits.esams traffic to amslvs3 using an increased BGP metric for amslvs1 routes on csw1-esams |
[production] |
| 20:03 |
<mark> |
Configured amslvs3 with option conn_tab_size=20 for module ip_vs |
[production] |
| 20:01 |
<mark> |
Rebooting amslvs3 again to test |
[production] |
| 19:53 |
<mark> |
Rebooting amslvs3 |
[production] |
| 19:51 |
<mark> |
Running apt-get upgrade on amslvs3 |
[production] |
| 19:50 |
<mark> |
Added "kernel-ppa" PPA to amslvs3, installing linux 2.6.36 |
[production] |
| 18:34 |
<Ryan_Lane> |
renaming payments3 and payments4 in puppet |
[production] |
| 18:27 |
<Ryan_Lane> |
restarting network and services on payments3 |
[production] |
| 18:00 |
<Ryan_Lane> |
moving payments3 and payments4 to a public subnet |
[production] |
| 17:59 |
<Ryan_Lane> |
disabling payments3 and payments4 from haproxy config on loudon |
[production] |
| 12:42 |
<mark> |
Revived down servers |
[production] |