2010-10-03
§
|
17:30 |
<mark> |
Stopped IPVS backup sync daemon on amslvs4 |
[production] |
17:27 |
<mark> |
Rebooting amslvs2 |
[production] |
17:19 |
<mark> |
Stopped PyBal on amslvs2 |
[production] |
17:14 |
<mark> |
Drive failed in thistle |
[production] |
17:09 |
<mark> |
Started temporary LVS state syncing between amslvs1->amslvs3 and amslvs2->amslvs4, preparing for reboot of amslvs1-2 |
[production] |
16:51 |
<mark> |
Rebooted (backup) LVS servers amslvs3 and amslvs4 |
[production] |
16:36 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'adding db28 to s5' |
[production] |
16:20 |
<mark> |
Selectively updated pybal/bgp.py on lvs2-4 and restarted PyBal |
[production] |
16:13 |
<mark> |
Upgraded PyBal to r0.1+74215 on amslvs1-4 |
[production] |
14:27 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
14:12 |
<domas> |
db22 was first wikimedia slave initialized using xtrabackup, hehe |
[production] |
14:08 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'adding db22 as commons 5.1 slave' |
[production] |
14:06 |
<mark> |
Restored nameservers order for mobile serverswq |
[production] |
13:36 |
<mark> |
lvs3 service ips were unreachable from outside the subnet for some reason. Restarting PyBal (and therefore the BGP session to the router) fixed it |
[production] |
13:22 |
<mark> |
Started pdns recursor on dobson again |
[production] |
13:19 |
<mark> |
Once again stopped pnds-recursor on dobson |
[production] |
13:18 |
<mark> |
Restarted Apache on the mobile servers one by one, to put the new resolv.conf in effect |
[production] |
13:10 |
<mark> |
Swapped nameservers for the mobile servers, so they don't crash the mobile site while I reinstall the primary recursor |
[production] |
13:02 |
<catrope> |
synchronized php-1.5/wmf-config/checkers.php 'Log no-UA requests for exempt IPs to /h/w/l/nouaexempt.log' |
[production] |
12:45 |
<mark> |
Mobile site broke with missing primary DNS recursor, restarted it |
[production] |
12:44 |
<mark> |
Stopped DNS recursor on dobson again |
[production] |
12:20 |
<mark> |
Installed DNS recursors on all LVS servers to reduce SPOFfiness |
[production] |
11:46 |
<domas> |
doing some DB maintenance |
[production] |
11:44 |
<mark> |
Installed nscd on lvs4 and restarted PyBal |
[production] |
11:30 |
<mark> |
Restarted pybal on lvs4 |
[production] |
11:25 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
11:21 |
<mark> |
Shutdown primary pmtpa DNS recursor (dobson) |
[production] |
11:20 |
<catrope> |
synchronized php-1.5/skins/common/wikibits.js 'r74198' |
[production] |
2010-09-29
§
|
22:59 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning CN back on as default' |
[production] |
22:55 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fix for db lookup' |
[production] |
22:55 |
<Ryan_Lane> |
rebooting ersch and alsted |
[production] |
22:54 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Turning test wikipedia to non infrastructure mode' |
[production] |
22:41 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling cn' |
[production] |
22:39 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes from 73993' |
[production] |
22:38 |
<tfinc> |
synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fixes from 73993' |
[production] |
22:36 |
<tfinc> |
synchronized php-1.5/includes/Skin.php 'Picking up fixes from 73993' |
[production] |