|
2010-10-05
§
|
| 15:25 |
<domas> |
fixed space on srv178, conf changes made apparmor audit log blow up |
[production] |
| 15:02 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'db31 mnt' |
[production] |
| 15:00 |
<mark> |
Restarted Apache on singer |
[production] |
| 14:55 |
<mark> |
Migrated the Apple Dictionary bridge from yongle to ekrem, and moved the search.wikimedia.org CNAME accordingly |
[production] |
| 14:51 |
<mark> |
ekrem reboot test |
[production] |
| 13:37 |
<mark> |
Had puppet install php5-curl on ekrem |
[production] |
| 13:29 |
<mark> |
Migrated the old mobile (WAP) site from yongle to ekrem and updated the Squid config |
[production] |
| 11:56 |
<mark> |
Installed Lucid on new server ekrem |
[production] |
| 11:23 |
<mark> |
Reused public ip address of isidore for new server ekrem |
[production] |
| 10:53 |
<mark> |
Reinstalled knsq1 as bits varnish server and had it join the esams bits cluster |
[production] |
| 09:34 |
<mark> |
Removed AS43821 outbound double prepend on BGP session with AS13030 |
[production] |
| 09:30 |
<mark> |
Moving European bits traffic back to the European cluster |
[production] |
|
2010-10-04
§
|
| 19:52 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 19:52 |
<catrope> |
synchronized php-1.5/extensions/ArticleAssessmentPilot/js/ArticleAssessment.combined.min.js 'r74268' |
[production] |
| 19:51 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php 'r74268' |
[production] |
| 11:57 |
<mark> |
Stopped PyBal on amslvs4 as a test |
[production] |
| 11:47 |
<mark> |
synchronized php-1.5/wmf-config/mc.php 'Replace srv87' |
[production] |
| 10:53 |
<mark> |
Setup BGP state monitoring for csw5-pmtpa and csw1-sdtpa as well |
[production] |
| 10:48 |
<mark> |
Setup BGP state monitoring for csw1-esams and csw2-esams |
[production] |
| 10:36 |
<mark> |
Restarted Apache on srv223 |
[production] |
| 10:24 |
<mark> |
Fixed Nagios by removing db34 from the listed hosts section in conf.php |
[production] |
| 09:47 |
<mark> |
Downpreffed AS13030 transit to local-pref 90 on br1-knams |
[production] |
|
2010-10-03
§
|
| 19:25 |
<catrope> |
synchronized php-1.5/wmf-config/checkers.php |
[production] |
| 18:59 |
<catrope> |
synchronized php-1.5/wmf-config/checkers.php 'Block search abuse reported by Robert' |
[production] |
| 17:43 |
<mark> |
Restarted apache on srv219-221, 224 |
[production] |
| 17:38 |
<mark> |
Stopped IPVS backup sync daemon on amslvs3 |
[production] |
| 17:37 |
<mark> |
Rebooting amslvs1 |
[production] |
| 17:37 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 17:32 |
<mark> |
Stopped PyBal on amslvs1 |
[production] |
| 17:30 |
<mark> |
Stopped IPVS backup sync daemon on amslvs4 |
[production] |
| 17:27 |
<mark> |
Rebooting amslvs2 |
[production] |
| 17:19 |
<mark> |
Stopped PyBal on amslvs2 |
[production] |
| 17:14 |
<mark> |
Drive failed in thistle |
[production] |
| 17:09 |
<mark> |
Started temporary LVS state syncing between amslvs1->amslvs3 and amslvs2->amslvs4, preparing for reboot of amslvs1-2 |
[production] |
| 16:51 |
<mark> |
Rebooted (backup) LVS servers amslvs3 and amslvs4 |
[production] |
| 16:36 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'adding db28 to s5' |
[production] |
| 16:20 |
<mark> |
Selectively updated pybal/bgp.py on lvs2-4 and restarted PyBal |
[production] |
| 16:13 |
<mark> |
Upgraded PyBal to r0.1+74215 on amslvs1-4 |
[production] |
| 14:27 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 14:12 |
<domas> |
db22 was first wikimedia slave initialized using xtrabackup, hehe |
[production] |
| 14:08 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'adding db22 as commons 5.1 slave' |
[production] |
| 14:06 |
<mark> |
Restored nameservers order for mobile serverswq |
[production] |
| 13:36 |
<mark> |
lvs3 service ips were unreachable from outside the subnet for some reason. Restarting PyBal (and therefore the BGP session to the router) fixed it |
[production] |
| 13:22 |
<mark> |
Started pdns recursor on dobson again |
[production] |
| 13:19 |
<mark> |
Once again stopped pnds-recursor on dobson |
[production] |
| 13:18 |
<mark> |
Restarted Apache on the mobile servers one by one, to put the new resolv.conf in effect |
[production] |
| 13:10 |
<mark> |
Swapped nameservers for the mobile servers, so they don't crash the mobile site while I reinstall the primary recursor |
[production] |
| 13:02 |
<catrope> |
synchronized php-1.5/wmf-config/checkers.php 'Log no-UA requests for exempt IPs to /h/w/l/nouaexempt.log' |
[production] |
| 12:45 |
<mark> |
Mobile site broke with missing primary DNS recursor, restarted it |
[production] |
| 12:44 |
<mark> |
Stopped DNS recursor on dobson again |
[production] |