|
2010-11-24
§
|
| 15:46 |
<RoanKattouw> |
mobile3 is broken, serving 500s for http://m.wikipedia.org . mobile1 is fine; this causes LVS HTTP status to flap in Nagios |
[production] |
| 03:57 |
<aaron> |
synchronizing Wikimedia installation... Revision: 77216: |
[production] |
| 03:50 |
<aaron> |
synchronizing Wikimedia installation... Revision: 77216: |
[production] |
| 03:15 |
<demon> |
synchronized php-1.5/extensions/FlaggedRevs/api/ApiReview.php 'r77216' |
[production] |
| 03:15 |
<demon> |
synchronized php-1.5/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'r77216' |
[production] |
| 02:55 |
<demon> |
synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'r77213' |
[production] |
| 02:43 |
<Ryan_Lane> |
restarted memcached on owa1/2 to read new config file |
[production] |
| 00:19 |
<demon> |
synchronizing Wikimedia installation... Revision: 77209: |
[production] |
| 00:17 |
<^demon> |
syncing FlaggedRevs changes to all wikis |
[production] |
|
2010-11-23
§
|
| 23:11 |
<demon> |
synchronized php-1.5/includes/specials/SpecialRecentchangeslinked.php 'r76485' |
[production] |
| 23:10 |
<demon> |
synchronized php-1.5/includes/specials/SpecialRecentchanges.php 'r76485' |
[production] |
| 23:09 |
<demon> |
synchronized php-1.5/includes/diff/DifferenceInterface.php 'r76444' |
[production] |
| 21:58 |
<^demon> |
restarted wmfsvndumper with new dump location |
[production] |
| 21:44 |
<^demon> |
running svndump test in screen on formey |
[production] |
| 21:43 |
<Ryan_Lane> |
installed screen on formey |
[production] |
| 21:42 |
<Ryan_Lane> |
added demon to svnadm |
[production] |
| 21:41 |
<Ryan_Lane> |
changed ownership of svn config files from adm to svnadm |
[production] |
| 21:41 |
<Ryan_Lane> |
added an svnadm group to ldap |
[production] |
| 15:07 |
<mark> |
Finished testing, amslvs1 is primary again |
[production] |
| 14:40 |
<mark> |
Shutdown BGP session to amslvs1 on csw1-esams for testing, traffic failing over to amslvs3 |
[production] |
| 13:50 |
<mark> |
Removed redistribution of BGP LVS routes into OSPF; rely on iBGP exclusively for LVS routes |
[production] |
| 12:06 |
<catrope> |
synchronized php-1.5/includes/api/ApiDelete.php 'r77146' |
[production] |
| 02:27 |
<RobH> |
had to restart morebots, it had left all channels. also had to change admin pass for listserv |
[production] |
|
2010-11-22
§
|
| 22:09 |
<mark> |
Restarted varnish on knsq4 |
[production] |
| 22:05 |
<mark> |
Restarting varnish on knsq1 |
[production] |
| 21:44 |
<Stifle> |
Does this work? |
[production] |
| 21:44 |
<Ryan_Lane> |
err - restarting varnish on knsq2,4,5 |
[production] |
| 21:44 |
<Ryan_Lane> |
restarting varnish on knsq2,3,4 |
[production] |
| 21:43 |
<Ryan_Lane> |
restarting knsq2,3,4 |
[production] |
| 21:29 |
<mark> |
Restarted varnish on knsq2,4,5 |
[production] |
| 21:24 |
<mark> |
Restarting Varnish on knsq1 |
[production] |
| 21:09 |
<mark> |
Set static routes 91.198.174.232/31 to 91.198.174.247 (csw1-esams) on br1-knams |
[production] |
| 21:03 |
<mark> |
Started pybal on amslvs1 again |
[production] |
| 20:53 |
<mark> |
Killed PyBal on amslvs1 |
[production] |
| 20:39 |
<mark> |
Lowered cache_mem and cache_dir sizes on amssq31 for testing |
[production] |
| 20:28 |
<Ryan_Lane> |
depooling knsq24 |
[production] |
| 20:25 |
<mark> |
Restarted pybal on amslvs1 with depool-threshold = 1 (temporarily) |
[production] |
| 17:27 |
<Ryan_Lane> |
restarting nagios |
[production] |
| 17:27 |
<Ryan_Lane> |
adding warning notifications via IRC to nagios |
[production] |
| 16:40 |
<mark> |
Suppressed announcements to AS16265 on csw1-esams |
[production] |
| 16:30 |
<mark> |
Suppressed BGP announcements to AS13030 on br1-knams |
[production] |