|
2010-03-24
§
|
| 22:07 |
<mark> |
Sending European traffic back to esams |
[production] |
| 22:01 |
<Rob> |
sq46 was unresponsive and rebooted, now is working fine |
[production] |
| 21:30 |
<Rob> |
secure.wikimedia.org back online |
[production] |
| 20:47 |
<mark> |
Renamed rr.wikimedia.org to text.wikimedia.org |
[production] |
| 19:38 |
<mark> |
Brought esams Sun V20z servers back up - restored rr.esams and upload.esams LVS |
[production] |
| 19:26 |
<Tempodivalse> |
Can I log things too? |
[production] |
| 19:24 |
<mark> |
Bringing lily back up |
[production] |
| 19:07 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '#22924 - Enable the Collection extension at el.wikipedia' |
[production] |
| 18:40 |
<jeluf> |
synchronized live-1.5/robots.php |
[production] |
| 18:39 |
<analyst> |
logger url |
[production] |
| 18:37 |
<river> |
delegated 1.24.10.in-addr.arpa to ha-dns-auth.toolserver.org |
[production] |
| 18:17 |
<Rob> |
techdetail: lowered maxclients on singer apache2 and disabled secure site in apache configuration, do not rollback changes to normal until after esams is back online |
[production] |
| 18:16 |
<Rob> |
techlbog back up, secure is NOT up, and is not expected to come back up, until the esams outage is resolved |
[production] |
| 18:01 |
<Rob> |
disabled secure on singer, as everyone trying to use it overloaded both secure and blogs. |
[production] |
| 18:01 |
<Rob> |
Secure is offline |
[production] |
| 17:51 |
<mark> |
http://techblog.wikimedia.org/2010/03/global-outage-cooling-failure-and-dns/ |
[production] |
| 17:50 |
<mark> |
Reduced Negative TTL setting in SOA record of wikimedia.org from 1H to 10M |
[production] |
| 17:40 |
<domas> |
NXDOMAINs with 1h TTL served on all records for short moment |
[production] |
| 17:40 |
<Rob> |
globally (robs bad) |
[production] |
| 17:40 |
<Rob> |
pending techblog post regarding the issue with accessibility in EU |
[production] |
| 17:25 |
<mark> |
DNS scenario dir esams-down was empty, all zone files missing. Fixed. |
[production] |
| 17:18 |
<mark> |
esams down due to overheating issues; moving traffic to Florida |
[production] |
| 14:18 |
<mark> |
csw2-oe12-esams connected to iris's serial console (temp) |
[production] |
| 04:24 |
<andrew> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 04:21 |
<Andrew> |
Running scap to update LiquidThreads beta |
[production] |
| 04:17 |
<Andrew> |
Preparing to update LiquidThreads beta (production) to alpha version. |
[production] |
|
2010-03-23
§
|
| 22:16 |
<apergos> |
latest patches and reboot for ms5 |
[production] |
| 22:13 |
<Rob> |
sq40 back online, sq41 reinstalling |
[production] |
| 21:50 |
<Rob> |
reinstalling sq40 & sq41 |
[production] |
| 21:27 |
<Rob> |
sq38 & sq39 reinstalled and online |
[production] |
| 20:43 |
<Rob> |
taking down sq38 & sq39 for reinstallation |
[production] |
| 20:43 |
<Rob> |
sq36 & sq37 back online and in rotation |
[production] |
| 19:39 |
<Rob> |
sq36 & sq37 being reinstalled |
[production] |
| 19:35 |
<jeluf> |
synchronized live-1.5/robots.php 'change charset to utf8' |
[production] |
| 19:32 |
<Rob> |
sq34 and sq35 (not 25) |
[production] |
| 19:32 |
<Rob> |
sq34 and sq25 back online and in rotation |
[production] |
| 19:27 |
<jeluf> |
synchronized all.dblist |
[production] |
| 19:24 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.hooks.php 'r64086' |
[production] |
| 19:24 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.php 'r64086' |
[production] |
| 19:22 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Add vector-collapsiblenav as a hiddenpref' |
[production] |
| 19:21 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed more group permission for closed wikis' |
[production] |
| 18:34 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove all simplewikibooks configs from Initialisesettings' |
[production] |
| 18:32 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks' |
[production] |
| 18:29 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks' |
[production] |
| 18:28 |
<Rob> |
sq34 & sq35 coming down for reinstallation |
[production] |
| 18:23 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks' |
[production] |
| 18:22 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks' |
[production] |
| 18:19 |
<Rob> |
sq33 is reinstalled and doing quite fine. |
[production] |
| 16:02 |
<Rob> |
puppet stopped on sq34-sq50, DO NOT RESTART PUPPET. These hosts need reinstallation before puppet can be restarted, or squid changes can be deployed to them. |
[production] |
| 15:52 |
<Rob> |
changed the setting for pmtpa to change partitioning schema and up the cache size. Do not deploy old squids with this information until Rob finishes reinstalling them |
[production] |