|
2011-11-10
§
|
| 17:26 |
<mark> |
Increased ms5 nginx worker processes from 4 to 32 (out of puppet) |
[production] |
| 17:25 |
<mark> |
Increased ms6 worker processes from 4 to 32 (out of puppet) |
[production] |
| 17:06 |
<mark> |
Moving esams traffic back to esams to seed the upload squid SSD caches |
[production] |
| 14:51 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php 'resyncing for [[bugzilla:32309|bug 32309]]. Some servers did timeout' |
[production] |
| 14:48 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php '[[bugzilla:32309|bug 32309]] - Add WMF domains to $wgNoFollowDomainException' |
[production] |
| 12:20 |
<mark> |
Switching esams traffic to pmtpa |
[production] |
| 11:09 |
<domas> |
otrs still broken, williams had old passwords, something else is going on with mail delivery |
[production] |
| 10:46 |
<domas> |
One needs to run 'FLUSH PRIVILEGES' after modifying mysql.* tables directly. OTRS delivery was broken for a day or so? |
[production] |
| 10:22 |
<guillom> |
OTRS issue: e-mail isn't getting in. No new messages have come through in the last 15h and 40 min. Outgoing e-mail works fine. |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.18) at Thu Nov 10 02:32:45 UTC 2011 |
[production] |
| 02:12 |
<awjr> |
Updating thank you module code for CiviCRM on aluminium/grosley; [[rev:762|r762]] |
[production] |
| 01:57 |
<LocalisationUpdate> |
failed |
[production] |
| 01:34 |
<awjr> |
Updating civicrm on aluminium/grosley to [[rev:760|r760]] |
[production] |
| 01:17 |
<RobH> |
ocg2 installed per rt1593 |
[production] |
| 01:14 |
<Tim> |
started puppet on fenari |
[production] |
| 01:12 |
<RobH> |
ocg1 installed, awaiting puppetization (pending labs-ocg1 deployment) |
[production] |
| 00:43 |
<RobH> |
mobile3 (now ocg1) coming offline for reinstallation, ignore any mobile3 nagios errors |
[production] |
| 00:36 |
<RobH> |
another dns update, this is me pushing my luck on dns not crashing |
[production] |
| 00:29 |
<awjrichards> |
synchronized php/extensions/DonationInterface/gateway_common/us-states.i18n.php '[[rev:102610|r102610]]' |
[production] |
| 00:29 |
<awjrichards> |
synchronized php/extensions/DonationInterface/gateway_common/interface.i18n.php '[[rev:102610|r102610]]' |
[production] |
| 00:28 |
<awjrichards> |
synchronized php/extensions/DonationInterface/gateway_common/countries.i18n.php '[[rev:102610|r102610]]' |
[production] |
| 00:26 |
<awjrichards> |
synchronized php/extensions/DonationInterface/globalcollect_gateway/globalcollect_gateway.i18n.php '[[rev:102610|r102610]]' |
[production] |
| 00:25 |
<RobH> |
ocg dns update, once more with feeling |
[production] |
| 00:25 |
<awjrichards> |
synchronized php/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.i18n.php '[[rev:102610|r102610]]' |
[production] |
| 00:09 |
<binasher> |
db10 is back up, and caught up on replication |
[production] |
| 00:05 |
<RobH> |
updating dns to rename mobile1-mobile5 to ocg1-ocg5 |
[production] |
|
2011-11-09
§
|
| 23:40 |
<Tim> |
running puppetd -tv on fenari to test I6146f4b5 |
[production] |
| 23:38 |
<reedy> |
synchronizing Wikimedia installation... : [[rev:102598|r102598]] |
[production] |
| 23:34 |
<reedy> |
synchronized php-1.18/extensions/AbuseFilter/abusefilter.css '[[rev:102602|r102602]]' |
[production] |
| 23:33 |
<reedy> |
synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.dialogs.config.js '[[rev:102601|r102601]]' |
[production] |
| 23:04 |
<reedy> |
synchronizing Wikimedia installation... : [[rev:102588|r102588]] [[rev:102592|r102592]] [[rev:102593|r102593]] |
[production] |
| 22:25 |
<mark> |
nagios is back up |
[production] |
| 21:55 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 32312 - Enable WikiLove on Japanese Wikipedia' |
[production] |
| 21:54 |
<Reedy> |
Creating Wikilove tables on jawiki |
[production] |
| 21:00 |
<mark> |
Running puppetd --test in ddsh, concurrency 8 |
[production] |
| 20:29 |
<mark> |
Undid the sockpuppet firewall rules |
[production] |
| 20:17 |
<mark> |
Revert nagios changes |
[production] |
| 19:52 |
<aaron> |
synchronized wmf-config/InitialiseSettings.php 'removed references to patrolother right that does not exist' |
[production] |
| 19:48 |
<aaron> |
synchronized wmf-config/InitialiseSettings.php 'removed references to autopatrolother right that does not exist' |
[production] |
| 19:36 |
<ben> |
synchronized wmf-config/db.php 'switching all external store clusters except cluster22 (current active) from srv/ms hosts to new ES hardware es3 and es4.' |
[production] |
| 19:31 |
<mark> |
Truncated several puppet db tables again |
[production] |
| 19:26 |
<aaron> |
synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php 'deployed [[rev:102559|r102559]]' |
[production] |
| 18:44 |
<notpeter> |
restarting exim on mchenry |
[production] |
| 18:19 |
<mark> |
Enabled PassengerGlobalQueueing on sockpuppet |
[production] |
| 17:45 |
<mark> |
Temporarily firewalled all servers from sockpuppet, except spence |
[production] |
| 17:42 |
<mark> |
Purged all nagios resources in the puppet db on db9 |
[production] |
| 17:31 |
<catrope> |
synchronized php-1.18/cache/interwiki-pr.cdb 'Rebuilt interwiki cache again for [[bugzilla:29591|bug 29591]]' |
[production] |
| 17:31 |
<catrope> |
synchronized php-1.18/cache/interwiki.cdb 'Rebuilt interwiki cache again for [[bugzilla:29591|bug 29591]]' |
[production] |
| 17:20 |
<catrope> |
synchronized php-1.18/includes/api/ApiQuerySearch.php '[[rev:102538|r102538]]' |
[production] |
| 16:21 |
<mark> |
Rebooting lvs1001 for kernel upgrade |
[production] |