|
2009-10-08
§
|
| 15:15 |
<Fred> |
rebooting srv156 |
[production] |
| 15:14 |
<Andrew> |
4-cpu apaches seem to be out of memory and swapping |
[production] |
| 15:04 |
<Andrew> |
Scapping |
[production] |
| 15:03 |
<Andrew> |
Applying LiquidThreads schema change to liquidthreads_labswikimedia and updating/deploying LiquidThreads to r57518 state. |
[production] |
| 14:52 |
<mark> |
Redeployed ntp server on linne using puppet |
[production] |
| 14:32 |
<mark> |
DNS cleanup: removed all traces of yaseo, and 66.230.200.0/24 |
[production] |
| 14:26 |
<mark> |
Unbound 66.230.200.10/24 range from csw5-pmtpa interface ve5. |
[production] |
| 14:24 |
<mark> |
Removed ip 66.230.200.234 from zwinger |
[production] |
| 13:38 |
<mark> |
Restarting all ServerTech CDUs for NTP configuration change to take effect |
[production] |
| 07:16 |
<apergos> |
shot the pr_query_count queries on dbs 17 18 19 21 25 as domas did his sync, they are all cleaned up now |
[production] |
| 07:02 |
<midom> |
synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'wmf:r57507' |
[production] |
| 06:00 |
<Fred> |
restarted srv100-139-150 for good measure since apache was not running. |
[production] |
|
2009-10-07
§
|
| 22:51 |
<AaronSchulz> |
running populateLogSearch.php to fill suppression log search filter data |
[production] |
| 22:39 |
<aaron> |
synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed r57501' |
[production] |
| 21:30 |
<Fred> |
powercycling pdf1 -> c�gone unresponsive even from serial console |
[production] |
| 21:29 |
<brion> |
synchronized php-1.5/includes/specials/SpecialContributions.php |
[production] |
| 21:27 |
<brion> |
pdf1 unreachable |
[production] |
| 20:12 |
<aaron> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'deleted content rights for checkusers on frwiki' |
[production] |
| 20:11 |
<aaron> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'deleted content rights for checkusers on frwiki' |
[production] |
| 19:41 |
<Fred> |
ganglia 3.1.2 has been installed on spence. Configuration still in progress, but mostly done. http://ganglia3.wikimedia.org |
[production] |
| 19:28 |
<Rob> |
resynced nagios for all the removed and decommissioned servers |
[production] |
| 19:28 |
<Rob> |
decommissioned srv142 & srv143. wiped, unracked |
[production] |
| 19:12 |
<Rob> |
srv85 decommissioned |
[production] |
| 19:10 |
<Rob> |
srv148 dead, decommissioned, removed from dsh, lvs, and server roles |
[production] |
| 19:04 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'removed second level for now' |
[production] |
| 19:00 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs wiki config' |
[production] |
| 18:59 |
<Rob> |
decommissioning srv51, removed from server roles, node groups, lvs, and wiping in rack |
[production] |
| 18:56 |
<Rob> |
decommissioned srv48, srv49, srv50. removed from node groups, server roles, wiped, unracked. |
[production] |
| 18:50 |
<aaron> |
synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57480' |
[production] |
| 18:41 |
<mark> |
Installed puppet on grosley |
[production] |
| 18:39 |
<brion> |
synchronized php-1.5/includes/specials/SpecialContributions.php 'sync r57479 spacing fix for contribs tools links' |
[production] |
| 18:34 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs wiki config' |
[production] |
| 18:33 |
<brion> |
cleared flaggedrevs, flaggedpage_config tables on flaggedrevs_labwikimedia: prep for new config test |
[production] |
| 18:28 |
<aaron> |
synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57475' |
[production] |
| 18:28 |
<mark> |
Manually updated ntp.conf on isidore |
[production] |
| 18:18 |
<brion> |
synchronized php-1.5/includes/OutputPage.php 'r57473 fix for fatals in edge case' |
[production] |
| 18:12 |
<mark> |
Manually updated ntp.conf on srv1, srv2 and bart |
[production] |
| 18:06 |
<brion> |
synchronized php-1.5/includes/HistoryPage.php 'r57471 perm fix for show/hide revs box on history page' |
[production] |
| 17:46 |
<brion> |
synchronized php-1.5/includes/OutputPage.php 'sync r57468 fix -- mdale changes to script loading broke wgStylePath again' |
[production] |
| 17:46 |
<mark> |
Installed puppet on loudon and locke |
[production] |
| 17:43 |
<mark> |
Installed puppet on sq1 |
[production] |
| 17:41 |
<mark> |
Installed puppet on tridge |
[production] |
| 17:28 |
<brion> |
js2-borne code is breaking wikibits.js again; mdale please make sure $wgStylePath is respected |
[production] |
| 17:22 |
<brion> |
during update some brief borkage with missing stylesheets. inconsistency in file update order? seems ok now |
[production] |
| 16:58 |
<Rob> |
decommissioning srv48, srv49, srv50. removed from server_roles, node groups, running wipe in rack |
[production] |
| 16:52 |
<Rob> |
srv81 crashed, decommissioning, removed from node groups, server roles, lvs, wiping in rack |
[production] |
| 16:35 |
<Rob> |
pulled srv44-srv47 from rack, decommissioned |
[production] |
| 16:17 |
<Rob> |
srv31 already wiped, unracked |
[production] |
| 16:15 |
<mark> |
Installed puppet on singer |
[production] |
| 16:14 |
<Rob> |
srv31 still racked, but is crashed, decommissioning. |
[production] |