|
2012-07-01
§
|
| 04:02 |
<Ryan_Lane> |
rebooting virt1000 |
[production] |
| 03:16 |
<LocalisationUpdate> |
completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012 |
[production] |
| 01:43 |
<notpeter> |
that worked. restarting all remaining search nodes. |
[production] |
| 01:39 |
<notpeter> |
problem with lucene persisting through service restart, but not node restart. restarting en pool nodes. |
[production] |
| 01:20 |
<paravoid> |
restarting opendj (nfs1/nfs2), load spike, possibly related to leap second |
[production] |
| 00:51 |
<notpeter> |
search1004 dead. powercycling. |
[production] |
| 00:50 |
<notpeter> |
based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between |
[production] |
|
2012-06-29
§
|
| 19:49 |
<hashar> |
restarting Jenkins to fix an issue with "parameterized builds" plugin. Updated git plugin as well. |
[production] |
| 19:35 |
<RobH> |
dns update via authdns-update for vanadium ip |
[production] |
| 18:05 |
<Jeff_Green> |
sync-apache and apache-graceful-all for http://donate.wikimedia.org-->https redirect |
[production] |
| 16:02 |
<RobH> |
ms-be1001 and ms-be1002 powering down for ssd installation |
[production] |
| 15:59 |
<RobH> |
authdns-update run |
[production] |
| 15:47 |
<RobH> |
updating dns |
[production] |
| 15:16 |
<mutante> |
dist-upgrading srv280,srv270,srv264 |
[production] |
| 15:11 |
<Jeff_Green> |
apache-graceful-all for redirect conf change |
[production] |
| 15:10 |
<Jeff_Green> |
sync-apache to push out new foundation.conf |
[production] |
| 14:50 |
<mark> |
Reinstalled chromium with precise |
[production] |
| 13:46 |
<hashar> |
fixed interwiki on http://wikisource.org/ main page by hacking a script in production and refreshing cache |
[production] |
| 13:46 |
<hashar> |
synchronized php-1.20wmf6/cache/interwiki.cdb 'Updating interwiki cache for 1.20wmf6' |
[production] |
| 13:32 |
<hashar> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 12:38 |
<dzahn> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 12:38 |
<mutante> |
dumping interwiki and updating interwiki cache (to fix broken interwiki links, like wikisource.org -> wikipedia.org) |
[production] |
| 09:31 |
<hashar> |
Jenkins: deployed gitsqlhaschanged patch ( d04f779 0f069c3 integration/jenkins.git ) |
[production] |
| 07:56 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'send header from CS.php only for non CLI scripts {{gerrit|13435}}' |
[production] |
| 07:08 |
<mutante> |
upgrading apt packages on brewster |
[production] |
| 02:51 |
<LocalisationUpdate> |
completed (1.20wmf5) at Fri Jun 29 02:51:13 UTC 2012 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.20wmf6) at Fri Jun 29 02:26:25 UTC 2012 |
[production] |
|
2012-06-28
§
|
| 23:30 |
<reedy> |
synchronized php-1.20wmf6/includes/resourceloader/ResourceLoader.php |
[production] |
| 23:15 |
<binasher> |
completed aft offload_large_feedback migration on enwiki |
[production] |
| 23:03 |
<asher> |
synchronized wmf-config/db.php 'returning db36' |
[production] |
| 21:50 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
| 21:39 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 21:13 |
<asher> |
synchronized wmf-config/db.php 'temp pulling db36' |
[production] |
| 20:38 |
<binasher> |
ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia |
[production] |
| 20:14 |
<RobH> |
dns update for pc1-pc3 |
[production] |
| 19:54 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
| 19:08 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 18:35 |
<hashar> |
so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-((( |
[production] |
| 18:34 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:33 |
<hashar> |
srv190 and srv281 got ssh timeout |
[production] |
| 18:31 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:30 |
<hashar> |
did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing |
[production] |
| 18:25 |
<hashar> |
updating mediawiki-config to grab a12545d edceb4c & eee97ad |
[production] |
| 18:23 |
<RobHalsell> |
swapped bad psu out of ms1001-array3, redundant so no downtime |
[production] |
| 15:40 |
<RobHalsell> |
pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium |
[production] |
| 15:34 |
<RobHalsell> |
dns updated |
[production] |