|
2012-07-02
§
|
| 09:47 |
<Tim> |
fixing leap second issue on formey,grosley,hooper,sanger,sockpuppet |
[production] |
| 09:43 |
<Tim> |
on fenari: fixed leap second issue with the mozilla method |
[production] |
| 09:39 |
<apergos> |
rebooting gallium, it's pretty unhappy (maybe related to leap second issue) |
[production] |
| 08:14 |
<hashar:> |
srv190 srv266 srv281 timeouts on sync-file |
[production] |
| 08:14 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php 'Bug 37457 - fix import sources for viwikibooks' |
[production] |
| 08:11 |
<hashar> |
Stopped Jenkins on gallium. It is not doing anything anyway. Asked to reboot box {{rt|3208}} |
[production] |
| 02:53 |
<LocalisationUpdate> |
completed (1.20wmf5) at Mon Jul 2 02:53:51 UTC 2012 |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.20wmf6) at Mon Jul 2 02:28:48 UTC 2012 |
[production] |
| 01:48 |
<Tim> |
kill -CONT on populateRevisionSha1.php processes |
[production] |
| 00:47 |
<Tim> |
on nfs1: trying leap second fix suggested at https://bugzilla.mozilla.org/show_bug.cgi?id=769972#c5 |
[production] |
| 00:26 |
<tstarling> |
synchronized wmf-config/db.php 'reduce db32 read load to zero due to persistent lag' |
[production] |
| 00:12 |
<Tim> |
switched enwiki back to r/w |
[production] |
| 00:12 |
<tstarling> |
synchronized wmf-config/db.php |
[production] |
| 00:06 |
<Tim> |
on hume: stopped all populateRevisionSha1.php processes with kill -STOP |
[production] |
| 00:03 |
<reedy> |
synchronized wmf-config/db.php 's1/enwiki into readonly' |
[production] |
|
2012-07-01
§
|
| 19:12 |
<reedy> |
synchronized php-1.20wmf6/extensions/WikimediaMaintenance/ 'Update to master for hashar' |
[production] |
| 17:55 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php 'more logging' |
[production] |
| 17:45 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php 'more logging' |
[production] |
| 17:43 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php 'more logging' |
[production] |
| 17:32 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php |
[production] |
| 17:30 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php |
[production] |
| 16:53 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php |
[production] |
| 16:48 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php 'logging' |
[production] |
| 12:54 |
<notpeter> |
also going to reboot all pmtpa search nodes. not in prod, but are still freaking out from leap second bug. |
[production] |
| 05:33 |
<aaron> |
synchronized php-1.20wmf5/includes/WikiPage.php 'logging' |
[production] |
| 04:25 |
<LocalisationUpdate> |
completed (1.20wmf5) at Sun Jul 1 04:25:25 UTC 2012 |
[production] |
| 04:06 |
<Ryan_Lane> |
virt1000 is back up, rebooting virt0 |
[production] |
| 04:02 |
<Ryan_Lane> |
rebooting virt1000 |
[production] |
| 03:16 |
<LocalisationUpdate> |
completed (1.20wmf6) at Sun Jul 1 03:16:39 UTC 2012 |
[production] |
| 01:43 |
<notpeter> |
that worked. restarting all remaining search nodes. |
[production] |
| 01:39 |
<notpeter> |
problem with lucene persisting through service restart, but not node restart. restarting en pool nodes. |
[production] |
| 01:20 |
<paravoid> |
restarting opendj (nfs1/nfs2), load spike, possibly related to leap second |
[production] |
| 00:51 |
<notpeter> |
search1004 dead. powercycling. |
[production] |
| 00:50 |
<notpeter> |
based on ganglia evidence, lucene seems to have been affected by leap second bug. restartig each instance, one minute wait in between |
[production] |
|
2012-06-29
§
|
| 19:49 |
<hashar> |
restarting Jenkins to fix an issue with "parameterized builds" plugin. Updated git plugin as well. |
[production] |
| 19:35 |
<RobH> |
dns update via authdns-update for vanadium ip |
[production] |
| 18:05 |
<Jeff_Green> |
sync-apache and apache-graceful-all for http://donate.wikimedia.org-->https redirect |
[production] |
| 16:02 |
<RobH> |
ms-be1001 and ms-be1002 powering down for ssd installation |
[production] |
| 15:59 |
<RobH> |
authdns-update run |
[production] |
| 15:47 |
<RobH> |
updating dns |
[production] |
| 15:16 |
<mutante> |
dist-upgrading srv280,srv270,srv264 |
[production] |
| 15:11 |
<Jeff_Green> |
apache-graceful-all for redirect conf change |
[production] |
| 15:10 |
<Jeff_Green> |
sync-apache to push out new foundation.conf |
[production] |
| 14:50 |
<mark> |
Reinstalled chromium with precise |
[production] |
| 13:46 |
<hashar> |
fixed interwiki on http://wikisource.org/ main page by hacking a script in production and refreshing cache |
[production] |
| 13:46 |
<hashar> |
synchronized php-1.20wmf6/cache/interwiki.cdb 'Updating interwiki cache for 1.20wmf6' |
[production] |