|
2011-07-28
§
|
| 17:44 |
<LocalisationUpdate> |
failed |
[production] |
| 17:23 |
<Reedy> |
Manual run of /home/wikipedia/bin/sync-l10nupdate after updated permissions |
[production] |
| 16:56 |
<aaron> |
synchronized wmf-config/InitialiseSettings.php 'touched file' |
[production] |
| 16:52 |
<aaron> |
synchronized wmf-config/CommonSettings.php 're-pushed Het Deploy changes with $dbSuffix fix' |
[production] |
| 16:51 |
<RobH> |
nagios wasnt running, didnt investigate, just restarted service |
[production] |
| 16:44 |
<AaronSchulz> |
Updated set-group-write2 to handle wmf-config being move up a dir |
[production] |
| 12:45 |
<Reedy> |
Also, LocalisationUpdate doesn't seem to be logging to /var/log/l10nupdate as it is supposed to |
[production] |
| 12:43 |
<Reedy> |
Seems LocalisationUpdate may be broken due to permission issues on l10n files on apaches |
[production] |
| 12:39 |
<mark> |
Replaced SSDs in amssq31 and dist-upgrade |
[production] |
| 12:27 |
<mark> |
Brought up BGP session to AS 38930 |
[production] |
| 12:21 |
<Reedy> |
Running l10nupdate manually on fenari |
[production] |
| 12:18 |
<Reedy> |
Reports of LocalisationUpdate may have stopped working again |
[production] |
| 12:12 |
<mark> |
Shutdown session to AS 16265 |
[production] |
| 01:18 |
<andrewS> |
dongle found. reconnected serial to pdu d3-sdtpa. |
[production] |
|
2011-07-27
§
|
| 23:47 |
<preilly> |
pushing bad redirect logging to mobile cluster |
[production] |
| 23:37 |
<maplebed> |
started reslave from db42 to db1047; poor performance expected for the duration of the sync. |
[production] |
| 23:33 |
<notpeter> |
restarting torrus again |
[production] |
| 23:19 |
<preilly> |
pushing badredirect log to mobile cluster |
[production] |
| 22:46 |
<notpeter> |
upgrading squid and squid-frontend on knsq27 and knsq28 |
[production] |
| 22:13 |
<notpeter> |
torrus stuck. assuming in deadlock. following instructions for restart |
[production] |
| 22:01 |
<AaronSchulz> |
Re-installed moodbar & edittracking. Cleaned up wmf-config so that we are on HEAD (but with the reverted stuff still gone) |
[production] |
| 22:00 |
<aaron> |
synchronizing Wikimedia installation... Revision: 93132: |
[production] |
| 20:19 |
<notpeter> |
upgrading squid and squid-frontend on knsq29 and knsq30 |
[production] |
| 19:54 |
<binasher> |
pulled sq78 frontend from lvs |
[production] |
| 19:40 |
<Reedy> |
Reports of users using non mobile browsers being redirected to mobile sites (enwiki and dewiki) |
[production] |
| 19:04 |
<mark> |
Temporarily disabled facilities.pp file in puppet, until there is time to debug |
[production] |
| 18:35 |
<binasher> |
pulled sq78 |
[production] |
| 18:32 |
<binasher> |
sq78 frontend is in service with mobile redirection |
[production] |
| 17:08 |
<binasher> |
pulling sq78 from lvs prior to mobile testing |
[production] |
| 16:56 |
<aaron> |
synchronized multiversion/MWMultiVersion.php 'deployed [[rev:93255|r93255]]' |
[production] |
| 13:40 |
<Reedy> |
Do not push any configuration change from wmf-config without speaking to Reedy/Ariel first |
[production] |
| 13:33 |
<Reedy> |
Seems some configuration files between [[rev:2087|r2087]] and [[rev:2088|r2088]] weren't pushed to site, pushing them broke most projects. Aaron to investigate when he's in the office |
[production] |
| 13:20 |
<reedy> |
synchronized wmf-config/extension-list '[[rev:2087|r2087]]' |
[production] |
| 13:19 |
<reedy> |
synchronized wmf-config/liquidthreads.php '[[rev:2087|r2087]]' |
[production] |
| 13:19 |
<reedy> |
synchronized wmf-config/ExtensionMessages.php '[[rev:2087|r2087]]' |
[production] |
| 13:18 |
<Reedy> |
Site should be somewhat back to normal, reverted to [[rev:2087|r2087]] of config from private SVN |
[production] |
| 13:17 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[rev:2087|r2087]]' |
[production] |
| 13:17 |
<reedy> |
synchronized wmf-config/CommonSettings.php '[[rev:2087|r2087]]' |
[production] |
| 13:15 |
<reedy> |
synchronized wmf-config/CommonSettings-old.php |
[production] |
| 13:14 |
<reedy> |
synchronized wmf-config/CommonSettings-new.php |
[production] |
| 13:12 |
<reedy> |
synchronized wmf-config/CommonSettings.php '[[rev:2088|r2088]]' |
[production] |
| 13:12 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php '[[rev:2088|r2088]]' |
[production] |
| 12:36 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Reverting last push' |
[production] |
| 12:36 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Reverting last push' |
[production] |
| 12:32 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Reverting last push' |
[production] |
| 12:31 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Reverting last push' |
[production] |
| 12:27 |
<andrewS> |
miguel gave OK to fill up d1 in sdtpa as the heat flow issues have been temporarily resolved (also said construction on new AC units was to begin Aug 1st but has not heard anything yet). |
[production] |
| 12:20 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 12:20 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 11:55 |
<mutante> |
sq63 - restored software RAID & dist-upgrade & reboot |
[production] |