2012-06-28
§
|
23:30 |
<reedy> |
synchronized php-1.20wmf6/includes/resourceloader/ResourceLoader.php |
[production] |
23:15 |
<binasher> |
completed aft offload_large_feedback migration on enwiki |
[production] |
23:03 |
<asher> |
synchronized wmf-config/db.php 'returning db36' |
[production] |
21:50 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
21:39 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
21:13 |
<asher> |
synchronized wmf-config/db.php 'temp pulling db36' |
[production] |
20:38 |
<binasher> |
ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia |
[production] |
20:14 |
<RobH> |
dns update for pc1-pc3 |
[production] |
19:54 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
19:08 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
18:35 |
<hashar> |
so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-((( |
[production] |
18:34 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
18:33 |
<hashar> |
srv190 and srv281 got ssh timeout |
[production] |
18:31 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
18:30 |
<hashar> |
did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing |
[production] |
18:25 |
<hashar> |
updating mediawiki-config to grab a12545d edceb4c & eee97ad |
[production] |
18:23 |
<RobHalsell> |
swapped bad psu out of ms1001-array3, redundant so no downtime |
[production] |
15:40 |
<RobHalsell> |
pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium |
[production] |
15:34 |
<RobHalsell> |
dns updated |
[production] |
15:31 |
<RobHalsell> |
boron appears to be unallocated, pulling IP allocation, rack allocation, moving to payments per 1227 |
[production] |
14:40 |
<mutante> |
svn server is rebooting.brb |
[production] |
14:38 |
<mutante> |
dist-upgrading formey (svn/gerrit), rebooting soon |
[production] |
14:38 |
<Jeff_Green> |
manganese rebooted for kernel update |
[production] |
14:38 |
<RobH> |
allocating yttrium to payments rack per rt 1227 |
[production] |
14:25 |
<Jeff_Green> |
manganese dist-upgrade |
[production] |
14:15 |
<Ryan_Lane> |
restarting apache on manganese |
[production] |
14:15 |
<Ryan_Lane> |
restarting gerrit |
[production] |
05:08 |
<Tim> |
srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common. |
[production] |
05:03 |
<Tim> |
fixed fatal.log on fenari, socat was writing to a deleted file |
[production] |
02:58 |
<LocalisationUpdate> |
completed (1.20wmf5) at Thu Jun 28 02:58:42 UTC 2012 |
[production] |
02:30 |
<LocalisationUpdate> |
completed (1.20wmf6) at Thu Jun 28 02:30:03 UTC 2012 |
[production] |
2012-06-27
§
|
23:50 |
<K4-7131> |
sync'd payments cluster to 592e0a5ba195 |
[production] |
23:43 |
<preilly> |
synchronized wmf-config/InitialiseSettings.php 'add rule for mediawiki' |
[production] |
22:37 |
<K4-7131> |
sync'd payments cluster to 7e9072c2d571c |
[production] |
22:23 |
<binasher> |
temporarily pulling srv211 from pybal |
[production] |
21:56 |
<RobH> |
mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall! (rt 3058) |
[production] |
21:01 |
<preilly> |
synchronized php-1.20wmf6/extensions/MobileFrontend 'fix CSRF' |
[production] |
21:00 |
<preilly> |
synchronized php-1.20wmf5/extensions/MobileFrontend 'fix CSRF' |
[production] |
20:55 |
<RobH> |
db1003 back online, replaced mgmt cable and mgmt is working now as well |
[production] |
20:51 |
<LeslieCarr> |
rebooting srv266 as it is unresponsive |
[production] |
20:44 |
<RobH> |
db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable |
[production] |
20:35 |
<RobH> |
clean mysql shutdown, db1003 now offline |
[production] |
20:33 |
<RobH> |
db1003 mgmt is not responsible, I need to remove power and reboot. confirmed iwth asher this is an s3 slave and can do a short downtime without issues |
[production] |
20:32 |
<maxsem> |
synchronized php-1.20wmf5/extensions/MobileFrontend/ 'MF fixes and logging' |
[production] |