|
2012-06-28
§
|
| 21:39 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 21:13 |
<asher> |
synchronized wmf-config/db.php 'temp pulling db36' |
[production] |
| 20:38 |
<binasher> |
ran aftv5 offload_large_feedback migrations on testwiki and en_labswikimedia |
[production] |
| 20:14 |
<RobH> |
dns update for pc1-pc3 |
[production] |
| 19:54 |
<kaldari> |
Finished syncing Wikimedia installation... : |
[production] |
| 19:08 |
<kaldari> |
Started syncing Wikimedia installation... : |
[production] |
| 18:35 |
<hashar> |
so the nicely reviewed changes broke the enwiki stylesheets :/ reverted change :-((( |
[production] |
| 18:34 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:33 |
<hashar> |
srv190 and srv281 got ssh timeout |
[production] |
| 18:31 |
<hashar> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 18:30 |
<hashar> |
did various tests using eval.php. Most important is $realm -> production. $cluster -> pmtpa. Syncing |
[production] |
| 18:25 |
<hashar> |
updating mediawiki-config to grab a12545d edceb4c & eee97ad |
[production] |
| 18:23 |
<RobHalsell> |
swapped bad psu out of ms1001-array3, redundant so no downtime |
[production] |
| 15:40 |
<RobHalsell> |
pulling the following servers, relocating to payments rack: payments1001-1004, boron, beryllium, lithium |
[production] |
| 15:34 |
<RobHalsell> |
dns updated |
[production] |
| 15:31 |
<RobHalsell> |
boron appears to be unallocated, pulling IP allocation, rack allocation, moving to payments per 1227 |
[production] |
| 14:40 |
<mutante> |
svn server is rebooting.brb |
[production] |
| 14:38 |
<mutante> |
dist-upgrading formey (svn/gerrit), rebooting soon |
[production] |
| 14:38 |
<Jeff_Green> |
manganese rebooted for kernel update |
[production] |
| 14:38 |
<RobH> |
allocating yttrium to payments rack per rt 1227 |
[production] |
| 14:25 |
<Jeff_Green> |
manganese dist-upgrade |
[production] |
| 14:15 |
<Ryan_Lane> |
restarting apache on manganese |
[production] |
| 14:15 |
<Ryan_Lane> |
restarting gerrit |
[production] |
| 05:08 |
<Tim> |
srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common. |
[production] |
| 05:03 |
<Tim> |
fixed fatal.log on fenari, socat was writing to a deleted file |
[production] |
| 02:58 |
<LocalisationUpdate> |
completed (1.20wmf5) at Thu Jun 28 02:58:42 UTC 2012 |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.20wmf6) at Thu Jun 28 02:30:03 UTC 2012 |
[production] |
|
2012-06-27
§
|
| 23:50 |
<K4-7131> |
sync'd payments cluster to 592e0a5ba195 |
[production] |
| 23:43 |
<preilly> |
synchronized wmf-config/InitialiseSettings.php 'add rule for mediawiki' |
[production] |
| 22:37 |
<K4-7131> |
sync'd payments cluster to 7e9072c2d571c |
[production] |
| 22:23 |
<binasher> |
temporarily pulling srv211 from pybal |
[production] |
| 21:56 |
<RobH> |
mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall! (rt 3058) |
[production] |
| 21:01 |
<preilly> |
synchronized php-1.20wmf6/extensions/MobileFrontend 'fix CSRF' |
[production] |
| 21:00 |
<preilly> |
synchronized php-1.20wmf5/extensions/MobileFrontend 'fix CSRF' |
[production] |
| 20:55 |
<RobH> |
db1003 back online, replaced mgmt cable and mgmt is working now as well |
[production] |
| 20:51 |
<LeslieCarr> |
rebooting srv266 as it is unresponsive |
[production] |
| 20:44 |
<RobH> |
db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable |
[production] |
| 20:35 |
<RobH> |
clean mysql shutdown, db1003 now offline |
[production] |
| 20:33 |
<RobH> |
db1003 mgmt is not responsible, I need to remove power and reboot. confirmed iwth asher this is an s3 slave and can do a short downtime without issues |
[production] |
| 20:32 |
<maxsem> |
synchronized php-1.20wmf5/extensions/MobileFrontend/ 'MF fixes and logging' |
[production] |
| 20:29 |
<maxsem> |
synchronized php-1.20wmf6/extensions/MobileFrontend/ 'MF fixes and logging' |
[production] |
| 19:56 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'disable LastModified and LastModified/E3Experiment' |
[production] |
| 19:54 |
<reedy> |
synchronized php-1.20wmf6/maintenance/runJobs.php |
[production] |
| 19:53 |
<reedy> |
synchronized php-1.20wmf5/maintenance/runJobs.php |
[production] |
| 19:32 |
<reedy> |
synchronized php-1.20wmf6/extensions/Translate/specials/SpecialLanguageStats.php |
[production] |
| 19:19 |
<reedy> |
synchronized php-1.20wmf6/extensions/Translate/ |
[production] |
| 19:06 |
<reedy> |
synchronized php-1.20wmf6/extensions/Translate/ |
[production] |
| 19:03 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: meta back to wmf6, not cause of translate issues |
[production] |
| 18:53 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: closed to 1.20wmf6 |
[production] |
| 18:51 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia wikis to 1.20wmf6 |
[production] |