|
2009-06-30
§
|
| 23:20 |
<Andrew> |
23:19 <@brion> !log s1 switched to db16, enwiki back online! (thanks tim!) tomasz rebooted wikitech linode which also borked. :P |
[production] |
| 22:58 |
<brion> |
synchronized php-1.5/CommonSettings.php 'put non-enwiki back to read/write' |
[production] |
| 22:57 |
<brion> |
db14 (s1 master) is in some kind of borked state. slaves seem up to date; gonna try a master switch. |
[production] |
| 22:49 |
<brion> |
synchronized php-1.5/CommonSettings.php |
[production] |
| 22:49 |
<brion> |
putting site to read-only |
[production] |
| 22:44 |
<Andrew> |
Rolling back updates, first vector skin, then config changes. If that doesn't help, will roll back UsabilityInitiative changes |
[production] |
| 22:43 |
<andrew> |
synchronized php-1.5/skins/Vector.php 'Rolling back updates to r52581' |
[production] |
| 22:29 |
<Andrew> |
db14 seems to be overloaded |
[production] |
| 22:27 |
<Andrew> |
reports of (Cannot contact the database server: Unknown error (10.0.6.24)) |
[production] |
| 22:24 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Remove vector from wgSkipSkins on usabilitywiki' |
[production] |
| 22:21 |
<Andrew> |
Updated Vector and UsabilityInitiative, deployed UsabilityInitiative to usabilitywiki. Scapping to apply. |
[production] |
| 21:06 |
<tomaszf> |
restarting lighttpd on storage2 due to large i/o wait |
[production] |
| 20:01 |
<andrew> |
synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php |
[production] |
| 18:44 |
<tomaszf> |
dropping -grosley dns entries in favor of dev. and civicrm. |
[production] |
| 16:07 |
<Fred> |
restarted mwserve on pdf1 as it was not processing pdfs anymore. |
[production] |
| 15:56 |
<brion> |
activity on pdf1 seems to have croaked around 15:30. pdf daemon might need a restart? |
[production] |
| 04:28 |
<tstarling> |
synchronized php-1.5/extensions/timeline/Timeline.php |
[production] |
| 04:28 |
<tstarling> |
synchronized php-1.5/extensions/timeline/EasyTimeline.pl |
[production] |
| 04:27 |
<Tim> |
updated EasyTimeline to r52591 |
[production] |
| 03:34 |
<Rob> |
seeing good results with the firewall plugin on the corporate blog, copying it over to the techblog and setting it to active. |
[production] |
| 02:57 |
<Rob> |
It was net gremlins! |
[production] |
| 02:56 |
<brion> |
appears to have been temporary networking issue (external?) affecting access to some clients |
[production] |
| 02:48 |
<brion> |
some sort of downtime reported for a few minutes. no apparent current problems persisting, all looks well |
[production] |
| 02:11 |
<hcatlin> |
Mobile site is having some encoding issues with UTF-8 characters, therefore the redirects have been disabled from common.js |
[production] |
|
2009-06-29
§
|
| 23:03 |
<andrew> |
synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'Updated l10n' |
[production] |
| 23:00 |
<Fred> |
updated DNS for *.m.wikipedia.org to point to mobile1 instead of eiximenis |
[production] |
| 22:05 |
<andrew> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'Trevor told me to!' |
[production] |
| 22:05 |
<andrew> |
synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.css 'Trevor told me to!' |
[production] |
| 21:47 |
<Fred> |
apache stuck on srv[210-214] - restarting. |
[production] |
| 21:36 |
<Andrew> |
scapping for vector updates |
[production] |
| 21:35 |
<Andrew> |
updated Vector skin to r52581 |
[production] |
| 21:18 |
<Andrew> |
Lost connection to zwinger mid-scap. Running scap again to make sure everything's fine. |
[production] |
| 21:10 |
<Andrew> |
Scapping to deploy UsabilityInitiative CSS/JS changes, and license updates done by Fred. |
[production] |
| 21:09 |
<Andrew> |
updated WikimediaMessages to push latest changes pushed to svn by siebrand. |
[production] |
| 20:58 |
<Andrew> |
not scapping yet, deploying license change stuff too, waiting for Fred to do the rest of the updates for that |
[production] |
| 20:52 |
<Andrew> |
scapping |
[production] |
| 20:49 |
<Andrew> |
Activating UsabilityInitiative extension and removing vector from $wgSkipSkins on testwiki |
[production] |
| 20:38 |
<RobH_A90> |
racked and installed mobile1 |
[production] |
| 20:38 |
<RobH_A90> |
pulled nehlam test server |
[production] |
| 19:01 |
<Andrew> |
pdated UsabilityInitiative extension on test as prep for deployment on testwiki. |
[production] |
|
2009-06-28
§
|
| 19:39 |
<RobH_away> |
ran authdns-update |
[production] |
| 19:39 |
<RobH_away> |
changing bayes mgmt IP (to make it sane and in line with the rest of the mgmt IP ranges) and adding the new IP to dns and reverse template files |
[production] |
| 18:58 |
<brion> |
bayes back up |
[production] |
| 18:52 |
<midom> |
synchronized php-1.5/db.php 'db28 live' |
[production] |
| 18:49 |
<brion> |
poking LOM on bayes; box is down, EZ requested reboot. |
[production] |
| 08:54 |
<domas> |
manually reset db28 position, based on innodb internal info, got some filesystem relaylog corruption |
[production] |
| 08:52 |
<midom> |
synchronized php-1.5/includes/GlobalFunctions.php 'removing messages profiling hook' |
[production] |
| 06:43 |
<Tim> |
rebooted db28 with /proc/sysrq-trigger |
[production] |
| 06:29 |
<Tim> |
depooled db28, locked up in kswapd, needs reboot |
[production] |