|
2011-06-24
§
|
| 22:26 |
<notpeter> |
initialized and started repl between nfs1 and 2 for opendj |
[production] |
| 22:08 |
<Ryan_Lane> |
depooling sq69 to test geoiplookup changes for https |
[production] |
| 21:36 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '28529 - Change move-rootuserpages right in Portuguese Wikipedia' |
[production] |
| 21:32 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '28398 - Custom namespaces for ku.wiktionary and ku.wikipedia' |
[production] |
| 21:24 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29527 - Enable AbuseFilter on Catalan Wikipedia' |
[production] |
| 21:23 |
<jeluf> |
synchronized php-1.17/wmf-config/abusefilter.php '29527 - Enable AbuseFilter on Catalan Wikipedia' |
[production] |
| 20:00 |
<andrewS> |
pmtpa: we should have power and the overhead railings to run data cables by late monday, as per Matt. |
[production] |
| 19:20 |
<Ryan_Lane> |
depooled srv301 |
[production] |
| 12:37 |
<Tim-away> |
finished fixing ExtensionDistributor, re-enabled cron job and xinetd |
[production] |
| 07:37 |
<Tim> |
XO appears to be back up, after ~3 hours of downtime |
[production] |
| 07:30 |
<Tim> |
fixing ExtensionDistributor snapshot permissions, was owned by mark instead of extdist |
[production] |
| 06:21 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29168 - Define namespace aliases for Author, Page, Index on all Wikisources' |
[production] |
| 06:10 |
<Tim> |
configured csw5-pmtpa to send its syslogs to 10.0.5.8 instead of 10.0.0.1 |
[production] |
| 00:19 |
<tstarling> |
synchronized php-1.17/includes/db/DatabaseMysql.php 'reapplied r90644, it was fixing an actual problem' |
[production] |
|
2011-06-23
§
|
| 22:36 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 14407. Added Login icon for incubatorwiki' |
[production] |
| 22:33 |
<pdhanda> |
synchronized images/sul/incubatorwiki.png 'Bug 14407. Login icon for incubatorwiki' |
[production] |
| 22:18 |
<pdhanda> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Bug 14407. Added incubatorwiki to CentralAuthAutoLoginWikis' |
[production] |
| 21:30 |
<catrope> |
synchronized php-1.17/maintenance/update.php 'Sync out update.php too, just because Ariel is being paranoid about it :D' |
[production] |
| 21:27 |
<^demon> |
svn up'd /h/w/c/p/m/update.php for nice warning about not running update.php on the cluster |
[production] |
| 21:06 |
<pdhanda> |
ran sync-common-all 'Updating to r90676 of extensions/UploadWizard' |
[production] |
| 21:03 |
<awjrichards> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgNoticeInfrastructure for testwiki per request by Kaldari' |
[production] |
| 20:36 |
<pdhanda> |
updated to r90667 of extensions/Narayam |
[production] |
| 20:33 |
<pdhanda> |
ran sync-common-all |
[production] |
| 18:38 |
<JeLuF> |
#21237 - removed brion's email address from apache config |
[production] |
| 18:10 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '28909 - Change path to Low Saxon Wiktionary logo' |
[production] |
| 18:04 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29391 - Enable e-mail notification for watchlists on br.wikimedia' |
[production] |
| 18:03 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29348 - Email notifications for stewardwiki' |
[production] |
| 18:00 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29360 - Define $wgUploadMissingFileUrl for ptwikibooks' |
[production] |
| 17:02 |
<binasher> |
sync-file r90664 includes/db/LoadMonitor.php |
[production] |
| 17:01 |
<root> |
synchronized php-1.17/includes/db/LoadMonitor.php 'max threads logging' |
[production] |
| 16:47 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'attempting to set max threads on testwiki only' |
[production] |
| 16:32 |
<RoanKattouw> |
Site came back up instantly after Tim disabled max_threads |
[production] |
| 16:28 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'disabled max threads' |
[production] |
| 16:28 |
<catrope> |
synchronized php-1.17/includes/db/LoadBalancer.php 'Make the "LB failure with no last connection" DB error log message more verbose' |
[production] |
| 16:24 |
<binasher> |
upping max threads in db.php |
[production] |
| 16:24 |
<root> |
synchronized php-1.17/wmf-config/db.php 'upping max threads' |
[production] |
| 16:17 |
<laner> |
synchronized php-1.17/includes/db/DatabaseMysql.php 'Roll back Tim's getLag() change from 03:18 today (r90644)' |
[production] |
| 15:58 |
<Ryan_Lane> |
restarting pybal on lvs3 |
[production] |
| 15:52 |
<catrope> |
synchronized php-1.17/includes/db/Database.php 'Roll back Tim's getLag() change from 03:18 today (r90644)' |
[production] |
| 15:50 |
<RoanKattouw> |
Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' |
[production] |
| 15:46 |
<RoanKattouw> |
Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC |
[production] |
| 15:43 |
<RoanKattouw> |
Torrus broken, reporting lots of NaNs |
[production] |
| 15:40 |
<RoanKattouw> |
Strike that, site not completely down. Intermittent service reported in US/Canada |
[production] |
| 15:39 |
<RoanKattouw> |
AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide |
[production] |
| 15:37 |
<RoanKattouw> |
Mark not near a computer, texting Ryan instead |
[production] |
| 15:36 |
<RoanKattouw> |
All wikis seem to be down worldwide. Have texted mark |
[production] |