|
2011-06-23
§
|
| 18:03 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29348 - Email notifications for stewardwiki' |
[production] |
| 18:00 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29360 - Define $wgUploadMissingFileUrl for ptwikibooks' |
[production] |
| 17:02 |
<binasher> |
sync-file r90664 includes/db/LoadMonitor.php |
[production] |
| 17:01 |
<root> |
synchronized php-1.17/includes/db/LoadMonitor.php 'max threads logging' |
[production] |
| 16:47 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'attempting to set max threads on testwiki only' |
[production] |
| 16:32 |
<RoanKattouw> |
Site came back up instantly after Tim disabled max_threads |
[production] |
| 16:28 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'disabled max threads' |
[production] |
| 16:28 |
<catrope> |
synchronized php-1.17/includes/db/LoadBalancer.php 'Make the "LB failure with no last connection" DB error log message more verbose' |
[production] |
| 16:24 |
<binasher> |
upping max threads in db.php |
[production] |
| 16:24 |
<root> |
synchronized php-1.17/wmf-config/db.php 'upping max threads' |
[production] |
| 16:17 |
<laner> |
synchronized php-1.17/includes/db/DatabaseMysql.php 'Roll back Tim's getLag() change from 03:18 today (r90644)' |
[production] |
| 15:58 |
<Ryan_Lane> |
restarting pybal on lvs3 |
[production] |
| 15:52 |
<catrope> |
synchronized php-1.17/includes/db/Database.php 'Roll back Tim's getLag() change from 03:18 today (r90644)' |
[production] |
| 15:50 |
<RoanKattouw> |
Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' |
[production] |
| 15:46 |
<RoanKattouw> |
Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC |
[production] |
| 15:43 |
<RoanKattouw> |
Torrus broken, reporting lots of NaNs |
[production] |
| 15:40 |
<RoanKattouw> |
Strike that, site not completely down. Intermittent service reported in US/Canada |
[production] |
| 15:39 |
<RoanKattouw> |
AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide |
[production] |
| 15:37 |
<RoanKattouw> |
Mark not near a computer, texting Ryan instead |
[production] |
| 15:36 |
<RoanKattouw> |
All wikis seem to be down worldwide. Have texted mark |
[production] |
| 09:18 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29215 - Add import source for dearbcomwiki' |
[production] |
| 09:14 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29380 - Change logo of Kannada wiktionary (kn.wiktionary.org)' |
[production] |
| 08:48 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29515 - install Narayam in all Sanskrit wiki projects' |
[production] |
| 08:42 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29474 - Search Index and Author namespaces by default on all Wikisources which don't have a different configuration' |
[production] |
| 08:27 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29172 - Add Page, Author and Index namespaces to $wgContentNamespaces on all Wikisources' |
[production] |
| 07:53 |
<JeLuF> |
fixed broken ganglia installation on project2 (reported by puppetmon) |
[production] |
| 07:51 |
<JeLuF> |
added new class mobile::disabled in puppet that has apache disabled, assigned mobile1 to it. |
[production] |
| 07:19 |
<tstarling> |
synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90654' |
[production] |
| 07:12 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'enabling SubPageList3 on testwiki' |
[production] |
| 07:11 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 07:07 |
<root> |
synchronizing Wikimedia installation... Revision: 90493: |
[production] |
| 07:05 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 90493: |
[production] |
| 07:04 |
<Tim> |
running scap to push out SubPageList3 source directory (not enabled) |
[production] |
| 06:36 |
<Tim> |
killed puppet-monitor on fenari, I'm pretty sure that's the thing that was spamming #wikimedia-tech every 3 minutes |
[production] |
| 05:34 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled mysql parser cache on testwiki' |
[production] |
| 05:33 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'mysql parser cache pre-deploy changes' |
[production] |
| 05:32 |
<Tim> |
created "parsercache" database on db40 and added suitable wikiuser/wikiadmin grants |
[production] |
| 04:46 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'put db33 back' |
[production] |
| 04:46 |
<Tim> |
restarted the parser cache hit rate monitor |
[production] |
| 03:20 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'restored max lag to 30s, now that r90644 is deployed it shouldn't break anything' |
[production] |
| 03:18 |
<tstarling> |
synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' |
[production] |
| 03:03 |
<Tim> |
started ntpd on db36, not sure why it was stopped, it has an init script. Clock skew was 1.7 seconds. |
[production] |
| 02:55 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'increasing max lag to 3000' |
[production] |
| 00:43 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'removed db33 for TS dump' |
[production] |