|
2011-06-23
§
|
| 15:50 |
<RoanKattouw> |
Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' |
[production] |
| 15:46 |
<RoanKattouw> |
Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC |
[production] |
| 15:43 |
<RoanKattouw> |
Torrus broken, reporting lots of NaNs |
[production] |
| 15:40 |
<RoanKattouw> |
Strike that, site not completely down. Intermittent service reported in US/Canada |
[production] |
| 15:39 |
<RoanKattouw> |
AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide |
[production] |
| 15:37 |
<RoanKattouw> |
Mark not near a computer, texting Ryan instead |
[production] |
| 15:36 |
<RoanKattouw> |
All wikis seem to be down worldwide. Have texted mark |
[production] |
| 09:18 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29215 - Add import source for dearbcomwiki' |
[production] |
| 09:14 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29380 - Change logo of Kannada wiktionary (kn.wiktionary.org)' |
[production] |
| 08:48 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29515 - install Narayam in all Sanskrit wiki projects' |
[production] |
| 08:42 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29474 - Search Index and Author namespaces by default on all Wikisources which don't have a different configuration' |
[production] |
| 08:27 |
<jeluf> |
synchronized php-1.17/wmf-config/InitialiseSettings.php '29172 - Add Page, Author and Index namespaces to $wgContentNamespaces on all Wikisources' |
[production] |
| 07:53 |
<JeLuF> |
fixed broken ganglia installation on project2 (reported by puppetmon) |
[production] |
| 07:51 |
<JeLuF> |
added new class mobile::disabled in puppet that has apache disabled, assigned mobile1 to it. |
[production] |
| 07:19 |
<tstarling> |
synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90654' |
[production] |
| 07:12 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'enabling SubPageList3 on testwiki' |
[production] |
| 07:11 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 07:07 |
<root> |
synchronizing Wikimedia installation... Revision: 90493: |
[production] |
| 07:05 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 90493: |
[production] |
| 07:04 |
<Tim> |
running scap to push out SubPageList3 source directory (not enabled) |
[production] |
| 06:36 |
<Tim> |
killed puppet-monitor on fenari, I'm pretty sure that's the thing that was spamming #wikimedia-tech every 3 minutes |
[production] |
| 05:34 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled mysql parser cache on testwiki' |
[production] |
| 05:33 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'mysql parser cache pre-deploy changes' |
[production] |
| 05:32 |
<Tim> |
created "parsercache" database on db40 and added suitable wikiuser/wikiadmin grants |
[production] |
| 04:46 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'put db33 back' |
[production] |
| 04:46 |
<Tim> |
restarted the parser cache hit rate monitor |
[production] |
| 03:20 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'restored max lag to 30s, now that r90644 is deployed it shouldn't break anything' |
[production] |
| 03:18 |
<tstarling> |
synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' |
[production] |
| 03:03 |
<Tim> |
started ntpd on db36, not sure why it was stopped, it has an init script. Clock skew was 1.7 seconds. |
[production] |
| 02:55 |
<tstarling> |
synchronized php-1.17/wmf-config/db.php 'increasing max lag to 3000' |
[production] |
| 00:43 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'removed db33 for TS dump' |
[production] |
|
2011-06-22
§
|
| 23:08 |
<pdhanda> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled extension NewUserMessage on rowikisource. Bug 28307.' |
[production] |
| 22:47 |
<Ryan_Lane> |
repooling sq69 |
[production] |
| 22:07 |
<Ryan_Lane> |
depooling sq69 to test vcl changes for https geoiplookup |
[production] |
| 21:23 |
<notpeter> |
resized db10:/a to 220G |
[production] |
| 19:10 |
<Reedy> |
Wikimedia repo post commit hook fixed for CR |
[production] |
| 18:43 |
<Reedy> |
Wikimedia repo set up as a CodeReview repo on mw.org |
[production] |
| 18:04 |
<JeLuF> |
fixed puppetmon, log format has changed |
[production] |
| 13:29 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'put db21 back' |
[production] |
| 10:41 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'put db34 back' |
[production] |
| 02:14 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'remove db34 to dump for TS' |
[production] |
| 02:08 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'remove db21 to dump s6 for TS' |
[production] |
| 01:59 |
<kate> |
synchronized php-1.17/wmf-config/db.php 'put db18 back' |
[production] |