|
2012-02-11
§
|
| 20:31 |
<apergos> |
restarted lightty on dataset2 |
[production] |
| 17:28 |
<RobH> |
manual test of each affected service complete, db9 fully online. |
[production] |
| 17:26 |
<RobH> |
db9 moved, all systems online |
[production] |
| 17:08 |
<RobH> |
db9 shutting down to move racks, offline during this includes: blogs, bugzilla, racktables, rt, survey, etherpad, observium |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.18) at Sat Feb 11 02:18:36 UTC 2012 |
[production] |
| 00:17 |
<reedy> |
synchronizing Wikimedia installation... : |
[production] |
|
2012-02-10
§
|
| 22:17 |
<LeslieCarr> |
fixing the labs apache2 puppet groups |
[production] |
| 21:48 |
<RobH> |
memory in cp1017 wasnt properly seated as far as i can tell, if it doesnt mess up again it should be ok. |
[production] |
| 21:41 |
<RobH> |
cp1017 being tested for bad memory |
[production] |
| 21:36 |
<RobH> |
powercycling msw-a2-eqiad resolves all mgmt issues in rack |
[production] |
| 21:34 |
<RobH> |
powercycling msw-a1-eqiad. |
[production] |
| 21:29 |
<RobH> |
db1001 rebooting, locked up |
[production] |
| 20:53 |
<RobH> |
updating dns for new db hosts |
[production] |
| 19:59 |
<Reedy> |
Checking out 1.19wmf1 to /tmp on fenari |
[production] |
| 19:12 |
<RobH> |
oxygen setup and installed per rt2343, still needs puppet runs and full deployment per rt 2430 |
[production] |
| 17:58 |
<RobH> |
updating dns for oxygen internal ip |
[production] |
| 17:21 |
<mutante> |
labs logging is broken |
[production] |
| 17:14 |
<RobH> |
oxygen offline for hard disk upgrade to replace locke |
[production] |
| 16:50 |
<mutante> |
running sync-apache, trying to redirect office.wm to https |
[production] |
| 16:07 |
<mark> |
Rebalanced appserver load balancing by giving the new mw* pmtpa app servers weight 150 in the pybal server list |
[production] |
| 15:17 |
<mark> |
Turned on KeepAlive on apaches for better miss service times from eqiad |
[production] |
| 13:42 |
<mark> |
Configured cp1001 and cp1020 to contact backend servers directly instead of via pmtpa squids |
[production] |
| 12:02 |
<mark> |
Decommissioning sq38, sq46 and sq47 in squid configurator |
[production] |
| 11:50 |
<mark> |
Making cp1001-1005 API squids |
[production] |
| 05:08 |
<maplebed> |
deployed squid config to uploads to send 100% of thumbnail traffic to swift |
[production] |
| 02:49 |
<maplebed> |
deploying fix for & bug with swift (files with an & in the name wouldn't load properly) |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.18) at Fri Feb 10 02:18:37 UTC 2012 |
[production] |
| 00:22 |
<LeslieCarr> |
increased nagios max concurrent checks on spence and lowered the interval between processing them |
[production] |
| 00:20 |
<maplebed> |
deployed squid config to upload squids rolling thumbnails back to 75% handled by swift to test the & bug |
[production] |
|
2012-02-09
§
|
| 22:22 |
<Jeff_Green> |
adding community-analytics.wikimedia.org to DNS |
[production] |
| 21:36 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths' |
[production] |
| 20:04 |
<binasher> |
started 1.19 schema migrations on seconadry db's (mwscript upgrade-1.19wmf1-1.php --secondary) |
[production] |
| 19:32 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths' |
[production] |
| 17:31 |
<maplebed> |
deployed squid config to upload squids sending 100% of all thumbnail traffic to swift |
[production] |
| 16:31 |
<mark> |
Sending ALL non-european wikipedia traffic to eqiad text squids |
[production] |
| 16:17 |
<mark> |
Added Brazil traffic to eqiad text squids |
[production] |
| 15:49 |
<mark> |
Sending some Asian wikipedia traffic to wikipedia-lb.eqiad |
[production] |
| 15:13 |
<mark> |
Sending Canadian wikipedia traffic to wikipedia-lb.eqiad |
[production] |
| 14:27 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 34183 - add alias redirection in ml wikisource' |
[production] |
| 14:23 |
<mark> |
Redirecting {wikiquote,wikisource,wikiversity,wiktionary}-lb.pmtpa traffic to .eqiad (geodns) |
[production] |
| 14:17 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 34253 - Change site logo for lb.wiktionary' |
[production] |
| 14:11 |
<mark> |
Redirecting {foundation,wikibooks,wikinews}-lb.pmtpa traffic to .eqiad (geodns) |
[production] |
| 14:10 |
<hashar> |
synchronized docroot/mediawiki/xml 'update html pages for 0.1 and 0.2 schemes' |
[production] |
| 14:06 |
<hashar> |
synchronized docroot/mediawiki/xml/index.html 'Give a home page to our XML namespaces homepage' |
[production] |
| 13:52 |
<hashar> |
synchronized docroot/mediawiki/xml |
[production] |
| 13:45 |
<mark> |
Redirecting wikimedia-lb.pmtpa traffic to wikimedia-lb.eqiad (geodns) |
[production] |
| 13:28 |
<mark> |
Redirecting mediawiki-lb.pmtpa traffic to mediawiki-lb.eqiad (geodns) |
[production] |
| 12:49 |
<apergos> |
around 11:05 UTC, increased /proc/sys/vm/min_free_kbytes from 16259 to 262144 to check impact on network alloc issues, on dataset1001 |
[production] |
| 11:56 |
<mark_> |
Power cycled cp1017 |
[production] |
| 11:50 |
<mark_> |
Changed topology of eqiad text squids to request from pmtpa (similar to esams) |
[production] |