|
2012-10-11
§
|
| 16:49 |
<K4-713> |
updated the payments cluster to 4d0223ebb48f9b |
[production] |
| 16:39 |
<RobH> |
authdns update for pc1001-1003 mgmt ip |
[production] |
| 15:35 |
<RobH> |
working on analytics1007 per rt 3582 |
[production] |
| 14:56 |
<RobH> |
shutting down db1012 to remove evaluation ssd card |
[production] |
| 14:14 |
<paravoid> |
apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise |
[production] |
| 08:59 |
<apergos> |
restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well |
[production] |
| 08:50 |
<apergos> |
restarted ms-fe4 swift-proxy |
[production] |
| 04:13 |
<mutante> |
activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org) |
[production] |
| 04:12 |
<mutante> |
marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated |
[production] |
| 02:27 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012 |
[production] |
| 01:55 |
<LeslieCarr> |
moved traffic back to normal with authdns-scenario |
[production] |
| 01:48 |
<LeslieCarr> |
reenabling nagios notifications |
[production] |
| 01:22 |
<LeslieCarr> |
upgrading csw2-esams member 1 |
[production] |
| 01:22 |
<LeslieCarr> |
upgrading csw2-esams member 0 |
[production] |
| 01:13 |
<binasher> |
disabled nagios notifications |
[production] |
| 01:04 |
<LeslieCarr> |
upgrading csw2-esams member 2 |
[production] |
| 01:02 |
<LeslieCarr> |
upgrading csw2-esams member 3 |
[production] |
| 00:44 |
<binasher> |
restarted pdns on ns2 |
[production] |
| 00:44 |
<LeslieCarr> |
restarting dns on ns2 |
[production] |
| 00:33 |
<LeslieCarr> |
draining esams via authdns-scenarios |
[production] |
| 00:11 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/includes/api/ApiUpload.php |
[production] |
|
2012-10-10
§
|
| 23:02 |
<RobH> |
db1008.mgmt returned to service |
[production] |
| 23:02 |
<RobH> |
msw-a2-eqiad has a bad set of ports, needs replacement per rt 3683 |
[production] |
| 22:46 |
<RobH> |
reboot seems to have restored system (is posting) but drac still not responding, possible bad cable, will test shortly |
[production] |
| 22:40 |
<RobH> |
db1008 being cold reset (removing power cords) |
[production] |
| 22:05 |
<mutante> |
apt-get dist-upgrade on marmontel (blog) |
[production] |
| 21:20 |
<LeslieCarr> |
undraining esams via authdns-scenario normal |
[production] |
| 20:43 |
<LeslieCarr> |
draining esams via authdns-scenario for upgrade |
[production] |
| 20:20 |
<Jeff_Green> |
returning payments to pmtpa |
[production] |
| 20:14 |
<Jeff_Green> |
flipped payments.wikimedia.org to the new eqiad cluster, live testing |
[production] |
| 20:06 |
<logmsgbot_> |
awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/javascripts/common/mf-application.js 'touchfile' |
[production] |
| 18:46 |
<cmjohnson1> |
storage3 power cycling |
[production] |
| 18:24 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedia projects to 1.21wmf1 |
[production] |
| 17:58 |
<logmsgbot_> |
reedy synchronized php-1.20wmf12/cache/interwiki.cdb 'Update interwiki cache' |
[production] |
| 17:58 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/cache/interwiki.cdb 'Update interwiki cache' |
[production] |
| 17:15 |
<notpeter> |
created indices for fdcwiki on searchidx2 and searchidx1001 |
[production] |
| 17:13 |
<logmsgbot_> |
reedy synchronized docroot/fdc/ |
[production] |
| 16:57 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Add meta as import source for fdcwiki' |
[production] |
| 16:43 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Set wgServer for fdcwiki' |
[production] |
| 15:24 |
<logmsgbot_> |
reedy ran sync-common-all |
[production] |
| 14:48 |
<logmsgbot_> |
reedy synchronized s3.dblist |
[production] |
| 14:39 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 14:37 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 12:40 |
<mark> |
Throttled SnapMirror originals transfer to 75 MB/s |
[production] |
| 12:20 |
<mark> |
Increased SnapMirror TCP window size on nas1-a and nas1001-a for the originals snapmirror relationship, and restarted the initial transfer at the last checkpoint |
[production] |
| 06:28 |
<apergos> |
restarted lucene search on search1016 |
[production] |
| 03:01 |
<binasher> |
completed fa_sha1 migration |
[production] |
| 02:41 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf12) at Wed Oct 10 02:41:09 UTC 2012 |
[production] |
| 02:20 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Wed Oct 10 02:20:42 UTC 2012 |
[production] |
| 00:02 |
<binasher> |
started filearchive.fa_sha1 migration, starting with s1 |
[production] |