|
2012-10-11
§
|
| 23:19 |
<mutante> |
manually copy index.html for noc from /common/docroot/noc to /h/w/htdocs/noc/ |
[production] |
| 23:15 |
<logmsgbot_> |
dzahn synchronized docroot/noc/index.html |
[production] |
| 23:13 |
<logmsgbot_> |
dzahn synchronized docroot/noc/index.html |
[production] |
| 23:08 |
<logmsgbot_> |
olivneh synchronized php-1.21wmf1/extensions/E3Experiments |
[production] |
| 22:16 |
<logmsgbot_> |
olivneh synchronized php-1.21wmf1/extensions/E3Experiments |
[production] |
| 21:58 |
<logmsgbot_> |
bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master |
[production] |
| 21:36 |
<logmsgbot_> |
bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master |
[production] |
| 20:48 |
<Jeff_Green> |
flipping payments.wikimedia.org to eqiad cluster to live-test |
[production] |
| 20:12 |
<mutante> |
rebooting analytics1007 |
[production] |
| 19:02 |
<notpeter> |
stopping puppet on brewster |
[production] |
| 18:50 |
<RobH> |
i need to pull virt1001-1003 out of nagios later, as they are now renamed |
[production] |
| 16:52 |
<notpeter> |
restarting pdns on ns0 |
[production] |
| 16:51 |
<logmsgbot_> |
preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page' |
[production] |
| 16:50 |
<notpeter> |
dist-upgrade and reboot on oxygen |
[production] |
| 16:49 |
<K4-713> |
updated the payments cluster to 4d0223ebb48f9b |
[production] |
| 16:39 |
<RobH> |
authdns update for pc1001-1003 mgmt ip |
[production] |
| 15:35 |
<RobH> |
working on analytics1007 per rt 3582 |
[production] |
| 14:56 |
<RobH> |
shutting down db1012 to remove evaluation ssd card |
[production] |
| 14:14 |
<paravoid> |
apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise |
[production] |
| 08:59 |
<apergos> |
restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well |
[production] |
| 08:50 |
<apergos> |
restarted ms-fe4 swift-proxy |
[production] |
| 04:13 |
<mutante> |
activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org) |
[production] |
| 04:12 |
<mutante> |
marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated |
[production] |
| 02:27 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012 |
[production] |
| 01:55 |
<LeslieCarr> |
moved traffic back to normal with authdns-scenario |
[production] |
| 01:48 |
<LeslieCarr> |
reenabling nagios notifications |
[production] |
| 01:22 |
<LeslieCarr> |
upgrading csw2-esams member 1 |
[production] |
| 01:22 |
<LeslieCarr> |
upgrading csw2-esams member 0 |
[production] |
| 01:13 |
<binasher> |
disabled nagios notifications |
[production] |
| 01:04 |
<LeslieCarr> |
upgrading csw2-esams member 2 |
[production] |
| 01:02 |
<LeslieCarr> |
upgrading csw2-esams member 3 |
[production] |
| 00:44 |
<binasher> |
restarted pdns on ns2 |
[production] |
| 00:44 |
<LeslieCarr> |
restarting dns on ns2 |
[production] |
| 00:33 |
<LeslieCarr> |
draining esams via authdns-scenarios |
[production] |
| 00:11 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/includes/api/ApiUpload.php |
[production] |
|
2012-10-10
§
|
| 23:02 |
<RobH> |
db1008.mgmt returned to service |
[production] |
| 23:02 |
<RobH> |
msw-a2-eqiad has a bad set of ports, needs replacement per rt 3683 |
[production] |
| 22:46 |
<RobH> |
reboot seems to have restored system (is posting) but drac still not responding, possible bad cable, will test shortly |
[production] |
| 22:40 |
<RobH> |
db1008 being cold reset (removing power cords) |
[production] |
| 22:05 |
<mutante> |
apt-get dist-upgrade on marmontel (blog) |
[production] |
| 21:20 |
<LeslieCarr> |
undraining esams via authdns-scenario normal |
[production] |
| 20:43 |
<LeslieCarr> |
draining esams via authdns-scenario for upgrade |
[production] |
| 20:20 |
<Jeff_Green> |
returning payments to pmtpa |
[production] |
| 20:14 |
<Jeff_Green> |
flipped payments.wikimedia.org to the new eqiad cluster, live testing |
[production] |
| 20:06 |
<logmsgbot_> |
awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/javascripts/common/mf-application.js 'touchfile' |
[production] |