|
2012-10-12
§
|
| 18:19 |
<cmjohnson1> |
troubleshooting db42 w/IBM tech |
[production] |
| 17:46 |
<cmjohnson1> |
pc3 shutting down to replace hdd's with ssd's rt3703 |
[production] |
| 17:29 |
<cmjohnson1> |
pc2 shutting down to replaced hdd w/ ssd rt3703 |
[production] |
| 16:27 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'Fix extdist mount related warnings' |
[production] |
| 16:14 |
<cmjohnson1> |
running installer on storage3 |
[production] |
| 15:54 |
<paravoid> |
moving bits from esams to pmtpa (authdns normal scenario) |
[production] |
| 15:52 |
<cmjohnson1> |
storage 3 replacing disk 12 and rebuilding raid |
[production] |
| 15:32 |
<mark> |
Disabled csw1-esams:e8/2 |
[production] |
| 14:37 |
<mark> |
Rebooting cr1-esams |
[production] |
| 14:17 |
<mark> |
Upgrading JunOS on cr1-esams to 11.4R5.5, preparing for reboot |
[production] |
| 13:59 |
<mark> |
Activated aggregate generation on cr1-esams |
[production] |
| 13:49 |
<mark> |
Moved AS38930 transit from csw1-esams:e8/4 to cr1-esams:xe-1/3/0 |
[production] |
| 13:24 |
<cmjohnson1> |
storage3 powering down |
[production] |
| 12:27 |
<mark> |
Upgraded new stack member asw-oe11-esams to junos 11.4R5.5 and rebooted it |
[production] |
| 12:16 |
<mark> |
Added new EX4200-48T asw-oe11-esams to csw2-esams stack |
[production] |
| 11:08 |
<mark> |
Disabled csw2-oe12-esams:vcp-1 |
[production] |
| 02:01 |
<logmsgbot_> |
LocalisationUpdate failed: git pull of extensions failed |
[production] |
| 01:21 |
<mutante> |
moving ancient copy of bugzilla out of /h/w/htdocs (to /root/backup on fenari temp. just in case) |
[production] |
| 01:15 |
<mutante> |
deleting old document root dirs on fenari. /home/wikipedia/htdocs/noc and noc.old (use /h/w/common/docroot/noc and mediawiki-config repo) |
[production] |
| 01:08 |
<mutante> |
restarting Apache on fenari for noc.wm change |
[production] |
| 00:42 |
<Jeff_Green> |
switched payments back to pmtpa and increased TTL to 5 min |
[production] |
| 00:37 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'And back to master' |
[production] |
| 00:31 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Revert due to error' |
[production] |
| 00:29 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection 'Update collection to master' |
[production] |
| 00:09 |
<cmjohnson1> |
storage3 being reimaged |
[production] |
|
2012-10-11
§
|
| 23:19 |
<mutante> |
manually copy index.html for noc from /common/docroot/noc to /h/w/htdocs/noc/ |
[production] |
| 23:15 |
<logmsgbot_> |
dzahn synchronized docroot/noc/index.html |
[production] |
| 23:13 |
<logmsgbot_> |
dzahn synchronized docroot/noc/index.html |
[production] |
| 23:08 |
<logmsgbot_> |
olivneh synchronized php-1.21wmf1/extensions/E3Experiments |
[production] |
| 22:16 |
<logmsgbot_> |
olivneh synchronized php-1.21wmf1/extensions/E3Experiments |
[production] |
| 21:58 |
<logmsgbot_> |
bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master |
[production] |
| 21:36 |
<logmsgbot_> |
bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, MoodBar, PageTriage to master |
[production] |
| 20:48 |
<Jeff_Green> |
flipping payments.wikimedia.org to eqiad cluster to live-test |
[production] |
| 20:12 |
<mutante> |
rebooting analytics1007 |
[production] |
| 19:02 |
<notpeter> |
stopping puppet on brewster |
[production] |
| 18:50 |
<RobH> |
i need to pull virt1001-1003 out of nagios later, as they are now renamed |
[production] |
| 16:52 |
<notpeter> |
restarting pdns on ns0 |
[production] |
| 16:51 |
<logmsgbot_> |
preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page' |
[production] |
| 16:50 |
<notpeter> |
dist-upgrade and reboot on oxygen |
[production] |
| 16:49 |
<K4-713> |
updated the payments cluster to 4d0223ebb48f9b |
[production] |
| 16:39 |
<RobH> |
authdns update for pc1001-1003 mgmt ip |
[production] |
| 15:35 |
<RobH> |
working on analytics1007 per rt 3582 |
[production] |
| 14:56 |
<RobH> |
shutting down db1012 to remove evaluation ssd card |
[production] |
| 14:14 |
<paravoid> |
apt: including php-parsekit, python-async, python-git, python-gitdb, python-smmap backports for precise |
[production] |
| 08:59 |
<apergos> |
restarted swift proxy on ms-fe3 and ms-fe2 with some delay between. note I had to shoot stale processes from Sept on all these boxes as well |
[production] |
| 08:50 |
<apergos> |
restarted ms-fe4 swift-proxy |
[production] |
| 04:13 |
<mutante> |
activated W3 Total Cache performance plugin in Wordpress (blog.wikimedia.org) |
[production] |
| 04:12 |
<mutante> |
marmontel (blog): fixed some file permissions in ./blog/wp-content for www-data the wordpress caching plugin needs to be re-activated |
[production] |
| 02:27 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Thu Oct 11 02:27:51 UTC 2012 |
[production] |
| 01:55 |
<LeslieCarr> |
moved traffic back to normal with authdns-scenario |
[production] |