|
2012-03-22
§
|
| 22:19 |
<notpeter> |
all 3 dns servers are responding to digs after reload |
[production] |
| 22:10 |
<notpeter> |
pushing a new zone file to add 2 more search-related vips for eqiad |
[production] |
| 20:52 |
<notpeter> |
stopping puppet on brewster temporarily |
[production] |
| 20:25 |
<notpeter> |
rebuilding search1015 and 1016 for disk shuffles |
[production] |
| 20:01 |
<RobH> |
magnesium goign down and up again, troubleshooting the disks |
[production] |
| 19:47 |
<apergos> |
rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging. |
[production] |
| 18:08 |
<preilly> |
synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki' |
[production] |
| 15:45 |
<RobH> |
search 1015 and search1016 back up with added disks |
[production] |
| 15:08 |
<RobH> |
shutting down search1015 & search1016 for hdd additions |
[production] |
| 14:45 |
<RobH> |
db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today |
[production] |
| 14:33 |
<reedy> |
synchronizing Wikimedia installation... : |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Thu Mar 22 02:17:47 UTC 2012 |
[production] |
| 01:14 |
<K4-713> |
Re-enabled the donations queue consumer in Jenkins |
[production] |
| 00:28 |
<binasher> |
started enwiki.revision alter on db32 |
[production] |
| 00:26 |
<binasher> |
disabled lvm snapshots and puppet on db32 for revision sha1 alter |
[production] |
| 00:24 |
<asher> |
synchronized wmf-config/db.php 'pullin db32 for revision alter' |
[production] |
|
2012-03-21
§
|
| 22:27 |
<^demon|away> |
wmf-deployed extensions now r/o in SVN |
[production] |
| 21:52 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php |
[production] |
| 21:27 |
<Ryan_Lane> |
bringing up all instances on virt3 |
[production] |
| 21:08 |
<cmjohnson1> |
swapped 2 DIMMS in virt3 (b2 and b5) |
[production] |
| 21:01 |
<Ryan_Lane> |
shutting down virt3 to replace dimms |
[production] |
| 20:47 |
<^demon> |
/trunk/phase3 is now r/o in SVN |
[production] |
| 20:21 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch' |
[production] |
| 20:10 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki' |
[production] |
| 18:59 |
<maplebed> |
rebooted ms-be3 after it crashed. |
[production] |
| 18:51 |
<binasher> |
brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built |
[production] |
| 18:51 |
<catrope> |
synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update |
[production] |
| 18:46 |
<asher> |
synchronized wmf-config/db.php 'returning db36' |
[production] |
| 18:35 |
<asher> |
synchronized wmf-config/db.php 'pulling db24, failing hw' |
[production] |
| 18:03 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php |
[production] |
| 18:01 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5' |
[production] |
| 17:59 |
<K4-713> |
updated and synchronized payments cluster to [[rev:114382|r114382]] |
[production] |
| 17:47 |
<catrope> |
synchronized php-1.19/resources/startup.js 'touch' |
[production] |
| 12:26 |
<notpeter> |
disabling notifications for search-pool1 |
[production] |
| 08:58 |
<mutante> |
rebooting ms-be4 |
[production] |
| 08:37 |
<mutante> |
stopped/started lsearchd on search9 |
[production] |
| 08:05 |
<mutante> |
ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt |
[production] |
| 07:58 |
<mutante> |
restarted lsearchd on search3 and 9 |
[production] |
| 05:23 |
<tstarling> |
synchronized php-1.19/includes/parser/CoreParserFunctions.php |
[production] |
| 05:23 |
<tstarling> |
synchronized php-1.19/includes/parser/Parser.php |
[production] |
| 05:23 |
<tstarling> |
synchronized php-1.19/includes/parser/StripState.php |
[production] |
| 05:22 |
<tstarling> |
synchronized php-1.19/tests/parser/parserTests.txt |
[production] |
| 03:51 |
<mutante> |
added "lez" to langlist and running authdns-update, for lez.wikipedia per RT-2665 |
[production] |
| 03:29 |
<mutante> |
magnesium - shutting down, has existing RT-2669 to replace disk |
[production] |
| 03:18 |
<mutante> |
magnesium - "..drive on port B of the Srial ATA controller is operating outsde of normal specifications.. Strike F1 key to continue".. |
[production] |
| 03:16 |
<mutante> |
powercycling magnesium - down and just "init: tty4 main" on mgmt, frozen |
[production] |
| 03:10 |
<mutante> |
running puppet on aluminium |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Wed Mar 21 02:18:10 UTC 2012 |
[production] |
| 01:06 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php '[[rev:114342|r114342]]' |
[production] |
| 00:25 |
<preilly> |
synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'fix beta logo' |
[production] |