|
2012-03-23
§
|
| 16:44 |
<RobH> |
cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day |
[production] |
| 16:09 |
<RobH> |
raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that |
[production] |
| 15:53 |
<RobH> |
magnesium coming back online |
[production] |
| 15:44 |
<RobH> |
shutting down magnesium for disk swap |
[production] |
| 15:37 |
<RobH> |
firmware updating on cp1017, no one touch it please |
[production] |
| 15:30 |
<RobH> |
db1020 can go back into whatever rotation Asher wants it in |
[production] |
| 15:29 |
<RobH> |
db20 memory error on raid controller resolved with firmware updarte |
[production] |
| 06:39 |
<tstarling> |
synchronized php-1.19/includes/filerepo/file/LocalFile.php '[[rev:114442|r114442]]' |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Fri Mar 23 02:18:35 UTC 2012 |
[production] |
| 01:55 |
<mutante> |
deleting puppet report files older than 60hours on stafford to free disk space |
[production] |
|
2012-03-22
§
|
| 23:30 |
<reedy> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 23:18 |
<RobH> |
db1020 firmware still updating, will check on it later tonight. offline until then |
[production] |
| 22:19 |
<notpeter> |
all 3 dns servers are responding to digs after reload |
[production] |
| 22:10 |
<notpeter> |
pushing a new zone file to add 2 more search-related vips for eqiad |
[production] |
| 20:52 |
<notpeter> |
stopping puppet on brewster temporarily |
[production] |
| 20:25 |
<notpeter> |
rebuilding search1015 and 1016 for disk shuffles |
[production] |
| 20:01 |
<RobH> |
magnesium goign down and up again, troubleshooting the disks |
[production] |
| 19:47 |
<apergos> |
rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging. |
[production] |
| 18:08 |
<preilly> |
synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki' |
[production] |
| 15:45 |
<RobH> |
search 1015 and search1016 back up with added disks |
[production] |
| 15:08 |
<RobH> |
shutting down search1015 & search1016 for hdd additions |
[production] |
| 14:45 |
<RobH> |
db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today |
[production] |
| 14:33 |
<reedy> |
synchronizing Wikimedia installation... : |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Thu Mar 22 02:17:47 UTC 2012 |
[production] |
| 01:14 |
<K4-713> |
Re-enabled the donations queue consumer in Jenkins |
[production] |
| 00:28 |
<binasher> |
started enwiki.revision alter on db32 |
[production] |
| 00:26 |
<binasher> |
disabled lvm snapshots and puppet on db32 for revision sha1 alter |
[production] |
| 00:24 |
<asher> |
synchronized wmf-config/db.php 'pullin db32 for revision alter' |
[production] |
|
2012-03-21
§
|
| 22:27 |
<^demon|away> |
wmf-deployed extensions now r/o in SVN |
[production] |
| 21:52 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php |
[production] |
| 21:27 |
<Ryan_Lane> |
bringing up all instances on virt3 |
[production] |
| 21:08 |
<cmjohnson1> |
swapped 2 DIMMS in virt3 (b2 and b5) |
[production] |
| 21:01 |
<Ryan_Lane> |
shutting down virt3 to replace dimms |
[production] |
| 20:47 |
<^demon> |
/trunk/phase3 is now r/o in SVN |
[production] |
| 20:21 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable prefswitch' |
[production] |
| 20:10 |
<catrope> |
synchronized wmf-config/CommonSettings.php 'Set $wgArticleFeedbackv5OversightEmails on enwiki' |
[production] |
| 18:59 |
<maplebed> |
rebooted ms-be3 after it crashed. |
[production] |
| 18:51 |
<binasher> |
brought db24 back up after hang, and reslaving, but leaving out of db.php. just replicating until a replacement s2 snapshot host is built |
[production] |
| 18:51 |
<catrope> |
synchronizing Wikimedia installation... : Deploying ArticleFeedbackv5 update |
[production] |
| 18:46 |
<asher> |
synchronized wmf-config/db.php 'returning db36' |
[production] |
| 18:35 |
<asher> |
synchronized wmf-config/db.php 'pulling db24, failing hw' |
[production] |
| 18:03 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFrontend.body.php |
[production] |
| 18:01 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Temporarily disable ShortUrl on testwiki because we think it might conflict with ArticleFeedbackv5' |
[production] |
| 17:59 |
<K4-713> |
updated and synchronized payments cluster to [[rev:114382|r114382]] |
[production] |
| 17:47 |
<catrope> |
synchronized php-1.19/resources/startup.js 'touch' |
[production] |
| 12:26 |
<notpeter> |
disabling notifications for search-pool1 |
[production] |
| 08:58 |
<mutante> |
rebooting ms-be4 |
[production] |
| 08:37 |
<mutante> |
stopped/started lsearchd on search9 |
[production] |
| 08:05 |
<mutante> |
ms-be4 down but cant powercycle it yet..Unable to establish LAN session / ipmitool /ipmi_mgmt |
[production] |
| 07:58 |
<mutante> |
restarted lsearchd on search3 and 9 |
[production] |