|
2012-03-08
§
|
| 09:17 |
<mutante> |
running puppet on mw1010 - finished quickly without problems - uh, wonder why Nagios reported puppet freshness then |
[production] |
| 08:22 |
<mutante> |
cp1019 - Hitting F1 to continue reboot ( "Alert! System fatal error during previous boot") |
[production] |
| 08:21 |
<mutante> |
cp1019 went down, then rebooted by itself (i think) after showing "idrac-8W82BP1 Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted" |
[production] |
| 07:54 |
<mutante> |
cadmium fixed by adding groups::wikidev |
[production] |
| 07:41 |
<mutante> |
puppet on cadmium broken due to dependency Group[500] for User[catrope] |
[production] |
| 07:20 |
<mutante> |
ms1004 ran out of disk - caused by 17G HTCPurger.log.1, trying to gzip it now |
[production] |
| 06:52 |
<tstarling> |
synchronized multiversion/MWMultiVersion.php |
[production] |
| 06:51 |
<tstarling> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 03:04 |
<Guest32353> |
powercycled ms-be5; it has been unresponsive for 2 hours. |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Thu Mar 8 02:18:02 UTC 2012 |
[production] |
| 01:32 |
<AaronSchulz> |
fixBug34995.php done |
[production] |
| 01:26 |
<AaronSchulz> |
running fixBug34995 on all wikis |
[production] |
| 00:17 |
<Ryan_Lane> |
adding zero cnames |
[production] |
| 00:16 |
<Ryan_Lane> |
installing newer wikimedia-task-dns-auth on all dns servers |
[production] |
| 00:15 |
<Ryan_Lane> |
added wikimedia-task-dns-auth_0.18 to the repo, to add support for zero |
[production] |
|
2012-03-07
§
|
| 23:05 |
<aaron> |
synchronized php-1.19/includes/filerepo/file/LocalFile.php 'deployed [[rev:113319|r113319]]' |
[production] |
| 22:39 |
<maplebed> |
set swift weight for ms1 to 0 initiating the process to move data off the host in preparation for decomissioning it. |
[production] |
| 21:17 |
<Jeff_Green> |
running apt upgrades and puppetd --test on srv194, srv197, srv203, srv212, srv213, srv230, srv244, srv245, srv252, srv282 and manually restarting nrpe because they're reporting funky in nagios |
[production] |
| 20:20 |
<catrope> |
synchronized php-1.19/resources/startup.js 'touch' |
[production] |
| 20:17 |
<Jeff_Green> |
yet another redirects.conf change, per RT#2498 redirect wikimedia.com-->wikimedia.org |
[production] |
| 20:05 |
<binasher> |
reverted no-pagecache rsync on search nodes - without corresponding index warmup in lsearchd, it just pushes back the pain a bit and does more harm than good |
[production] |
| 20:04 |
<binasher> |
deployed support for zero.wikipedia.org and carrier tagging to mobile varnish servers |
[production] |
| 19:38 |
<catrope> |
synchronized php-1.19/includes/resourceloader/ '[[rev:113278|r113278]]' |
[production] |
| 19:28 |
<Jeff_Green> |
manual apt-upgrade, puppetd --refresh, and repeat on srv265 because it was running on outdated apache config |
[production] |
| 18:44 |
<RobH> |
correction sq39 |
[production] |
| 18:36 |
<RobH> |
pulled sq39 from text pybal config, pulled sq46 from upload pybal config |
[production] |
| 18:36 |
<catrope> |
synchronized php-1.19/resources/startup.js 'touch' |
[production] |
| 18:36 |
<catrope> |
synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch' |
[production] |
| 18:12 |
<RobH> |
shutting down sq38 and sq46 per rt 2581 for testing |
[production] |
| 16:02 |
<cmjohnson1> |
replacing hdd for disk 10 on db22 |
[production] |
| 16:00 |
<cmjohnson1> |
pulling disk 10 from db22 |
[production] |
| 13:28 |
<mark> |
Removed torrus from streber |
[production] |
| 13:00 |
<pp-pdf2> |
updated mwlib to 0.13.6 |
[production] |
| 13:00 |
<pp-pdf3> |
updated mwlib to 0.13.6 |
[production] |
| 13:00 |
<pp-pdf1> |
updated mwlib to 0.13.6 |
[production] |
| 11:29 |
<hashar> |
synchronizing Wikimedia installation... : trigger a rebuild of l10n cache |
[production] |
| 04:53 |
<mutante> |
added ms-be5 drives to swift cluster |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Wed Mar 7 02:18:01 UTC 2012 |
[production] |
| 02:11 |
<catrope> |
synchronized php-1.19/includes/api/ApiBase.php '[[rev:113212|r113212]]' |
[production] |
| 01:58 |
<aaron> |
synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB' |
[production] |
| 00:27 |
<maplebed> |
put ms-be4 into rotation as a new production swift backend storage node |
[production] |
| 00:21 |
<maplebed> |
put ms-be3 into rotation as a new production swift backend storage node |
[production] |
| 00:05 |
<maplebed> |
put ms-be2 into rotation as a new production swift backend storage node |
[production] |