|
2012-09-27
§
|
| 19:10 |
<cmjohnson1> |
search32 going down to swap cpu's (dell request) |
[production] |
| 18:58 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: |
[production] |
| 17:42 |
<cmjohnson1> |
bring db62 down to swap disk controller card per ct/asher |
[production] |
| 17:16 |
<demon> |
synchronized wmf-config/throttle.php 'Syncing I8c44415b: Account creation throttle for ptwikiversity event' |
[production] |
| 16:42 |
<cmjohnson1> |
ms-be6 powering down |
[production] |
| 16:37 |
<RobH> |
cp1031 mainboard replaced per rt3614, will need reinstall |
[production] |
| 15:59 |
<RobH> |
mc1006 memory repaired rt3613 |
[production] |
| 15:54 |
<RobH> |
mc1002 repaired rt3612 |
[production] |
| 15:33 |
<RobH> |
mc1002 (rt3612) & mc1006 (rt3613) offline for memory repair/swap |
[production] |
| 15:21 |
<RobH> |
authdns-update for wikimediafoundation.info |
[production] |
| 15:06 |
<Jeff_Green> |
indium dist-upgrade & reboot |
[production] |
| 13:58 |
<Jeff_Green> |
dist-upgrade & reboot grosley |
[production] |
| 10:55 |
<apergos> |
ignore ms-be6 messages, I'm trying to get into the bleeping lsi raid util |
[production] |
| 04:04 |
<olivneh> |
synchronized php-1.20wmf12/extensions/E3Experiments/experiments/postEditFeedback.js |
[production] |
| 02:36 |
<Jeff_Green> |
powercycle fenari, it's nonresponsive even via drac terminal |
[production] |
|
2012-09-26
§
|
| 20:42 |
<RobH> |
mc1011 drac fixed |
[production] |
| 20:39 |
<RobH> |
mc1009, 1010 drac fixed |
[production] |
| 20:30 |
<RobH> |
cp1031 will remain offline until replacement parts arrive tomorrow rt3614 |
[production] |
| 18:44 |
<RobH> |
correction rt 3613 |
[production] |
| 18:43 |
<RobH> |
mc1006 testing done, single bad dimm, support ticket filed rt3614 |
[production] |
| 18:16 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12 |
[production] |
| 18:06 |
<RobH> |
mc1006 offline for memory troubleshooting per rt 3613 |
[production] |
| 16:52 |
<RobH> |
mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow |
[production] |
| 16:14 |
<RobH> |
carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime |
[production] |
| 15:57 |
<Jeff_Green> |
dist-upgrade & reboot most payments boxes |
[production] |
| 15:21 |
<RobH> |
bast1001 link speed fixed, done working on it |
[production] |
| 15:19 |
<RobH> |
troubleshooting bast1001 link speed per 3414 |
[production] |
| 15:12 |
<cmjohnson1> |
sq37 has fatal error...powering down to replace disk controller card |
[production] |
| 13:56 |
<cmjohnson1> |
ms-be8 powercycling |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012 |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012 |
[production] |
| 01:00 |
<mutante> |
reinstalling cp1030 with precise |
[production] |
| 00:34 |
<mutante> |
fixing puppet on singer..finally |
[production] |
|
2012-09-25
§
|
| 23:25 |
<binasher> |
ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config |
[production] |
| 23:20 |
<binasher> |
repooled cp1044 |
[production] |
| 23:16 |
<binasher> |
depooling cp1044 from lvs for testing |
[production] |
| 22:44 |
<binasher> |
stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures |
[production] |
| 22:28 |
<mutante> |
generating locales on singer, add ro UTF-8 locale, run ro.planet |
[production] |
| 21:50 |
<Jeff_Green> |
dropped apparently-unused IP alias on loudon, deprecated from old haproxy install |
[production] |
| 21:50 |
<Jeff_Green> |
apt-update and reboot loudon |
[production] |
| 21:36 |
<Jeff_Green> |
replaced iptables ruleset for loudon |
[production] |
| 21:02 |
<notpeter> |
stopping puppet on brewster |
[production] |
| 20:50 |
<mutante> |
sync-common-file new favicon.ico for wikidata.org |
[production] |
| 20:49 |
<dzahn> |
synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia' |
[production] |
| 17:57 |
<Jeff_Green> |
restarted udp2log on locke to clean up ~20 defunct procs |
[production] |
| 17:57 |
<notpeter> |
temp stopping puppet on brewster |
[production] |
| 17:27 |
<mutante> |
adding otto to ops group in LDAP |
[production] |
| 14:39 |
<cmjohnson1> |
ms-be7 shutting down to check jumpers |
[production] |
| 14:36 |
<cmjohnson1> |
ms-be8 shutting down to check jumpers |
[production] |
| 12:14 |
<mark> |
Powercycled carbon, has been down for 4d |
[production] |