|
2012-09-26
§
|
| 20:42 |
<RobH> |
mc1011 drac fixed |
[production] |
| 20:39 |
<RobH> |
mc1009, 1010 drac fixed |
[production] |
| 20:30 |
<RobH> |
cp1031 will remain offline until replacement parts arrive tomorrow rt3614 |
[production] |
| 18:44 |
<RobH> |
correction rt 3613 |
[production] |
| 18:43 |
<RobH> |
mc1006 testing done, single bad dimm, support ticket filed rt3614 |
[production] |
| 18:16 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12 |
[production] |
| 18:06 |
<RobH> |
mc1006 offline for memory troubleshooting per rt 3613 |
[production] |
| 16:52 |
<RobH> |
mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow |
[production] |
| 16:14 |
<RobH> |
carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime |
[production] |
| 15:57 |
<Jeff_Green> |
dist-upgrade & reboot most payments boxes |
[production] |
| 15:21 |
<RobH> |
bast1001 link speed fixed, done working on it |
[production] |
| 15:19 |
<RobH> |
troubleshooting bast1001 link speed per 3414 |
[production] |
| 15:12 |
<cmjohnson1> |
sq37 has fatal error...powering down to replace disk controller card |
[production] |
| 13:56 |
<cmjohnson1> |
ms-be8 powercycling |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012 |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012 |
[production] |
| 01:00 |
<mutante> |
reinstalling cp1030 with precise |
[production] |
| 00:34 |
<mutante> |
fixing puppet on singer..finally |
[production] |
|
2012-09-25
§
|
| 23:25 |
<binasher> |
ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config |
[production] |
| 23:20 |
<binasher> |
repooled cp1044 |
[production] |
| 23:16 |
<binasher> |
depooling cp1044 from lvs for testing |
[production] |
| 22:44 |
<binasher> |
stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures |
[production] |
| 22:28 |
<mutante> |
generating locales on singer, add ro UTF-8 locale, run ro.planet |
[production] |
| 21:50 |
<Jeff_Green> |
dropped apparently-unused IP alias on loudon, deprecated from old haproxy install |
[production] |
| 21:50 |
<Jeff_Green> |
apt-update and reboot loudon |
[production] |
| 21:36 |
<Jeff_Green> |
replaced iptables ruleset for loudon |
[production] |
| 21:02 |
<notpeter> |
stopping puppet on brewster |
[production] |
| 20:50 |
<mutante> |
sync-common-file new favicon.ico for wikidata.org |
[production] |
| 20:49 |
<dzahn> |
synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia' |
[production] |
| 17:57 |
<Jeff_Green> |
restarted udp2log on locke to clean up ~20 defunct procs |
[production] |
| 17:57 |
<notpeter> |
temp stopping puppet on brewster |
[production] |
| 17:27 |
<mutante> |
adding otto to ops group in LDAP |
[production] |
| 14:39 |
<cmjohnson1> |
ms-be7 shutting down to check jumpers |
[production] |
| 14:36 |
<cmjohnson1> |
ms-be8 shutting down to check jumpers |
[production] |
| 12:14 |
<mark> |
Powercycled carbon, has been down for 4d |
[production] |
| 05:58 |
<Tim> |
updated Scribunto for [[bugzilla:40480|bug 40480]] |
[production] |
| 05:45 |
<paravoid> |
upgrading all PDF boxes for USN-1581-1/CVE-2012-4405 |
[production] |
| 05:45 |
<paravoid> |
upgrading all imagescalers for USN-1581-1/CVE-2012-4405 |
[production] |
|
2012-09-24
§
|
| 19:44 |
<RobH> |
db1012 card install done, system powered down for asher |
[production] |
| 19:36 |
<RobH> |
db1012 shutting down for pci card install per rt3558 |
[production] |
| 19:14 |
<reedy> |
synchronized php-1.20wmf12/extensions/TimedMediaHandler 'Updating to master' |
[production] |
| 18:54 |
<pgehres> |
added new columns to fundraising contribution_tracking table |
[production] |
| 18:20 |
<notpeter> |
temp stopping puppet on brewster |
[production] |
| 18:15 |
<hashar> |
synchronized wmf-config/filebackend-wmflabs.php 'beta: disable commons as a default foreign file repo' |
[production] |
| 18:02 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf12 |
[production] |
| 17:03 |
<cmjohnson1> |
powering up and toying with ms-be8 for h/w checks |
[production] |
| 16:35 |
<RobH> |
mc1001-1008 memory upgrade complete |
[production] |
| 16:35 |
<cmjohnson1> |
ms-be6 shutting down for h/w testing |
[production] |
| 15:51 |
<RobH> |
mc1001-mc1008 are having memory upgrades installed. hosts already offline. |
[production] |
| 13:53 |
<pp-pdf1> |
restart all services |
[production] |