1-50 of 10000 results (9ms)
2012-09-26 §
20:42 <RobH> mc1011 drac fixed [production]
20:39 <RobH> mc1009, 1010 drac fixed [production]
20:30 <RobH> cp1031 will remain offline until replacement parts arrive tomorrow rt3614 [production]
18:44 <RobH> correction rt 3613 [production]
18:43 <RobH> mc1006 testing done, single bad dimm, support ticket filed rt3614 [production]
18:16 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12 [production]
18:06 <RobH> mc1006 offline for memory troubleshooting per rt 3613 [production]
16:52 <RobH> mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow [production]
16:14 <RobH> carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime [production]
15:57 <Jeff_Green> dist-upgrade & reboot most payments boxes [production]
15:21 <RobH> bast1001 link speed fixed, done working on it [production]
15:19 <RobH> troubleshooting bast1001 link speed per 3414 [production]
15:12 <cmjohnson1> sq37 has fatal error...powering down to replace disk controller card [production]
13:56 <cmjohnson1> ms-be8 powercycling [production]
02:48 <LocalisationUpdate> completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012 [production]
02:23 <LocalisationUpdate> completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012 [production]
01:00 <mutante> reinstalling cp1030 with precise [production]
00:34 <mutante> fixing puppet on singer..finally [production]
2012-09-25 §
23:25 <binasher> ran "varnishadm param.set nuke_limit 300" on all mobile varnish front and back instances to match new default config [production]
23:20 <binasher> repooled cp1044 [production]
23:16 <binasher> depooling cp1044 from lvs for testing [production]
22:44 <binasher> stopping puppet on cp1044, experimenting with varnish lru / nuke params to solve allocation failures [production]
22:28 <mutante> generating locales on singer, add ro UTF-8 locale, run ro.planet [production]
21:50 <Jeff_Green> dropped apparently-unused IP alias on loudon, deprecated from old haproxy install [production]
21:50 <Jeff_Green> apt-update and reboot loudon [production]
21:36 <Jeff_Green> replaced iptables ruleset for loudon [production]
21:02 <notpeter> stopping puppet on brewster [production]
20:50 <mutante> sync-common-file new favicon.ico for wikidata.org [production]
20:49 <dzahn> synchronized docroot/www.wikidata.org/favicon.ico 'updating wikidata favicon per Lydia' [production]
17:57 <Jeff_Green> restarted udp2log on locke to clean up ~20 defunct procs [production]
17:57 <notpeter> temp stopping puppet on brewster [production]
17:27 <mutante> adding otto to ops group in LDAP [production]
14:39 <cmjohnson1> ms-be7 shutting down to check jumpers [production]
14:36 <cmjohnson1> ms-be8 shutting down to check jumpers [production]
12:14 <mark> Powercycled carbon, has been down for 4d [production]
05:58 <Tim> updated Scribunto for [[bugzilla:40480|bug 40480]] [production]
05:45 <paravoid> upgrading all PDF boxes for USN-1581-1/CVE-2012-4405 [production]
05:45 <paravoid> upgrading all imagescalers for USN-1581-1/CVE-2012-4405 [production]
2012-09-24 §
19:44 <RobH> db1012 card install done, system powered down for asher [production]
19:36 <RobH> db1012 shutting down for pci card install per rt3558 [production]
19:14 <reedy> synchronized php-1.20wmf12/extensions/TimedMediaHandler 'Updating to master' [production]
18:54 <pgehres> added new columns to fundraising contribution_tracking table [production]
18:20 <notpeter> temp stopping puppet on brewster [production]
18:15 <hashar> synchronized wmf-config/filebackend-wmflabs.php 'beta: disable commons as a default foreign file repo' [production]
18:02 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf12 [production]
17:03 <cmjohnson1> powering up and toying with ms-be8 for h/w checks [production]
16:35 <RobH> mc1001-1008 memory upgrade complete [production]
16:35 <cmjohnson1> ms-be6 shutting down for h/w testing [production]
15:51 <RobH> mc1001-mc1008 are having memory upgrades installed. hosts already offline. [production]
13:53 <pp-pdf1> restart all services [production]