2901-2950 of 10000 results (21ms)
2012-03-25 §
16:14 <RobH> ps1-a1-eqiad firmware updated successfully [production]
16:09 <RobH> updating firmware on ps1-s1-eqiad and ps1-b1-sdtpa [production]
16:07 <RobH> updated firmware successfully on ps1-a8-eqiad, if it has observium alarms now then there are bigger issues. [production]
02:17 <LocalisationUpdate> completed (1.19) at Sun Mar 25 02:17:21 UTC 2012 [production]
00:59 <LeslieCarr> admin down asw-a-eqiad xe-1/1/2 and cr2-eqiad xe-5/0/0 due to framing errors causing packet loss and lacp sporadic timeouts. source of the issue [production]
2012-03-24 §
19:46 <preilly> synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()' [production]
19:46 <preilly> synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'Following a performance regression reported on wikitech-l, added merciless profiling to ExtMobileFrontend::DOMParse()' [production]
17:35 <mark> Migration from br1-knams to cr2-knams completed. [production]
17:09 <mark> Migrated second knams-esams dark fiber link from br1-knams to cr2-knams [production]
16:36 <mark> Corrected MTU setting on cr2-knams's AMS-IX interface [production]
16:20 <Reedy> Some european users reporting oruting issues [production]
16:01 <mark> Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear [production]
15:40 <mark> Brought up AMS-IX ipv4 BGP sessions [production]
15:30 <mark> Brought up AMS-IX ipv6 BGP sessions [production]
15:25 <mark> Moved AMS-IX connection to cr2-knams:xe-1/1/0 [production]
15:22 <mark> Shutdown all AMS-IX BGP sessions [production]
15:06 <mark> Disabled BFD on OSPF3 between cr2-knams and csw1-esams [production]
14:49 <mark> Moved AS6908 and AS1257 PIs to cr2-knams [production]
14:18 <mark> Brought up AS13030 and AS1299 BGP sessions on cr2-knams [production]
13:57 <mark> Shutdown AS1299 BGP session on br1-knams [production]
13:14 <mark> Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity. [production]
12:48 <mark> Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0 [production]
12:44 <mark> Disabled br1-knams:e1/2 (DF leg 1 to esams) [production]
12:43 <mark> Rack mounted and powered up cr2-knams [production]
02:17 <LocalisationUpdate> completed (1.19) at Sat Mar 24 02:17:02 UTC 2012 [production]
2012-03-23 §
23:49 <reedy> synchronized php-1.19/extensions/MoodBar/ '[[rev:114466|r114466]]' [production]
23:15 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia' [production]
23:13 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia' [production]
23:09 <reedy> synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce' [production]
23:08 <reedy> synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce' [production]
23:07 <reedy> synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce' [production]
23:05 <reedy> synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays' [production]
22:24 <RobH> scs-a1-eqiad back online [production]
21:58 <RobH> scs-a8-eqiad coming down for re-grounding [production]
19:51 <RobH> all power strips in eqiad are now properly grounded [production]
18:12 <maplebed> removed ms1 and most of ms2 from the production swift rings. no effect expected. [production]
18:04 <asher> synchronized wmf-config/db.php 'returning db32, pulling db52 for migration' [production]
16:44 <RobH> cp1019 in middle of firmware update, please dont touch [production]
16:44 <RobH> cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day [production]
16:09 <RobH> raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that [production]
15:53 <RobH> magnesium coming back online [production]
15:44 <RobH> shutting down magnesium for disk swap [production]
15:37 <RobH> firmware updating on cp1017, no one touch it please [production]
15:30 <RobH> db1020 can go back into whatever rotation Asher wants it in [production]
15:29 <RobH> db20 memory error on raid controller resolved with firmware updarte [production]
06:39 <tstarling> synchronized php-1.19/includes/filerepo/file/LocalFile.php '[[rev:114442|r114442]]' [production]
02:18 <LocalisationUpdate> completed (1.19) at Fri Mar 23 02:18:35 UTC 2012 [production]
01:55 <mutante> deleting puppet report files older than 60hours on stafford to free disk space [production]
2012-03-22 §
23:30 <reedy> synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [production]
23:18 <RobH> db1020 firmware still updating, will check on it later tonight. offline until then [production]