|
2012-03-24
§
|
| 16:20 |
<Reedy> |
Some european users reporting oruting issues |
[production] |
| 16:01 |
<mark> |
Cleared OSPF session between csw1-esams and csw2-esams which magically made some internal routes reappear |
[production] |
| 15:40 |
<mark> |
Brought up AMS-IX ipv4 BGP sessions |
[production] |
| 15:30 |
<mark> |
Brought up AMS-IX ipv6 BGP sessions |
[production] |
| 15:25 |
<mark> |
Moved AMS-IX connection to cr2-knams:xe-1/1/0 |
[production] |
| 15:22 |
<mark> |
Shutdown all AMS-IX BGP sessions |
[production] |
| 15:06 |
<mark> |
Disabled BFD on OSPF3 between cr2-knams and csw1-esams |
[production] |
| 14:49 |
<mark> |
Moved AS6908 and AS1257 PIs to cr2-knams |
[production] |
| 14:18 |
<mark> |
Brought up AS13030 and AS1299 BGP sessions on cr2-knams |
[production] |
| 13:57 |
<mark> |
Shutdown AS1299 BGP session on br1-knams |
[production] |
| 13:14 |
<mark> |
Established full iBGP mesh with added router cr2-knams. cr2-knams now has full Internet connectivity. |
[production] |
| 12:48 |
<mark> |
Moved fiber from br1-knams:e1/2 to cr2-knams:xe-0/0/0 |
[production] |
| 12:44 |
<mark> |
Disabled br1-knams:e1/2 (DF leg 1 to esams) |
[production] |
| 12:43 |
<mark> |
Rack mounted and powered up cr2-knams |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Sat Mar 24 02:17:02 UTC 2012 |
[production] |
|
2012-03-23
§
|
| 23:49 |
<reedy> |
synchronized php-1.19/extensions/MoodBar/ '[[rev:114466|r114466]]' |
[production] |
| 23:15 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia' |
[production] |
| 23:13 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 34005 - Change uploader flag configuration on Russian Wikipedia' |
[production] |
| 23:09 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce' |
[production] |
| 23:08 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce' |
[production] |
| 23:07 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'wgAutopromoteOnce' |
[production] |
| 23:05 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'wmgAutopromoteOnce empty arrays' |
[production] |
| 22:24 |
<RobH> |
scs-a1-eqiad back online |
[production] |
| 21:58 |
<RobH> |
scs-a8-eqiad coming down for re-grounding |
[production] |
| 19:51 |
<RobH> |
all power strips in eqiad are now properly grounded |
[production] |
| 18:12 |
<maplebed> |
removed ms1 and most of ms2 from the production swift rings. no effect expected. |
[production] |
| 18:04 |
<asher> |
synchronized wmf-config/db.php 'returning db32, pulling db52 for migration' |
[production] |
| 16:44 |
<RobH> |
cp1019 in middle of firmware update, please dont touch |
[production] |
| 16:44 |
<RobH> |
cp1017 memory error seems ot have cleared post firmware update, will keep an eye on it for the rest of the day |
[production] |
| 16:09 |
<RobH> |
raid rebuilding on magnesium, however swift stuff is kind of black box mystery right now to me, need Ben to review magnesium later for that |
[production] |
| 15:53 |
<RobH> |
magnesium coming back online |
[production] |
| 15:44 |
<RobH> |
shutting down magnesium for disk swap |
[production] |
| 15:37 |
<RobH> |
firmware updating on cp1017, no one touch it please |
[production] |
| 15:30 |
<RobH> |
db1020 can go back into whatever rotation Asher wants it in |
[production] |
| 15:29 |
<RobH> |
db20 memory error on raid controller resolved with firmware updarte |
[production] |
| 06:39 |
<tstarling> |
synchronized php-1.19/includes/filerepo/file/LocalFile.php '[[rev:114442|r114442]]' |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Fri Mar 23 02:18:35 UTC 2012 |
[production] |
| 01:55 |
<mutante> |
deleting puppet report files older than 60hours on stafford to free disk space |
[production] |
|
2012-03-22
§
|
| 23:30 |
<reedy> |
synchronized php/cache/interwiki.cdb 'Updating interwiki cache' |
[production] |
| 23:18 |
<RobH> |
db1020 firmware still updating, will check on it later tonight. offline until then |
[production] |
| 22:19 |
<notpeter> |
all 3 dns servers are responding to digs after reload |
[production] |
| 22:10 |
<notpeter> |
pushing a new zone file to add 2 more search-related vips for eqiad |
[production] |
| 20:52 |
<notpeter> |
stopping puppet on brewster temporarily |
[production] |
| 20:25 |
<notpeter> |
rebuilding search1015 and 1016 for disk shuffles |
[production] |
| 20:01 |
<RobH> |
magnesium goign down and up again, troubleshooting the disks |
[production] |
| 19:47 |
<apergos> |
rebooting ms1002, had stuck rsyncs, and kswapds at 100% cpu, weirdness like "ls /export/upload/wikipedia/am/0/00" hanging. |
[production] |
| 18:08 |
<preilly> |
synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header of landing page only for mswiki' |
[production] |
| 15:45 |
<RobH> |
search 1015 and search1016 back up with added disks |
[production] |
| 15:08 |
<RobH> |
shutting down search1015 & search1016 for hdd additions |
[production] |
| 14:45 |
<RobH> |
db1020 still offline, requires firmware update on raid controller per rt 2621, will perform later today |
[production] |