|
2013-03-04
§
|
| 23:12 |
<mutante> |
starting dhcp3 on brewster |
[production] |
| 23:10 |
<LeslieCarr> |
p.s. bonding on foundry sucks |
[production] |
| 23:10 |
<LeslieCarr> |
bringing down virt2 network to bond it |
[production] |
| 22:18 |
<mutante> |
gracefulling eqiad Apaches to push indiawikipedia redirect |
[production] |
| 22:17 |
<logmsgbot_> |
dzahn gracefulled all apaches |
[production] |
| 22:03 |
<logmsgbot_> |
py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves' |
[production] |
| 21:50 |
<LeslieCarr> |
messing around with the network of virt2 - expect possible pmtpa labs network issues |
[production] |
| 21:50 |
<ottomata> |
staring puppet back up on oxygen, packet loss is better after disabling a filter |
[production] |
| 21:30 |
<cmjohnson1> |
authdns update |
[production] |
| 21:17 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11 |
[production] |
| 21:16 |
<logmsgbot_> |
reedy synchronized php-1.21wmf11 |
[production] |
| 21:15 |
<logmsgbot_> |
reedy synchronized php-1.21wmf10 |
[production] |
| 21:14 |
<RobH> |
shutting down professor for a fan swap, should be a short window |
[production] |
| 21:11 |
<cmjohnson1> |
authdns update |
[production] |
| 21:05 |
<ottomata> |
stopping puppet on oxygen to see if we can disable filters and reduce packet loss |
[production] |
| 20:55 |
<Ryan_Lane> |
restarted icinga |
[production] |
| 20:52 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11 |
[production] |
| 20:52 |
<RobH> |
had to restart morebots, but it wasnt really dead, opps |
[production] |
| 20:51 |
<RobH> |
had to restart icinga |
[production] |
| 20:51 |
<RobH> |
Why arent you taking all the !log |
[production] |
| 20:50 |
<RobH> |
i hate you morebots |
[production] |
| 20:28 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11 |
[production] |
| 19:55 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache |
[production] |
| 19:41 |
<logmsgbot_> |
reedy synchronized php-1.21wmf11/ |
[production] |
| 19:34 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 19:34 |
<logmsgbot_> |
reedy synchronized live-1.5/ |
[production] |
| 19:33 |
<logmsgbot_> |
reedy synchronized docroot |
[production] |
| 19:30 |
<hashar> |
on gallium: stopping puppet, cleaning defunct and restarting puppet |
[production] |
| 19:23 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 19:20 |
<mutante> |
installing package upgrades on sockpuppet |
[production] |
| 19:13 |
<sbernardin> |
taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497 |
[production] |
| 18:44 |
<cmjohnson1> |
authdns update |
[production] |
| 18:17 |
<K4-713> |
updated payments cluster to 059969c3d29f685c |
[production] |
| 17:44 |
<cmjohnson1> |
authdns update |
[production] |
| 16:56 |
<logmsgbot_> |
py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error' |
[production] |
| 16:24 |
<sbernardin> |
rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628 |
[production] |
| 16:14 |
<sbernardin> |
db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623 |
[production] |
| 10:12 |
<apergos> |
cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid |
[production] |
| 10:12 |
<apergos> |
reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things |
[production] |
| 02:04 |
<logmsgbot_> |
LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013 |
[production] |