|
2013-03-05
§
|
| 19:18 |
<K4-713> |
updated payments cluster to 9b063ee2e399a3b |
[production] |
| 18:42 |
<logmsgbot_> |
mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki' |
[production] |
| 18:04 |
<cmjohnson1> |
restarting dhcpd3 service brewster |
[production] |
| 16:55 |
<cmjohnson1> |
mw1085 powering off to troubleshoot DIMM error |
[production] |
| 05:35 |
<ori-l> |
EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1) |
[production] |
| 02:49 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013 |
[production] |
| 02:28 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013 |
[production] |
| 02:03 |
<K4-713> |
updated payments cluster to 5361c795f4bf31980f |
[production] |
| 00:46 |
<mutante> |
fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir |
[production] |
|
2013-03-04
§
|
| 23:12 |
<mutante> |
starting dhcp3 on brewster |
[production] |
| 23:10 |
<LeslieCarr> |
p.s. bonding on foundry sucks |
[production] |
| 23:10 |
<LeslieCarr> |
bringing down virt2 network to bond it |
[production] |
| 22:18 |
<mutante> |
gracefulling eqiad Apaches to push indiawikipedia redirect |
[production] |
| 22:17 |
<logmsgbot_> |
dzahn gracefulled all apaches |
[production] |
| 22:03 |
<logmsgbot_> |
py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves' |
[production] |
| 21:50 |
<LeslieCarr> |
messing around with the network of virt2 - expect possible pmtpa labs network issues |
[production] |
| 21:50 |
<ottomata> |
staring puppet back up on oxygen, packet loss is better after disabling a filter |
[production] |
| 21:30 |
<cmjohnson1> |
authdns update |
[production] |
| 21:17 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11 |
[production] |
| 21:16 |
<logmsgbot_> |
reedy synchronized php-1.21wmf11 |
[production] |
| 21:15 |
<logmsgbot_> |
reedy synchronized php-1.21wmf10 |
[production] |
| 21:14 |
<RobH> |
shutting down professor for a fan swap, should be a short window |
[production] |
| 21:11 |
<cmjohnson1> |
authdns update |
[production] |
| 21:05 |
<ottomata> |
stopping puppet on oxygen to see if we can disable filters and reduce packet loss |
[production] |
| 20:55 |
<Ryan_Lane> |
restarted icinga |
[production] |
| 20:52 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11 |
[production] |
| 20:52 |
<RobH> |
had to restart morebots, but it wasnt really dead, opps |
[production] |
| 20:51 |
<RobH> |
had to restart icinga |
[production] |
| 20:51 |
<RobH> |
Why arent you taking all the !log |
[production] |
| 20:50 |
<RobH> |
i hate you morebots |
[production] |
| 20:28 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11 |
[production] |
| 19:55 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache |
[production] |
| 19:41 |
<logmsgbot_> |
reedy synchronized php-1.21wmf11/ |
[production] |
| 19:34 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 19:34 |
<logmsgbot_> |
reedy synchronized live-1.5/ |
[production] |
| 19:33 |
<logmsgbot_> |
reedy synchronized docroot |
[production] |
| 19:30 |
<hashar> |
on gallium: stopping puppet, cleaning defunct and restarting puppet |
[production] |
| 19:23 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 19:20 |
<mutante> |
installing package upgrades on sockpuppet |
[production] |
| 19:13 |
<sbernardin> |
taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497 |
[production] |
| 18:44 |
<cmjohnson1> |
authdns update |
[production] |
| 18:17 |
<K4-713> |
updated payments cluster to 059969c3d29f685c |
[production] |
| 17:44 |
<cmjohnson1> |
authdns update |
[production] |
| 16:56 |
<logmsgbot_> |
py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error' |
[production] |
| 16:24 |
<sbernardin> |
rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628 |
[production] |
| 16:14 |
<sbernardin> |
db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623 |
[production] |
| 10:12 |
<apergos> |
cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid |
[production] |
| 10:12 |
<apergos> |
reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things |
[production] |
| 02:04 |
<logmsgbot_> |
LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013 |
[production] |