|
2012-04-30
ยง
|
| 22:28 |
<Ryan_Lane> |
rebooting virt1 |
[production] |
| 22:23 |
<Ryan_Lane> |
bringing down all instances (yay gluster) |
[production] |
| 21:12 |
<pgehres> |
re-enabled Jenkins jobs on Aluminium after db1008 reboot |
[production] |
| 21:11 |
<pgehres> |
CiviCRM back to normal after db1008 reboot |
[production] |
| 21:07 |
<Jeff_Green> |
db1008 gets kernel update and reboot |
[production] |
| 21:00 |
<pgehres> |
put CiviCRM on Aluminium in maintenance mode for db1008 reboot |
[production] |
| 20:59 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2/resources/startup.js 'touch' |
[production] |
| 20:57 |
<pgehres> |
disabled all Jenkins jobs on Aluminium in prep for db1008 reboot |
[production] |
| 20:50 |
<Jeff_Green> |
db1025 and storage3 get new kernels and reboot |
[production] |
| 20:43 |
<notpeter> |
removing ssl4 from pool, stopping puppet on ssl4, adding 3rd udp2log host for testing, restarting nginx |
[production] |
| 20:28 |
<notpeter> |
restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen |
[production] |
| 20:24 |
<Reedy> |
Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers |
[production] |
| 20:18 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory' |
[production] |
| 19:59 |
<notpeter> |
restarting nagios to get rid of some old checks |
[production] |
| 19:57 |
<Jeff_Green> |
payments cluster gets kernel updates and reboots |
[production] |
| 19:55 |
<logmsgbot_> |
reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2 |
[production] |
| 19:49 |
<logmsgbot_> |
reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file' |
[production] |
| 19:49 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php' |
[production] |
| 19:48 |
<paravoid> |
upgraded & rebooted ssl3001, ssl3002, ssl3003 |
[production] |
| 19:46 |
<logmsgbot_> |
reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2 |
[production] |
| 19:30 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions' |
[production] |
| 19:28 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions' |
[production] |
| 19:26 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment' |
[production] |
| 19:18 |
<Reedy> |
Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while... |
[production] |
| 19:14 |
<paravoid> |
rebooting ssl1004 |
[production] |
| 19:06 |
<paravoid> |
rebooting ssl1003 |
[production] |
| 19:00 |
<paravoid> |
rebooting ssl1002 |
[production] |
| 18:59 |
<notpeter> |
starting innobackupex from db1034 to db57 for new s2 slave |
[production] |
| 18:50 |
<paravoid> |
rebooting ssl1001 |
[production] |
| 18:42 |
<Jeff_Green> |
grosley gets new kernel + reboot |
[production] |
| 18:35 |
<Jeff_Green> |
aluminium gets kernel update, yayyyyyyy! |
[production] |
| 18:34 |
<paravoid> |
pooled back ssl1; depooling ssl3 and rebooting |
[production] |
| 18:29 |
<binasher> |
rebooting mw45 for kernel upgrade |
[production] |
| 18:27 |
<Jeff_Green> |
power cycling aluminium which faceplanted |
[production] |
| 18:22 |
<binasher> |
rebooting mw45 |
[production] |
| 18:21 |
<notpeter> |
rebuilding db57 again, this time with more correct raid level! |
[production] |
| 18:19 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights' |
[production] |
| 18:16 |
<paravoid> |
depooled & rebooting ssl1 |
[production] |
| 18:09 |
<logmsgbot_> |
aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes. |
[production] |
| 18:00 |
<logmsgbot_> |
aaron synchronized multiversion |
[production] |
| 17:58 |
<logmsgbot_> |
reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135' |
[production] |
| 17:44 |
<logmsgbot_> |
aaron synchronized wikiversions.cdb |
[production] |
| 17:43 |
<AaronSchulz> |
updating multiversion code |
[production] |
| 08:34 |
<mutante> |
reinstalling srv266 |
[production] |
| 08:08 |
<mutante> |
upgraded mw1,mw2,mw35 |
[production] |
| 07:59 |
<mutante> |
reinstalling srv206 |
[production] |
| 07:50 |
<mutante> |
upgrading mw36 |
[production] |
| 07:37 |
<apergos> |
powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted |
[production] |
| 07:22 |
<mutante> |
installing upgrades on srv212 |
[production] |
| 07:19 |
<apergos> |
reinstalled srv284, seems to be up now |
[production] |