401-450 of 10000 results (4ms)
2012-04-30 ยง
22:28 <Ryan_Lane> rebooting virt1 [production]
22:23 <Ryan_Lane> bringing down all instances (yay gluster) [production]
21:12 <pgehres> re-enabled Jenkins jobs on Aluminium after db1008 reboot [production]
21:11 <pgehres> CiviCRM back to normal after db1008 reboot [production]
21:07 <Jeff_Green> db1008 gets kernel update and reboot [production]
21:00 <pgehres> put CiviCRM on Aluminium in maintenance mode for db1008 reboot [production]
20:59 <logmsgbot_> reedy synchronized php-1.20wmf2/resources/startup.js 'touch' [production]
20:57 <pgehres> disabled all Jenkins jobs on Aluminium in prep for db1008 reboot [production]
20:50 <Jeff_Green> db1025 and storage3 get new kernels and reboot [production]
20:43 <notpeter> removing ssl4 from pool, stopping puppet on ssl4, adding 3rd udp2log host for testing, restarting nginx [production]
20:28 <notpeter> restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen [production]
20:24 <Reedy> Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers [production]
20:18 <logmsgbot_> reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory' [production]
19:59 <notpeter> restarting nagios to get rid of some old checks [production]
19:57 <Jeff_Green> payments cluster gets kernel updates and reboots [production]
19:55 <logmsgbot_> reedy synchronizing Wikimedia installation... : Rebuiild l10n for 1.20wmf2 [production]
19:49 <logmsgbot_> reedy synchronized wmf-config/ExtensionMessages-1.20wmf2.php 'Syncing file' [production]
19:49 <logmsgbot_> reedy synchronized php-1.20wmf2/LocalSettings.php 'Pushing LocalSettings.php' [production]
19:48 <paravoid> upgraded & rebooted ssl3001, ssl3002, ssl3003 [production]
19:46 <logmsgbot_> reedy synchronizing Wikimedia installation... : Pushing out new symlinks etc, moving test2wiki to 1.20wmf2 [production]
19:30 <logmsgbot_> reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 live hack revisions' [production]
19:28 <logmsgbot_> reedy synchronized php-1.20wmf2 'Syncing 1.20wmf1 live hack revisions' [production]
19:26 <logmsgbot_> reedy synchronized php-1.20wmf2 'Syncing 1.20wmf2 for deployment' [production]
19:18 <Reedy> Syncing php-1.20wmf2 files from NFS to apaches. Likely to upset NFS (or the uplink for the switch nfs is on) for a little while... [production]
19:14 <paravoid> rebooting ssl1004 [production]
19:06 <paravoid> rebooting ssl1003 [production]
19:00 <paravoid> rebooting ssl1002 [production]
18:59 <notpeter> starting innobackupex from db1034 to db57 for new s2 slave [production]
18:50 <paravoid> rebooting ssl1001 [production]
18:42 <Jeff_Green> grosley gets new kernel + reboot [production]
18:35 <Jeff_Green> aluminium gets kernel update, yayyyyyyy! [production]
18:34 <paravoid> pooled back ssl1; depooling ssl3 and rebooting [production]
18:29 <binasher> rebooting mw45 for kernel upgrade [production]
18:27 <Jeff_Green> power cycling aluminium which faceplanted [production]
18:22 <binasher> rebooting mw45 [production]
18:21 <notpeter> rebuilding db57 again, this time with more correct raid level! [production]
18:19 <logmsgbot_> asher synchronized wmf-config/db.php 'adding db59,60 to s1 with low weights' [production]
18:16 <paravoid> depooled & rebooting ssl1 [production]
18:09 <logmsgbot_> aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Sanity run after script changes. [production]
18:00 <logmsgbot_> aaron synchronized multiversion [production]
17:58 <logmsgbot_> reedy synchronized php-1.20wmf1/includes/MagicWord.php 'https://gerrit.wikimedia.org/r/6135' [production]
17:44 <logmsgbot_> aaron synchronized wikiversions.cdb [production]
17:43 <AaronSchulz> updating multiversion code [production]
08:34 <mutante> reinstalling srv266 [production]
08:08 <mutante> upgraded mw1,mw2,mw35 [production]
07:59 <mutante> reinstalling srv206 [production]
07:50 <mutante> upgrading mw36 [production]
07:37 <apergos> powercycling srv266, had this message on mgmt console: Severity: Non Recoverable, SEL:CPU Machine Chk: Processor sensor, transition to non-recoverable was asserted [production]
07:22 <mutante> installing upgrades on srv212 [production]
07:19 <apergos> reinstalled srv284, seems to be up now [production]