|
2012-05-01
§
|
| 16:27 |
<notpeter> |
starting innobackupex from db1034 to db53 for new s2 slave |
[production] |
| 16:27 |
<notpeter> |
starting innobackupex from db57 to db52 for new s2 slave |
[production] |
| 16:03 |
<notpeter> |
rebuilding db52 and db53 as s2 slaves |
[production] |
| 15:47 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 's1: raising db59,60 weights, pulling db53/54 for reuse' |
[production] |
| 09:23 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'hewiki account creation high throttle limits' |
[production] |
| 04:04 |
<Tim> |
on all apaches, running "chmod -R a+rX /usr/local/apache/common-local/" to clean up after killed rsyncs which left files unreadable |
[production] |
| 02:23 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf2) at Tue May 1 02:23:29 UTC 2012 |
[production] |
| 02:21 |
<logmsgbot_> |
preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileFeedback.php |
[production] |
| 02:14 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf1) at Tue May 1 02:14:06 UTC 2012 |
[production] |
| 02:06 |
<logmsgbot_> |
preilly synchronized php-1.20wmf1/extensions/MobileFrontend/specials/SpecialMobileOptions.php |
[production] |
| 01:51 |
<Ryan_Lane> |
bringing up all labs instances with a 60 second lag |
[production] |
| 01:40 |
<Ryan_Lane> |
rebooting virt0 |
[production] |
| 01:35 |
<Ryan_Lane> |
rebooting virt3 |
[production] |
| 01:33 |
<logmsgbot_> |
preilly synchronized php-1.20wmf1/extensions/MobileFrontend/HtmlFormatter.php |
[production] |
| 01:26 |
<Ryan_Lane> |
rebooting virt5 |
[production] |
| 01:18 |
<Ryan_Lane> |
rebooting virt4 |
[production] |
| 01:03 |
<Ryan_Lane> |
rebooting virt2 |
[production] |
| 00:51 |
<LeslieCarr> |
restarted swift-container-auditor on ms-be5 |
[production] |
| 00:38 |
<logmsgbot_> |
tstarling synchronizing Wikimedia installation... : |
[production] |
| 00:26 |
<Tim> |
removed large syslogs from mw60 and ran sync-common |
[production] |
| 00:18 |
<Tim> |
on mw60 there was an actual directory at /usr/local/apache/common/php where a symlink should have been. fixed |
[production] |
|
2012-04-30
§
|
| 23:58 |
<logmsgbot_> |
aaron synchronized php |
[production] |
| 23:44 |
<RoanKattouw> |
Started Apache back up on mw60 |
[production] |
| 23:39 |
<RoanKattouw> |
Running scap-1 on the Apaches with dsh |
[production] |
| 23:38 |
<RoanKattouw> |
Moved /home/catrope/php-1.19 to /home/wikipedia/lazy-backups/php-1.19 |
[production] |
| 23:38 |
<Reedy> |
mediawiki.org to 1.20wmf2 |
[production] |
| 23:37 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mw.org to 1.20wmf2 |
[production] |
| 23:35 |
<RoanKattouw> |
Strike that, instead moving /home/w/common/php-1.19 to /home/catrope/php-1.19 |
[production] |
| 23:34 |
<RoanKattouw> |
Removing /home/w/common/php-1.19 , NFS might freak out a bit |
[production] |
| 23:31 |
<RoanKattouw> |
Removed php-1.19 from mw60 , synced it, and restarted Apache |
[production] |
| 23:28 |
<RoanKattouw> |
Synced docroot and purged varnish for static-1.20wmf2, bits seems to be working for 1.20wmf2 now |
[production] |
| 23:27 |
<RoanKattouw> |
mw60 has full disk, stopping Apache for now |
[production] |
| 22:50 |
<Ryan_Lane> |
rebooting virt5 |
[production] |
| 22:42 |
<Ryan_Lane> |
rebooting virt3 |
[production] |
| 22:35 |
<Ryan_Lane> |
rebooting virt4 |
[production] |
| 22:28 |
<Ryan_Lane> |
rebooting virt1 |
[production] |
| 22:23 |
<Ryan_Lane> |
bringing down all instances (yay gluster) |
[production] |
| 21:12 |
<pgehres> |
re-enabled Jenkins jobs on Aluminium after db1008 reboot |
[production] |
| 21:11 |
<pgehres> |
CiviCRM back to normal after db1008 reboot |
[production] |
| 21:07 |
<Jeff_Green> |
db1008 gets kernel update and reboot |
[production] |
| 21:00 |
<pgehres> |
put CiviCRM on Aluminium in maintenance mode for db1008 reboot |
[production] |
| 20:59 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2/resources/startup.js 'touch' |
[production] |
| 20:57 |
<pgehres> |
disabled all Jenkins jobs on Aluminium in prep for db1008 reboot |
[production] |
| 20:50 |
<Jeff_Green> |
db1025 and storage3 get new kernels and reboot |
[production] |
| 20:43 |
<notpeter> |
removing ssl4 from pool, stopping puppet on ssl4, adding 3rd udp2log host for testing, restarting nginx |
[production] |
| 20:28 |
<notpeter> |
restarting, once again, innobackupex from db1034 to db57 for new s2 slave after fenari crash killed my screen |
[production] |
| 20:24 |
<Reedy> |
Running ddsh -F30 -cM -g mediawiki-installation -o -oSetupTimeout=10 '/usr/bin/scap-1' in the hope it syncs all the files that would be nice to be on the app servers |
[production] |
| 20:18 |
<logmsgbot_> |
reedy synchronized php-1.20wmf2/cache/ 'Synching whole cache directory' |
[production] |
| 19:59 |
<notpeter> |
restarting nagios to get rid of some old checks |
[production] |
| 19:57 |
<Jeff_Green> |
payments cluster gets kernel updates and reboots |
[production] |