|
2011-02-10
§
|
| 23:10 |
<Ryan_Lane> |
rebooting nova-compute1 and nova-compute2 |
[production] |
| 22:36 |
<mark> |
bits.wikimeda.org on the backend is now only served by a special application server cluster (srv191, srv192), load balanced by Varnish |
[production] |
| 22:06 |
<Ryan_Lane> |
upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe, installing libjpeg62 on nova-compute1 |
[production] |
| 22:05 |
<Ryan_Lane> |
installing libjpeg62 on nova-compute2, as it is a requirement for qemu-common |
[production] |
| 22:04 |
<Ryan_Lane> |
upgrading those packages on nova-compute2 |
[production] |
| 22:03 |
<Ryan_Lane> |
upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe |
[production] |
| 19:58 |
<mark> |
Depooled sq67 for testing |
[production] |
| 19:45 |
<mark> |
Depooled srv192 as normal app server, converted to bits app server (in puppet) |
[production] |
| 19:41 |
<Ryan_Lane> |
deploying a new squid config to all squids |
[production] |
| 19:35 |
<mark> |
Depooled srv191 as normal app server, converted to bits app server (in puppet) |
[production] |
| 19:24 |
<Ryan_Lane> |
restarting squid processes on sq34 |
[production] |
| 19:09 |
<Ryan_Lane> |
restarting gmetad on spence |
[production] |
| 19:07 |
<Ryan_Lane> |
made a new ganglia group for api squids, made sq59 and sq60 aggregators for text squids, made sq31 and sq32 aggregators for api squid |
[production] |
| 19:07 |
<Ryan_Lane> |
moved sq34 and sq35 to api squids |
[production] |
| 19:07 |
<Ryan_Lane> |
depooling sq34, sq35 |
[production] |
| 18:40 |
<Ryan_Lane> |
adding sq34 and sq35 to api squids |
[production] |
| 18:35 |
<Ryan_Lane> |
powering off sq32 - has bad disks |
[production] |
| 18:26 |
<Ryan_Lane> |
powercycling sq32, it's dead, again |
[production] |
| 18:13 |
<Ryan_Lane> |
powercycling sq32 |
[production] |
| 16:41 |
<Ryan_Lane> |
cleaning up logs on ekrem, and restarting apache |
[production] |
| 15:30 |
<^demon> |
made the 1.17.dblist available with the others on noc.wikimedia.org |
[production] |
| 12:22 |
<tstarling> |
synchronized php-1.17/load.php |
[production] |
| 12:20 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'fix wgCacheDirectory' |
[production] |
| 11:57 |
<tstarling> |
synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'profiling fix' |
[production] |
| 11:53 |
<tstarling> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php |
[production] |
| 11:53 |
<tstarling> |
synchronized php-1.17/extensions/Gadgets/Gadgets_body.php 'profiling fix' |
[production] |
| 11:46 |
<tstarling> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?' |
[production] |
| 11:41 |
<tstarling> |
synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?' |
[production] |
| 07:09 |
<tstarling> |
synchronized php-1.17/load.php |
[production] |
| 07:03 |
<root> |
ran sync-common-all |
[production] |
| 06:56 |
<tstarling> |
synchronized php-1.17/StartProfiler.php 'splitting out test2 with 1:1 sample rate for profiling testing' |
[production] |
| 04:34 |
<Ryan_Lane> |
installing scsi-tools on nova-*.tesla |
[production] |
| 04:29 |
<Ryan_Lane> |
adding 50GB of storage to each nova-*.tesla vm |
[production] |
| 04:05 |
<Ryan_Lane> |
installing corosync on nova-*.tesla |
[production] |
| 04:04 |
<Ryan_Lane> |
installing sheepdog on nova-*.tesla |
[production] |
| 03:35 |
<RobH> |
mailing lists restarted |
[production] |
| 03:27 |
<RobH> |
correction, halting mail LIST servers, not all mail servers. |
[production] |
| 03:27 |
<RobH> |
halting mail servers to clean out an email from archives, will resume normal service shortly |
[production] |
| 02:53 |
<Tim> |
restarting all apaches to get apc.ini change |
[production] |
| 02:31 |
<Tim> |
deploying new version of php5-apc package to all apaches, to change a single line in /etc/php5/conf.d/apc.ini |
[production] |
| 00:12 |
<catrope> |
synchronized php-1.17/StartProfiler.php 'Put 1.17 wikis in their own profiling category' |
[production] |
| 00:10 |
<catrope> |
synchronized live-1.5/MWVersion.php 'Define ONE_SEVENTEEN for wikis running 1.17' |
[production] |