|
2012-10-03
§
|
| 07:20 |
<Tim> |
upgrading luasandbox on all apaches to version 1.1-1 |
[production] |
| 06:28 |
<apergos> |
restarted lucene search on search1016 |
[production] |
| 05:22 |
<logmsgbot_> |
tstarling synchronized php-1.21wmf1/cache/trusted-xff.cdb |
[production] |
| 05:19 |
<logmsgbot_> |
tstarling synchronized php-1.20wmf12/cache/trusted-xff.cdb |
[production] |
| 03:11 |
<logmsgbot_> |
tstarling synchronized wmf-config/db.php 'es10 back up' |
[production] |
| 03:03 |
<Tim> |
on es10: started mysqld |
[production] |
| 02:51 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Wed Oct 3 02:51:35 UTC 2012 |
[production] |
| 02:45 |
<Tim> |
es10 not responding to ssh, rebooting |
[production] |
| 02:43 |
<logmsgbot_> |
tstarling synchronized wmf-config/db.php |
[production] |
| 02:32 |
<logmsgbot_> |
LocalisationUpdate completed (1.20wmf12) at Wed Oct 3 02:31:56 UTC 2012 |
[production] |
| 02:09 |
<logmsgbot_> |
tstarling synchronized wmf-config/InitialiseSettings.php 'API log' |
[production] |
| 01:35 |
<logmsgbot_> |
krinkle synchronized live-1.5 |
[production] |
| 01:29 |
<logmsgbot_> |
tstarling synchronized live-1.5 |
[production] |
| 01:25 |
<Tim> |
on srv247, srv270, srv271, srv286, mw69: killing more stuck apt-get processes, will run apt-get upgrade |
[production] |
| 01:19 |
<Tim> |
on srv233: killed frozen apt-get, running since August 17, running apt-get upgrade |
[production] |
| 01:17 |
<logmsgbot_> |
krinkle Finished syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f |
[production] |
| 01:09 |
<Tim> |
upgrading wikimedia-task-appserver on all apaches |
[production] |
| 01:06 |
<Tim> |
on brewster: updated wikimedia-task-appserver |
[production] |
| 00:43 |
<mutante> |
moved logmsgbot to -operations channel |
[production] |
| 00:34 |
<krinkle> |
Started syncing Wikimedia installation... : Icd721011b40bb8d2c20aefa8b359a3e45413a07f |
[production] |
| 00:25 |
<Reedy> |
Restarted Jenkins |
[production] |
|
2012-10-02
§
|
| 23:45 |
<olivneh> |
synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js 'Updating event format for OpenTask funnel' |
[production] |
| 23:44 |
<reedy> |
synchronized php-1.21wmf1/extensions/Collection/ |
[production] |
| 23:19 |
<Ryan_Lane> |
deploying Damianz's OpenStackManager change for adding/removing bastion users on shell access on labsconsole |
[production] |
| 16:17 |
<asher> |
synchronized wmf-config/db.php 'adding db64 to s3 in place of db25' |
[production] |
| 16:02 |
<mark> |
Reclaimed space on brewster:/ - restarted squdi |
[production] |
| 15:19 |
<mark> |
Sending ALL upload traffic to upload-lb.eqiad (Varnish) |
[production] |
| 14:57 |
<mark> |
Sending traffic from India, Korea, China, Singapore, Thailand, Vietnam to upload-lb.eqiad (Varnish) |
[production] |
| 14:26 |
<mark> |
Sending Mexican upload traffic to upload-lb.eqiad (Varnish) |
[production] |
| 14:12 |
<mark> |
Sending Brazil upload traffic to upload-lb.eqiad (Varnish) |
[production] |
| 14:10 |
<mark> |
Restarted pdns on linne |
[production] |
| 14:03 |
<mark> |
Sending Japanese upload traffic to upload-lb.eqiad (Varnish) |
[production] |
| 13:57 |
<notpeter> |
rebooting srv194-199 and mw55-59 to pick up kernel upgrades, and then re-adding them to apaches pool |
[production] |
| 13:10 |
<notpeter> |
rebooting mw1-16 to pick up kernel upgrades before re-enabling them as (precise) jobrunners |
[production] |
| 13:05 |
<notpeter> |
enabling srv190 in imagescaler pool at half-weight to test precise image-scaling |
[production] |
| 11:52 |
<paravoid> |
apt: include new php5 built with libicu42 (precise-wikimedia/main), include libicu42 |
[production] |
| 11:46 |
<mark> |
Moving US upload traffic to upload-lb.eqiad (Varnish) |
[production] |
| 09:06 |
<paravoid> |
squid deploy all: switching math & timeline to be served from swift |
[production] |
| 05:23 |
<binasher> |
started replicating db64 from the s3 master |
[production] |
| 03:51 |
<olivneh> |
synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js |
[production] |
| 03:29 |
<olivneh> |
synchronized php-1.20wmf12/extensions/E3Experiments/experiments/openTask.js |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.21wmf1) at Tue Oct 2 02:37:57 UTC 2012 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.20wmf12) at Tue Oct 2 02:21:28 UTC 2012 |
[production] |
| 01:56 |
<Tim> |
send sessions to both the existing memcached and also redis, to fill redis and allow for a graceful complete switch later |
[production] |
| 01:55 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 01:41 |
<Tim> |
restarting all apaches to get redis PHP extension |
[production] |
| 01:35 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 01:33 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 01:16 |
<binasher> |
streaming hotbackup of db1019 to db64 (s3) |
[production] |
| 01:12 |
<binasher> |
stopping mysql on db25 and powering down - logging lots of ecc chipkill errors and periodically unresponsive. building db64 as a perm replacement |
[production] |