|
2012-05-25
§
|
| 22:57 |
<maplebed> |
powercycled kaulen on the mgmt interface |
[production] |
| 19:09 |
<maplebed> |
disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead. |
[production] |
| 18:57 |
<RobH> |
bugzilla appears back online |
[production] |
| 18:57 |
<RobH> |
kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access |
[production] |
| 18:55 |
<RobH> |
kaulen serial console unresponsive, rebooting |
[production] |
| 17:42 |
<paravoid> |
rebooting gurvin & yvon with new kernel |
[production] |
| 17:25 |
<paravoid> |
resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime |
[production] |
| 15:34 |
<mark> |
Power cycled kaulen |
[production] |
| 15:23 |
<hashar> |
kaulen (bugzilla) unreacheable :-( |
[production] |
| 13:56 |
<RobH> |
palladium disk replaced |
[production] |
| 13:52 |
<cmjohnson1> |
replacing ps2 on mw1017 |
[production] |
| 13:50 |
<RobH> |
palladium has a bad disk, goign to replace it |
[production] |
| 13:37 |
<RobH> |
updating drac on search18, shouldnt cause system reboot. |
[production] |
| 10:42 |
<apergos> |
restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?) |
[production] |
|
2012-05-23
§
|
| 21:46 |
<binasher> |
shutting down mysql on db12 in able to restart with binlogging disabled |
[production] |
| 21:45 |
<asher> |
synchronized wmf-config/db.php 'pulling db12 from enwiki, moving watchlist / special queries to db60' |
[production] |
| 21:34 |
<ottomata1> |
upgraded udp-filter to 0.2.4 on oxygen, emery, and locke (with maplebed's help) |
[production] |
| 18:11 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Last 286 wikis over to 1.20wmf3 |
[production] |
| 17:35 |
<maplebed> |
deployed change to varnish configs for preilly; adding more carriers |
[production] |
| 17:13 |
<ben> |
synchronized wmf-config/CommonSettings.php 'changing the URL for the mobile feedback page to the Project namespace' |
[production] |
| 17:12 |
<preilly> |
synchronized php-1.20wmf2/extensions/MobileFrontend/ 'RL hack' |
[production] |
| 17:00 |
<reedy> |
synchronized wmf-config/ 'Tidying unpushed changes' |
[production] |
| 15:35 |
<RobH> |
ns1 died on update, restarting pdns |
[production] |
| 15:34 |
<RobH> |
updated dns for analytics mgmt |
[production] |
| 02:45 |
<LocalisationUpdate> |
completed (1.20wmf2) at Wed May 23 02:45:44 UTC 2012 |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.20wmf3) at Wed May 23 02:23:33 UTC 2012 |
[production] |
| 01:12 |
<K4-713> |
synchronized payments cluster to DonationInterface 4be175e43f |
[production] |
| 00:16 |
<Ryan_Lane> |
flushed the varnish cache for mobile again |
[production] |
| 00:13 |
<preilly> |
synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again' |
[production] |
| 00:11 |
<preilly> |
synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again' |
[production] |
| 00:06 |
<preilly> |
synchronized php-1.20wmf2/extensions/MobileFrontend/ 'try again' |
[production] |
| 00:00 |
<preilly> |
synchronized php-1.20wmf3/extensions/MobileFrontend/ 'try again' |
[production] |
|
2012-05-22
§
|
| 23:49 |
<Ryan_Lane> |
flushed the varnish cache for mobile again |
[production] |
| 23:31 |
<Ryan_Lane> |
flushed the varnish cache for mobile |
[production] |
| 23:21 |
<preilly> |
synchronizing Wikimedia installation... : MobileFrontend Weekly Deployment |
[production] |
| 21:34 |
<RobH> |
updating dns for mgmt of new servers in eqiad |
[production] |
| 20:52 |
<reedy> |
synchronized php-1.20wmf3/extensions/MoodBar/ 'updating to master' |
[production] |
| 20:24 |
<notpeter> |
powering up db1003 |
[production] |
| 20:05 |
<notpeter> |
starting xtrabackup dump from db1004 to db1020 for new eqiad s4 slave |
[production] |
| 19:55 |
<notpeter> |
starting xtrabackup dump from db1033 to db1001 for new eqiad s1 slave |
[production] |
| 19:31 |
<notpeter> |
reimaging db1001 and db1020 |
[production] |
| 19:18 |
<RobH> |
dns update for new servers mgmt ips |
[production] |
| 19:12 |
<reedy> |
synchronizing Wikimedia installation... : Rebuilding message files for interwiki extension |
[production] |