|
2012-05-26
§
|
| 19:55 |
<apergos> |
restarting apache2 on kaulen |
[production] |
| 19:34 |
<Nemo_bis> |
bugzilla down again |
[production] |
| 18:14 |
<beta-logmsgbot> |
hashar: killed webtranscode job on commons |
[releng] |
| 14:19 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Settting wgBlacklistSettings' |
[production] |
| 11:05 |
<apergos> |
stopping and restarting apache on kaulen blah blah blah |
[production] |
| 09:12 |
<Reedy> |
Bugzilla is down, Kaulen looks to be in swap death again |
[production] |
| 06:46 |
<apergos> |
powercycling kaulen |
[production] |
| 05:59 |
<beta-logmsgbot> |
hashar: Edited squid.conf to limit memory to 1G and restarted squid |
[releng] |
| 05:53 |
<hashar> |
kaulen dead :-[ |
[production] |
| 05:47 |
<hashar> |
Bugzilla on Kaulen being super slow again |
[production] |
| 05:42 |
<beta-logmsgbot> |
hashar: squid was killed by linux OOM! |
[releng] |
| 02:22 |
<LocalisationUpdate> |
completed (1.20wmf3) at Sat May 26 02:22:37 UTC 2012 |
[production] |
| 00:40 |
<K4-713> |
synchronized payments cluster to DonationInterface 04809f1cf0d |
[production] |
|
2012-05-25
§
|
| 22:57 |
<maplebed> |
powercycled kaulen on the mgmt interface |
[production] |
| 19:09 |
<maplebed> |
disabled the outdated /etc/init.d/gmond on spence. use ganglia-monitor instead. |
[production] |
| 18:57 |
<RobH> |
bugzilla appears back online |
[production] |
| 18:57 |
<RobH> |
kaulen is rebooted, it may have had a runaway process or a memory leak, not sure yet, but it was locked up from access |
[production] |
| 18:55 |
<RobH> |
kaulen serial console unresponsive, rebooting |
[production] |
| 17:42 |
<paravoid> |
rebooting gurvin & yvon with new kernel |
[production] |
| 17:25 |
<paravoid> |
resetting gurvin, load spiking at 370+, SSH unreachable, 214 days of uptime |
[production] |
| 15:34 |
<mark> |
Power cycled kaulen |
[production] |
| 15:23 |
<hashar> |
kaulen (bugzilla) unreacheable :-( |
[production] |
| 15:18 |
<beta-logmsgbot> |
hashar: deleted jobrunner06 (precise), we just need one precise instance which will be jobrunner05 for now |
[releng] |
| 14:45 |
<beta-logmsgbot> |
hashar: installing applicationserver::homeless and applicationserver::jobrunner on jobrunner06 |
[releng] |
| 14:42 |
<beta-logmsgbot> |
hashar: installing applicationserver::homeless and applicationserver::jobrunner on jobrunner05 |
[releng] |
| 14:27 |
<beta-logmsgbot> |
hashar: installed jobrunner05 and 06 using Ubuntu precise. Should let get a 0.27 ffmpeg installation for {{bug|37043}} |
[releng] |
| 13:56 |
<RobH> |
palladium disk replaced |
[production] |
| 13:52 |
<cmjohnson1> |
replacing ps2 on mw1017 |
[production] |
| 13:50 |
<RobH> |
palladium has a bad disk, goign to replace it |
[production] |
| 13:37 |
<RobH> |
updating drac on search18, shouldnt cause system reboot. |
[production] |
| 10:42 |
<apergos> |
restarted apache on kaulen, was seeing page.cgi segfaults in dmesg and he logs, huge cpu wait spikes (why?) |
[production] |
| 08:38 |
<beta-logmsgbot> |
root: on dbdump, deleted /etc/logrotate.d/mw-udp2log . Most probably in conflict with the one from deployment-feed which host the udp2log process |
[releng] |
| 08:35 |
<beta-logmsgbot> |
root: gzipped /home/wikipedia/logs/archive/*20120525 see {{bug|37012}} :-( |
[releng] |
| 08:23 |
<beta-logmsgbot> |
hashar: killed stuck jobs on jobrunner 02 and 03. Restarted loop. |
[releng] |
|
2012-05-24
§
|
| 23:34 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 23:32 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php 'Updating technical feedback email address for mobile feedback' |
[production] |
| 23:17 |
<awjrichards> |
synchronizing Wikimedia installation... : Picking up changes to hide feedback form to prevent spamming of mobile feedback page - f6ed8ba |
[production] |
| 23:12 |
<awjrichards> |
synchronized wmf-config/CommonSettings.php 'Enabling 'technical feedback' link on mobile feedback form to disable feedback form' |
[production] |
| 23:00 |
<binasher> |
stopped replication on es1002 in order to rsync cluster23 to es1003 |
[production] |
| 21:53 |
<aaron> |
synchronized php-1.20wmf3/extensions/UploadWizard 'deployed 144b58854e38d910210ccd23402225e5b1d2d62d' |
[production] |
| 21:52 |
<RoanKattouw> |
Restarted morebots |
[production] |
| 11:52 |
<beta-logmsgbot> |
hashar: Rewrote log command to use dologmsg and the new beta-logmsgbot |
[releng] |
| 11:51 |
<beta-logmsgbot> |
hashar: yeah I do log |
[releng] |
| 11:47 |
<hashar> |
Moving /bin/log to /usr/local/bin/log |
[releng] |
| 11:45 |
<labs-logs-bottie> |
hashar: installing php5-dev and running `pecl install parsekit` on deployment-dbdump |
[releng] |
| 09:43 |
<labs-logs-bottie> |
hashar: installing php5-dev and running `pecl install parsekit` on deployment-dbdump |
[releng] |
| 09:42 |
<labs-logs-bottie> |
hashar: installing php5-dev and running `pecl install parsekit` |
[releng] |
| 07:05 |
<hashar> |
killed some stalled jobs on jobrunner02 |
[releng] |