|
2011-10-25
§
|
| 22:54 |
<reedy> |
synchronizing Wikimedia installation... : [[rev:100759|r100759]] |
[production] |
| 22:06 |
<Tim> |
rebooting hume, it went OOM and stopped responding |
[production] |
| 21:03 |
<awjr> |
synced civicrm installs on grosley and aluminium to [[rev:671|r671]] of the wikimedia repo |
[production] |
| 20:13 |
<LeslieCarr> |
updating dns |
[production] |
| 18:37 |
<LeslieCarr> |
puppetizing mw1001-1160 - please ignore all alerts about them for now |
[production] |
| 17:52 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Reenable ClickTracking UDP logger, now pointing to emery' |
[production] |
| 16:52 |
<catrope> |
synchronized wmf-config/InitialiseSettings.php 'Disable ClickTracking UDP logging to offload locke. Collector had already been killed' |
[production] |
| 16:49 |
<RoanKattouw> |
Killed AFT log collector on locke. Process was a direct child of init (PID 1), command line was /usr/bin/udp2log --config-file=/etc/udp2log/aft -p 842 |
[production] |
| 13:11 |
<catrope> |
synchronized php-1.18/cache/interwiki-pr.cdb '[[bugzilla:31428|bug 31428]]' |
[production] |
| 12:18 |
<mark> |
All eBGP peerings of br1-knams are off now, no more packet loss |
[production] |
| 12:10 |
<mark> |
Reenabled some eBGP peerings of br1-knams as cr1-esams transit couldn't handle it alone |
[production] |
| 12:00 |
<mark> |
Shutting down all BGP peerings of br1-knams |
[production] |
| 11:15 |
<mutante> |
mw64 - zombie apt-get process,updated APT and libpam,syslog: "nrpe invoked oom-killer","Out of memory: kill process ..(apache2)" etc.,check "free swap" in Ganglia |
[production] |
| 10:39 |
<mutante> |
mw64 - Apache HTTP flapping OK-CRIT-OK.. - high load avg.,swapping, swap Size/Used 975864 |
[production] |
| 09:08 |
<mutante> |
erzurumi - installed libpam security updates |
[production] |
| 05:59 |
<Tim> |
on hume: running refreshImageMetadata.php on all wikis |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.18) at Tue Oct 25 02:20:43 UTC 2011 |
[production] |
|
2011-10-24
§
|
| 22:36 |
<aaron> |
synchronized php-1.18/includes/User.php |
[production] |
| 22:31 |
<aaron> |
synchronized php-1.18/includes 'deployed [[rev:100657|r100657]]' |
[production] |
| 22:26 |
<aaron> |
synchronized php-1.18/includes 'deployed [[rev:100655|r100655]]' |
[production] |
| 21:57 |
<LeslieCarr> |
reformatting all of the wm****.eqiad machines |
[production] |
| 21:54 |
<Ryan_Lane> |
turning off ci2 |
[production] |
| 19:41 |
<cmjohnson1> |
resetting storage1 to access bios and check setup |
[production] |
| 19:20 |
<aaron> |
synchronized wmf-config/flaggedrevs.php 'pre-emptive wg(Add/Remove)Groups settings to account for [[rev:100636|r100636]]; should have no visible effect' |
[production] |
| 19:06 |
<reedy> |
synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php '[[rev:100635|r100635]]' |
[production] |
| 18:59 |
<mutante> |
sodium - initial mailman setup, created site list, will now notify ops@ via mail, storing pass in private repo ./files/misc |
[production] |
| 18:36 |
<LeslieCarr> |
restarting dead nagios on spence |
[production] |
| 17:40 |
<nikerabbit> |
synchronizing Wikimedia installation... : i18ndeploy [[rev:100623|r100623]] |
[production] |
| 16:05 |
<mark> |
restarted pdns on nescio |
[production] |
| 14:59 |
<notpeter> |
pdns_recursor hung on dobson. restarting |
[production] |
| 14:30 |
<Jeff_Green> |
installed nagios/nsca on spence |
[production] |
| 14:27 |
<mutante> |
snapshot4 - dist-upgrade/kernel, reboot |
[production] |
| 14:19 |
<mark> |
Manually fixed package situation on formey |
[production] |
| 14:10 |
<mutante> |
ekrem - gzipped access.log.1, can move/delete access.logs older than X months? |
[production] |
| 14:08 |
<mutante> |
ekrem at 99% disk space ..again |
[production] |
| 13:14 |
<mutante> |
something fails btw with etherpad using openoffice ([soffice] <defunct>), (was before and after reboot) |
[production] |
| 13:10 |
<mutante> |
hooper - fsck was forced and OK, back up, blog and etherpad reachable again |
[production] |
| 13:06 |
<mutante> |
hooper - dist-upgrade/kernel, reboot |
[production] |
| 13:03 |
<mutante> |
hooper - temp. shutting down blog and etherpad for kernel update/reboot |
[production] |
| 11:34 |
<mutante> |
added TERM=dumb to sudo line in /home/w/bin/scap to prevent "unknown terminal type" errors (RT 1767) |
[production] |
| 10:52 |
<mutante> |
gurvin - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) |
[production] |
| 10:52 |
<mutante> |
fixing date on yvon |
[production] |
| 10:43 |
<mutante> |
yvon - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) |
[production] |
| 04:22 |
<tstarling> |
synchronizing Wikimedia installation... : [[rev:100578|r100578]] |
[production] |
| 03:48 |
<Tim> |
gave the whole Engineering group write access to the ops-requests queue |
[production] |
| 03:44 |
<Tim> |
added myself to the operations group in RT so that I could resolve RT #1789 |
[production] |
| 02:53 |
<Tim> |
re-enabled cron job |
[production] |
| 02:26 |
<tstarling> |
synchronized php-1.18/extensions/CheckUser/maintenance/purgeOldData.php '[[rev:100574|r100574]]' |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.18) at Mon Oct 24 02:23:42 UTC 2011 |
[production] |
| 02:09 |
<aaron> |
synchronized /php-1.18/resources/jquery.ui/themes/vector/images 'deployed [[rev:100571|r100571]]' |
[production] |