701-750 of 10000 results (14ms)
2010-12-28 §
00:19 <Ryan_Lane> restarting apache on srv217 - puppet updated its apache2.conf file, but didn't restart it [production]
00:12 <Ryan_Lane> powercycling amssq53 - it's dead [production]
00:09 <Ryan_Lane> powercycling sq76 - it's dead [production]
00:08 <Ryan_Lane> knsq14 isn't coming back up - may be broken [production]
00:08 <Ryan_Lane> powercycling knsq24 - it's dead [production]
00:02 <Ryan_Lane> powercycling knsq14 - it's dead [production]
2010-12-27 §
23:54 <Ryan_Lane> starting apache on srv217, running puppet to ensure code is synched [production]
23:52 <Ryan_Lane> restarting apache on srv231 [production]
23:50 <laner> synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv226' [production]
23:48 <laner> synchronized php-1.5/wmf-config/mc.php 'Fixing IP address for srv227' [production]
23:39 <Ryan_Lane> powercycling srv217 - it's dead [production]
20:38 <Ryan_Lane> adding puppet schema to opendj on nova-controller.tesla [production]
19:51 <Ryan_Lane> specifically chgrp'd all files to svnadm and added write permissions for the group [production]
19:51 <Ryan_Lane> opened up file permissions on /srv/org/wikimedia/svn on formey so that ^demon can edit files [production]
19:48 <Ryan_Lane> disabled selenium service on windows7-1, and launched selenium services manually, while logged in as the selenium user [production]
18:04 <Ryan_Lane> installing puppet, puppet-el, puppetmaster, puppetmaster-passenger, and vim-puppet on nova-controller.tesla [production]
17:56 <Ryan_Lane> adding new 1TB "archive" logical volume on storage3, formatting as ext3, and mounting at /archive [production]
17:49 <Ryan_Lane> purging some bin logs on db9 to free up space [production]
16:27 <Ryan_Lane> installing graphviz on formey via puppet for bug 26404 [production]
09:21 <apergos> truncated log-all and log-index in /a/search/log on searchidx1 to get some space back [production]
2010-12-26 §
18:57 <Ryan_Lane> powercycling mobile1, as it is dead [production]
02:29 <Ryan_Lane> running /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedFlaggedRevs.php on s3/s7 flaggedrevs enabled wikis [production]
00:08 <ariel> synchronized wmf-deployment/wmf-config/db.php 'edits enabled for s3/s7, people should purge if they see weird things, flagged revs may still cause problems, hewiki still has issues' [production]
2010-12-25 §
23:39 <apergos> ran Platonides' script "purgeStaleMemcachedText.php" on s3/s7 projects, see /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedText.php on fenari (not added to the local branch yet) [production]
21:14 <ariel> synchronized wmf-deployment/wmf-config/db.php 'read-only for s3/s7 while we sort out edit sync and cache issues' [production]
20:17 <mark> New master bin-log db17-bin.003, position 406 [production]
20:16 <mark> Switched master on s3, s7 from db27 to db17, just one slave db25 [production]
20:16 <mark> synchronized php-1.5/wmf-config/db.php 'Going read-write on s3, s7' [production]
20:13 <mark> synchronized php-1.5/wmf-config/db.php 'Switch to new master db17, slave db25 for s3/s7' [production]
20:03 <mark> synchronized php-1.5/wmf-config/db.php [production]
20:02 <mark> synchronized php-1.5/wmf-config/db.php 'Remove db39 from rotation' [production]
19:58 <mark> Switching master to db17 [production]
19:58 <mark> synchronized php-1.5/wmf-config/db.php [production]
19:43 <Ryan_Lane> purging old bin logs on db27 [production]
14:32 <apergos> turned off Indexes option on wikitech in apache config files, just like on the cluster [production]
12:18 <apergos> powercycling searchidx1, can't get in through management console, nonresponsive to ssh, google says the weird ilom messages are a firmware issue, so... reset. [production]
11:11 <rainman-sr> searchidx1 is down for about 5 days now, can someone take a look at it? Cannot ssh to it. Also search group dissapeared from ganglia. [production]
00:22 <mark> Running apt-get dist-upgrade on mobile3 [production]
00:21 <mark> Rebalanced LVS weights of mobile servers on lvs4 from 20/50/30 to 30/50/50 [production]
00:06 <mark> powercycled mobile3 [production]
2010-12-24 §
18:18 <apergos> changed cname for dumps.wikimedia.org to point to dataset2 (so all related aliases will also point there) [production]
04:47 <Ryan_Lane> adding 512MB ram to nova-controller.tesla; shutting down VM to do so [production]
2010-12-23 §
18:23 <apergos> added export of /data for download host to class in puppet [production]
17:57 <Ryan_Lane> ignore that last message [production]
17:57 <Cobi> TODO: Give Cobi root on servers [production]
17:54 <Ryan_Lane> added vivek and aditya to nova-controller.tesla, nova-compute1, nova-compute2, and added them to sudoers [production]
17:46 <apergos> added misc::download class for dataset2 in puppet (handles lighttpd for download.wm) [production]
14:02 <hcatlin> deploying code update to m. [production]
10:42 <tfinc> synchronized php-1.5/wmf-config/reporting-setup.php 'Flipping back to db10 since weve cleared up our space issues' [production]
01:39 <apergos> dataset2 racked and installed (thanks rich, robh, mark). now copying data back off tridge, rsync in screen session from tridge as root. [production]