2651-2700 of 3269 results (9ms)
2012-08-03 §
13:38 <hashar> cache-bits02 : cd /var/lib/git/operations/puppet sudo GIT_SSH=/var/lib/git/ssh git fetch origin refs/changes/04/13304/14 && git checkout -b 13304/14 FETCH_HEAD (aka deploying {{gerrit|13304}} patchset 14 [releng]
10:21 <wm-bot> running 'git submodule foreach git gc --aggressive' in screen 2042 [releng]
10:11 <wm-bot> rebooting swapped -dbdump [releng]
10:10 <wm-bot> Probably killed -dbdump by launching 4 instances of git gc --aggressive :-((((((((((( [releng]
10:09 <wm-bot> hashar is going to kill us all [releng]
10:07 <wm-bot> running massive git gc --aggressive on all extensions [releng]
09:54 <wm-bot> Updating extensions to latest master: git submodule foreach git pull [releng]
09:54 <wm-bot> hashar :P [releng]
09:53 <hashar> Updating mediawiki core to latest master: Updating 0c1471c..d47c1e9 (fast forwarded) [releng]
09:49 <hashar> On extensions: git submodule foreach git checkout master [releng]
09:13 <hashar> deleting deployment-bastion, it lacks a DNS entry {{bug|38846}} [releng]
09:12 <hashar> running git gc --aggressive in /home/wikipedia/common/php-master [releng]
08:48 <hashar> making sure perms are correct in /data/project/apache/common-local/php-master : chmod -R g+w . ; chown mwdeploy:svn -R * .* [releng]
08:45 <hashar> On -dbdump /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start {{bug|38995}} [releng]
08:35 <hashar> running puppet on Apache32 / 33 [releng]
08:12 <hashar> dist-upgrading all instances to get the latest GlusterFS version (3.3.0) [releng]
08:11 <hashar> Dist upgrading apache32 and 33 and rebooting them [releng]
2012-08-02 §
18:06 <andrewbogott> Migrated all instances to new hardware [releng]
15:14 <hashar> yeah we lost udp2log again! -dbdump : /etc/init.d/udp2log-mw restart [releng]
07:58 <hashar> {{bug|38748}} deleting unused/corrupted deployment-wmsearch instance. (had stuff like: -bash: /usr/bin/groups: cannot execute binary file. Connection to deployment-wmsearch.pmtpa.wmflabs closed.) [releng]
2012-07-30 §
15:00 <hashar> deployment-bastion does not let us log in despite being a fresh instance. Logged as {{bug|38846}} [releng]
14:47 <hashar> rebooting -bastion [releng]
12:48 <hashar> Shutdowning deployment-nfs-memc for a while, will see if it is still needed around or if we can safely delete it (see {{bug|38084}}). All data should be on /data/project . [releng]
12:43 <hashar> Recreating deployment-bastion using a Precise image and s1.small (1CPU, 1GB RAM, 80G storage) [releng]
12:40 <hashar> Deleting -bastion , was corrupted. [releng]
2012-07-28 §
03:38 <labs-logs-bottie> j: only have one commons wgForeignFileRepos: wikimediacommons at commons.wikimedia.beta.wmflabs.org (/data/project/apache/common/wmf-config/filebackend-wmflabs.php) [releng]
01:09 <Platonides> testwiki is now showing random captchas [releng]
01:09 <Platonides> moved /mnt/upload6/captcha/random to /mnt/upload6/private/captcha/random [releng]
00:22 <Platonides> generating random-challenge captchas at /mnt/upload6/captcha/random [releng]
2012-07-27 §
22:50 <Platonides> Filesystem corruption signs in deployment-bastion, most debconf backends /usr/share/perl5/Debconf/FrontEnd are zeroed files. This explains some of the earlier apt-get problems, and maybe also bug 38747 (aka. magic tail -f fix) [releng]
22:49 <Damianz> Thumbnails are currently broken - thumbs folder seems empty, images are there as expected - beta quirk or need rebuilding? [releng]
22:22 <Damianz> DocumentRoot got wiped, broken all the sites - fixed the broken symlink on bastion for /data/project/apache to /usr/local/apache. Also fixed common-local to common symlink. [releng]
21:56 <Platonides> apt-get removed wikimedia-task-appserver in deployment-bastion :( [releng]
21:42 <Platonides> removed generated captcha files [releng]
21:33 <Platonides> installed in deployment-bastion the packages joe, python-imaging and wamerican [releng]
20:29 <labs-logs-bottie> j: deployment-bastion: removing all deployment-nfs-memc entries from /etc/fstab [releng]
13:14 <hashar> restarted job runner on jobrunner06 [releng]
13:11 <hashar> removing all deployment-nfs-memc entries from /etc/fstab [releng]
09:51 <hashar> {{bug|38749}} jobrunner06 : removed /usr/local/apache and /mnt/upload6 empty dir. Downgraded PHP manually. Rerunning puppet. [releng]
08:46 <hashar> migrating jobrunner06 to use the /data/project for uploads [releng]
2012-07-26 §
19:27 <labs-logs-bottie> j: delete broken deployment-video01 instance [releng]
18:42 <labs-logs-bottie> j: new instance deployment-video02 with videoscaler with access to gluster instead of nfs [releng]
16:15 <hashar> applying nfs::apache::labs on -dbdump to get /usr/local/apache from /data/project [releng]
16:12 <hashar> -dbdump umounted deployment-nfs-memc:/mnt/export/apache on /usr/local/apache [releng]
13:56 <hashar> archiving -nfs-memc:/mnt/export : root@deployment-nfs-memc:/mnt# mv export /data/project/deployment-nfs-memc_mnt-export_backup [releng]
13:46 <hashar> seems to work fine with /data/project now. [releng]
13:46 <hashar> rsync finished for both apache and upload6. Remounting and restarting apaches [releng]
13:35 <hashar> rsync from nfs-memc:/mnt/export/upload6 to /data/project/upload6 completed. YEAHHH [releng]
13:32 <hashar> on -dbdump, unmounted /mnt/upload (from nfs-memc). Please use the /mnt/upload6 -> /data/project/upload6 [releng]
13:29 <hashar> root@deployment-nfs-memc:/mnt/export# rsync -a --progress --delete --inplace /mnt/export/apache /data/project [releng]