5901-5950 of 7932 results (26ms)
2014-12-02 §
23:39 <bd808> Cause of full disk on deployment-mediawiki01 was an hhvm core file; fixed now [releng]
23:35 <bd808> /var full on deployment-mediawiki01 [releng]
17:09 <_joe_> upgrading HHVM to its latest version [releng]
17:08 <andrewbogott> this is a test message [releng]
11:27 <hashar> deleting /srv/vdb/varnish* files on all varnish instances ( https://phabricator.wikimedia.org/T76091 ) [releng]
10:23 <hashar> restarted parsoid on deployment-parsoid05 [releng]
05:26 <Krinkle> integration-slave1001 has been down since the failed reboot on 28 November 2014. Still unreachable over ssh and no Jenkins slave agent. [releng]
2014-12-01 §
21:50 <cscott-split> updated OCG to version a06e7c186796a6ee5d5af81e93688520abdf2596 [releng]
18:54 <bd808> Got jenkins updates working again by taking deployment-bastion node offline, killing waiting jobs and bringing it back online again. [releng]
18:51 <bd808> updates in beta suck with the "Waiting for next available executor" deadlock again [releng]
17:59 <bd808> Testing rsyslog event forwarding to logstash via puppet cherry-pick [releng]
2014-11-27 §
12:28 <hashar> enabled puppet master autoupdate by setting <tt>puppetmaster_autoupdate: true</tt> in [[Hiera:Integration]] . https://phabricator.wikimedia.org/T75878 [releng]
12:28 <hashar> rebased integration puppetmaster : 5d35de4..1a5ebee [releng]
00:32 <bd808> Testing local hack on deployment-salt to switch order of heira backends [releng]
00:16 <bd808> Testing a proposed puppet patch to allow pointing hhvm logs back to deploment-bastion [releng]
2014-11-26 §
20:47 <cscott> updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef [releng]
00:51 <bd808> cherry-picked patch for redis logstash input from MW {{gerrit|175896}} [releng]
00:50 <bd808> Restored puppet cherry-picks from reflog [phab:T75947] [releng]
2014-11-25 §
23:45 <hashar> Fixed upload cache on beta cluster, the Varnish backend had a mmap SILO error that prevented the backend from starting. https://phabricator.wikimedia.org/T75922 [releng]
21:05 <bd808> Running `sudo find . -type d ! -perm -o=w -exec chmod 0777 {} +` to fix upload permissions [releng]
18:01 <legoktm> cleared out renameuser_status table (old broken global merges) [releng]
18:00 <legoktm> 4086 rows deleted from localnames, 3929 from localuser [releng]
17:59 <legoktm> clearing out localnames/localuser where wikis don't exist on beta [releng]
17:10 <legoktm> ran migratePass0.php on all wikis [releng]
17:09 <legoktm> ran checkLocalUser.php --delete on all wikis [releng]
17:08 <legoktm> PHP Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/mediawiki/php-master/includes/SiteConfiguration.php on line 307 [releng]
17:07 <legoktm> ran checkLocalNames.php --delete on all wikis [releng]
04:37 <jgage> restarted jenkins at 20:31 [releng]
2014-11-24 §
17:24 <greg-g> stupid https [releng]
16:40 <bd808|deploy> My problem with en.wikipedia.beta.wmflabs.org was caused by a forceHTTPS cookie being set in my browser and redirecting to the broken https endpoint [releng]
16:33 <bd808|deploy> scap fixed by reverting bad config patch; still looking into failures from en.wikipedia.beta.wmflabs.org [releng]
16:27 <bd808> Looking at scap crash [releng]
15:18 <YuviPanda> restored local hacks + fixed 'em to account for 47dcefb74dd4faf8afb6880ec554c7e087aa947b on deployment-salt puppet repo, puppet failures recovering now [releng]
15:16 <YuviPanda> modified local hack to account for 47dcefb74dd4faf8afb6880ec554c7e087aa947b [releng]
14:58 <YuviPanda> cherry-picked 3e45c538978710113e6e28e9d533bf8d18c159a6 and 9d4614a8a352c78505212fd6e9d2a7be6d2e4927 to deployment-salt puppetmaster, restoring local hacks [releng]
2014-11-21 §
17:06 <bd808> deleted salt keys for deleted instances: i-00000289, i-0000028a, i-0000028b, i-0000028e, i-000002b7, i-000006ad [releng]
15:57 <hashar> fixed puppet cert on deployment-restbase01 [releng]
15:50 <hashar> deployment-sca01 regenerating puppet CA for deployment-sca01 [releng]
15:34 <hashar> Renerated puppet master certificate on deployment-salt. It needs to be named deployment-salt.eqiad.wmflabs not i-0000015c.eqiad.wmflabs. Puppet agent works on deployment-salt now. [releng]
15:19 <hashar> I have revoked the deployment-salt certificates. All puppet agent are thus broken! [releng]
15:01 <hashar> deployment-salt cleaning certs with puppet cert clean [releng]
14:52 <hashar> manually switching restbase01 puppet master from virt1000 to deployment-salt.eqiad.wmflabs [releng]
14:50 <hashar> deployment-restbase01 has some puppet error: Error 400 on SERVER: Must provide non empty value. on node i-00000727.eqiad.wmflabs . That is due to puppet pickle() function being given an empty variable [releng]
2014-11-20 §
15:25 <hashar> 15:01 Restarted Jenkins AND Zuul. Beta cluster jobs are still deadlocked. [releng]
13:21 <hashar> for integration, set puppet master report retention to 360 minutes ( https://wikitech.wikimedia.org/wiki/Hiera:Integration , see https://bugzilla.wikimedia.org/show_bug.cgi?id=73472#c14 ) [releng]
13:20 <hashar> rebased puppet master on integration project [releng]
13:20 <hashar> rebased puppet master [releng]
2014-11-19 §
21:27 <bd808> Ran `GIT_SSH=/var/lib/git/ssh git pull --rebase` in deployment-salt:/srv/var-lib/git/labs/private [releng]
21:19 <anomie> Cherry-picked https://gerrit.wikimedia.org/r/#/c/173336/3 to Beta [releng]
2014-11-18 §
15:32 <hashar> Deleting job https://integration.wikimedia.org/ci/job/mediawiki-vendor-integration/ replaced by mediawiki-phpunit. Clearing out workspaces {{bug|73515}} [releng]