5101-5150 of 7141 results (26ms)
2014-12-05 §
22:06 <bd808> /var full on deployment-mediawiki02 again :((( [releng]
10:50 <hashar> applying mediawiki::multimedia class on contint slaves ( https://phabricator.wikimedia.org/T76661 | https://gerrit.wikimedia.org/r/#/c/177770/ ) [releng]
01:01 <bd808> Deleted a ton of jeprof.*.heap files from deployment-mediawiki02:/ [releng]
00:54 <YuviPanda> cleared out pngs from mediawiki02 to kill low space warning [releng]
00:53 <YuviPanda> mediawiki02 instance is low on space, /tmp has lots of... pngs? [releng]
2014-12-04 §
22:48 <YuviPanda> manually rebased puppet on deployment-prep [releng]
00:29 <bd808> deleted instance "udplog" [releng]
2014-12-03 §
21:33 <cscott> updated OCG to version 08e94b19c3f17e699d7e53d9605f65c58e17ea0e [releng]
19:11 <bd808> Cleaned up legacy jobrunner scripts on deployment-jobrunner01 (/etc/default/mw-job-runner /etc/init.d/mw-job-runner /usr/local/bin/jobs-loop.sh) [releng]
2014-12-02 §
23:39 <bd808> Cause of full disk on deployment-mediawiki01 was an hhvm core file; fixed now [releng]
23:35 <bd808> /var full on deployment-mediawiki01 [releng]
17:09 <_joe_> upgrading HHVM to its latest version [releng]
17:08 <andrewbogott> this is a test message [releng]
11:27 <hashar> deleting /srv/vdb/varnish* files on all varnish instances ( https://phabricator.wikimedia.org/T76091 ) [releng]
10:23 <hashar> restarted parsoid on deployment-parsoid05 [releng]
05:26 <Krinkle> integration-slave1001 has been down since the failed reboot on 28 November 2014. Still unreachable over ssh and no Jenkins slave agent. [releng]
2014-12-01 §
21:50 <cscott-split> updated OCG to version a06e7c186796a6ee5d5af81e93688520abdf2596 [releng]
18:54 <bd808> Got jenkins updates working again by taking deployment-bastion node offline, killing waiting jobs and bringing it back online again. [releng]
18:51 <bd808> updates in beta suck with the "Waiting for next available executor" deadlock again [releng]
17:59 <bd808> Testing rsyslog event forwarding to logstash via puppet cherry-pick [releng]
2014-11-27 §
12:28 <hashar> enabled puppet master autoupdate by setting <tt>puppetmaster_autoupdate: true</tt> in [[Hiera:Integration]] . https://phabricator.wikimedia.org/T75878 [releng]
12:28 <hashar> rebased integration puppetmaster : 5d35de4..1a5ebee [releng]
00:32 <bd808> Testing local hack on deployment-salt to switch order of heira backends [releng]
00:16 <bd808> Testing a proposed puppet patch to allow pointing hhvm logs back to deploment-bastion [releng]
2014-11-26 §
20:47 <cscott> updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef [releng]
00:51 <bd808> cherry-picked patch for redis logstash input from MW {{gerrit|175896}} [releng]
00:50 <bd808> Restored puppet cherry-picks from reflog [phab:T75947] [releng]
2014-11-25 §
23:45 <hashar> Fixed upload cache on beta cluster, the Varnish backend had a mmap SILO error that prevented the backend from starting. https://phabricator.wikimedia.org/T75922 [releng]
21:05 <bd808> Running `sudo find . -type d ! -perm -o=w -exec chmod 0777 {} +` to fix upload permissions [releng]
18:01 <legoktm> cleared out renameuser_status table (old broken global merges) [releng]
18:00 <legoktm> 4086 rows deleted from localnames, 3929 from localuser [releng]
17:59 <legoktm> clearing out localnames/localuser where wikis don't exist on beta [releng]
17:10 <legoktm> ran migratePass0.php on all wikis [releng]
17:09 <legoktm> ran checkLocalUser.php --delete on all wikis [releng]
17:08 <legoktm> PHP Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/mediawiki/php-master/includes/SiteConfiguration.php on line 307 [releng]
17:07 <legoktm> ran checkLocalNames.php --delete on all wikis [releng]
04:37 <jgage> restarted jenkins at 20:31 [releng]
2014-11-24 §
17:24 <greg-g> stupid https [releng]
16:40 <bd808|deploy> My problem with en.wikipedia.beta.wmflabs.org was caused by a forceHTTPS cookie being set in my browser and redirecting to the broken https endpoint [releng]
16:33 <bd808|deploy> scap fixed by reverting bad config patch; still looking into failures from en.wikipedia.beta.wmflabs.org [releng]
16:27 <bd808> Looking at scap crash [releng]
15:18 <YuviPanda> restored local hacks + fixed 'em to account for 47dcefb74dd4faf8afb6880ec554c7e087aa947b on deployment-salt puppet repo, puppet failures recovering now [releng]
15:16 <YuviPanda> modified local hack to account for 47dcefb74dd4faf8afb6880ec554c7e087aa947b [releng]
14:58 <YuviPanda> cherry-picked 3e45c538978710113e6e28e9d533bf8d18c159a6 and 9d4614a8a352c78505212fd6e9d2a7be6d2e4927 to deployment-salt puppetmaster, restoring local hacks [releng]
2014-11-21 §
17:06 <bd808> deleted salt keys for deleted instances: i-00000289, i-0000028a, i-0000028b, i-0000028e, i-000002b7, i-000006ad [releng]
15:57 <hashar> fixed puppet cert on deployment-restbase01 [releng]
15:50 <hashar> deployment-sca01 regenerating puppet CA for deployment-sca01 [releng]
15:34 <hashar> Renerated puppet master certificate on deployment-salt. It needs to be named deployment-salt.eqiad.wmflabs not i-0000015c.eqiad.wmflabs. Puppet agent works on deployment-salt now. [releng]
15:19 <hashar> I have revoked the deployment-salt certificates. All puppet agent are thus broken! [releng]
15:01 <hashar> deployment-salt cleaning certs with puppet cert clean [releng]