1651-1700 of 3402 results (20ms)
2014-09-11 §
15:09 <bd808> Updated hhvm-luasandbox to latest version on mediawiki03 and verified that mediawiki0[12] were already updated [releng]
15:01 <bd808> Fixed incorrect $::deployment_server_override var on deployment-videoscaler01; deployment-bastion.eqiad.wmflabs is correct and deployment-salt.eqiad.wmflabs is not [releng]
10:05 <ori> deployment-prep upgraded luasandbox and hhvm across the cluster [releng]
08:41 <spagewmf> deployment-mediawiki01/02 are not getting latest code [releng]
05:10 <bd808> Reverted cherry-pick of I621d14e4b75a8415b16077fb27ca956c4de4c4c3 in scap; not the actual problem [releng]
05:02 <bd808> Cherry-picked I621d14e4b75a8415b16077fb27ca956c4de4c4c3 to scap to try and fix l10n update issue [releng]
02:29 <mutante> raised instance quota by 1 to 42 [releng]
2014-09-10 §
19:38 <bd808> Fixed beta-recompile-math-texvc-eqiad job on deployment-bastion [releng]
19:38 <bd808> Made /usr/local/apache/common-local a symlink to /srv/mediawiki on deployment-bastion [releng]
19:37 <bd808> Deleted old /srv/common-local on deployment-videoscaler01 [releng]
19:32 <bd808> Killed jobs-loop.sh tasks on deployment-jobrunner01 [releng]
19:30 <bd808> Removed old mw-job-runner cron job on deployment-jobrunner01 [releng]
19:19 <bd808> Deleted /var/log/account/pacct* and /var/log/atop.log.* on deployment-jobrunner01 to make some temporary room in /var [releng]
19:14 <bd808> Deleted /var/log/mediawiki/jobrunner.log and restarted jobrunner on deployment-jobrunner01: [releng]
19:11 <bd808> /var full on deployment-jobrunner01 [releng]
19:05 <bd808> Deleted /srv/common-local on deployment-jobrunner01 [releng]
19:04 <bd808> Changed /usr/local/apache/common-local symlink to point to /srv/mediawiki on deployment-jobrunner01 [releng]
19:03 <bd808> w00t!!! scap jobs is green again -- https://integration.wikimedia.org/ci/job/beta-scap-eqiad/20965/ [releng]
19:00 <bd808> sync-common finished on deployement-jobrunner01; trying Jenkins scap job again [releng]
18:53 <bd808> Removed symlink and make /srv/mediawiki a proper directory on deployment-jobrunner01; Running sync-common to populate. [releng]
18:45 <bd808> Made /srv/mediawiki a symling to /srv/common-local on deployment-jobrunner01 [releng]
10:20 <jeremyb> deployment-bastion /var at 97%, freed up ~500MB. apt-get clean && rm -rv /var/log/account/pacct* [releng]
10:17 <jeremyb> deployment-bastion good puppet run [releng]
10:16 <jeremyb> deployment-salt had an oom-kill recently. and some box (maybe master, maybe client?) had a disk fill up [releng]
10:15 <jeremyb> deployment-mediawiki0[12] both had good puppet runs [releng]
10:15 <jeremyb> deployment-salt started puppetmaster && puppet run [releng]
10:14 <jeremyb> deployment-bastion killed puppet lock [releng]
08:14 <Krinkle> bits.beta.wmflabs.org is down with 503 Service Unavailable (http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php) [releng]
03:04 <bd808> Ori made puppet changes that moved the MediaWiki install dir to /srv/mediawiki (https://gerrit.wikimedia.org/r/#/c/159431/). I didn't see that in SAL so I'm adding it here. [releng]
2014-09-09 §
20:08 <cscott> updated OCG to version c9a2b4cf2502479eeabed07ab2de728695d96e46 [releng]
03:06 <bd808> Restarted jenkins agent on delopment-bastion twice to resolve executor deadlock (bug 70597) [releng]
2014-09-07 §
23:48 <bd808> Added John F. Lewis to under_NDA sudo policy (bug 70539) [releng]
23:29 <bd808> Promoted John F. Lewis to project admin (bug 70539) [releng]
23:26 <bd808> Added Jalexander as project member (bug 70539) [releng]
07:00 <jeremyb> testing 1,2,3 [releng]
2014-09-05 §
17:54 <bd808> Purged varnish cache on deployment-cache-bits01 -- sudo varnishadm ban req.url '~' / [releng]
16:00 <YuviPanda> unfuck puppet on deployment-salt, puppet is stupid and does not properly report failed events on last_run_summary.yaml if there's a syntax error or a resource conflict. So I've to read last_run_report and do things with *that* instead now [releng]
15:49 <YuviPanda> deliberately fucking up puppet to see if icinga complains [releng]
09:52 <_joe_> cherry-picked I6ec53da483bebfa375eba2383cbf60123ff1ce26, it work [releng]
2014-09-04 §
16:06 <bd808> Manually cleaned bogus LocalRenameUserJob jobs from redis [releng]
13:54 <_joe_> stopped puppet on the appservers but mw03, testing an apache change [releng]
05:28 <legoktm> stopping jobrunner on deployment-jobrunner01 [releng]
05:22 <legoktm> restarted jobrunner on deployment-jobrunner01 [releng]
05:14 <bd808> Bad jobs in job queue filled up /var on jobrunner01 and killed jobrunner script. Leaving down for now until I find out how to delete the bad jobs. [releng]
01:41 <bd808> Killed old jobs-loop.sh processes on deployment-jobrunner01 [releng]
01:24 <bd808> Many jobrunner errors like "wikiversions-labs.cdb has no version entry for `amwiki`" with various wiki names [releng]
01:23 <bd808|AWAY> Started jobrunner service manually on jobrunner01. [releng]
00:44 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known when Trebuchet is running) [releng]
00:35 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known) [releng]
2014-09-03 §
15:02 <bd808> _joe_ rolled out a new hhvm package ~5 hours ago [releng]