9351-9400 of 10000 results (36ms)
2014-09-10 §
10:16 <jeremyb> deployment-salt had an oom-kill recently. and some box (maybe master, maybe client?) had a disk fill up [releng]
10:15 <jeremyb> deployment-mediawiki0[12] both had good puppet runs [releng]
10:15 <jeremyb> deployment-salt started puppetmaster && puppet run [releng]
10:14 <jeremyb> deployment-bastion killed puppet lock [releng]
08:14 <Krinkle> bits.beta.wmflabs.org is down with 503 Service Unavailable (http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php) [releng]
03:04 <bd808> Ori made puppet changes that moved the MediaWiki install dir to /srv/mediawiki (https://gerrit.wikimedia.org/r/#/c/159431/). I didn't see that in SAL so I'm adding it here. [releng]
2014-09-09 §
20:08 <cscott> updated OCG to version c9a2b4cf2502479eeabed07ab2de728695d96e46 [releng]
03:06 <bd808> Restarted jenkins agent on delopment-bastion twice to resolve executor deadlock (bug 70597) [releng]
2014-09-07 §
23:48 <bd808> Added John F. Lewis to under_NDA sudo policy (bug 70539) [releng]
23:29 <bd808> Promoted John F. Lewis to project admin (bug 70539) [releng]
23:26 <bd808> Added Jalexander as project member (bug 70539) [releng]
07:00 <jeremyb> testing 1,2,3 [releng]
2014-09-05 §
17:54 <bd808> Purged varnish cache on deployment-cache-bits01 -- sudo varnishadm ban req.url '~' / [releng]
16:00 <YuviPanda> unfuck puppet on deployment-salt, puppet is stupid and does not properly report failed events on last_run_summary.yaml if there's a syntax error or a resource conflict. So I've to read last_run_report and do things with *that* instead now [releng]
15:49 <YuviPanda> deliberately fucking up puppet to see if icinga complains [releng]
09:52 <_joe_> cherry-picked I6ec53da483bebfa375eba2383cbf60123ff1ce26, it work [releng]
2014-09-04 §
16:06 <bd808> Manually cleaned bogus LocalRenameUserJob jobs from redis [releng]
13:54 <_joe_> stopped puppet on the appservers but mw03, testing an apache change [releng]
05:28 <legoktm> stopping jobrunner on deployment-jobrunner01 [releng]
05:22 <legoktm> restarted jobrunner on deployment-jobrunner01 [releng]
05:14 <bd808> Bad jobs in job queue filled up /var on jobrunner01 and killed jobrunner script. Leaving down for now until I find out how to delete the bad jobs. [releng]
01:41 <bd808> Killed old jobs-loop.sh processes on deployment-jobrunner01 [releng]
01:24 <bd808> Many jobrunner errors like "wikiversions-labs.cdb has no version entry for `amwiki`" with various wiki names [releng]
01:23 <bd808|AWAY> Started jobrunner service manually on jobrunner01. [releng]
00:44 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known when Trebuchet is running) [releng]
00:35 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known) [releng]
2014-09-03 §
15:02 <bd808> _joe_ rolled out a new hhvm package ~5 hours ago [releng]
15:01 <bd808> morebots is back thanks to petan [releng]
2014-09-02 §
15:34 <bd808> False alarm. SSL is borked in beta and we know that [releng]
15:29 <bd808> `curl -vL -H 'Host: en.wikipedia.beta.wmflabs.org' localhost` works from deployment-cache-text02 [releng]
15:27 <bd808> https://en.wikipedia.beta.wmflabs.org/ returning ERR_CONNECTION_REFUSED (is varnish down?) [releng]
2014-08-29 §
22:56 <bd808> Got puppet to run cleanly on deployment-mediawiki03. Should be ready for serving traffic. [releng]
22:39 <bd808> Fixed a merge conflict in operations/puppet on deployment-salt [releng]
21:46 <bd808> Forced install of "right version of libvips-tools on mediawiki03 `sudo apt-get install libvips-tools=7.38.5-2` [releng]
08:40 <hashar> rebooting deployment-cache-mobile03 (kernel up) [releng]
2014-08-28 §
21:32 <bd808> Added "Greg Grossmeier" to UnderNDA sudoers group [releng]
17:12 <bd808> Changed centralauth db to rename labswiki -> deploymentwiki [releng]
16:49 <bd808> CentralAuth looks broken on http://deployment.wikimedia.beta.wmflabs.org/ [releng]
16:49 <bd808> Apache vhosts look good again [releng]
16:34 <bd808> Restarted varnishes on deployment-cache-text02 [releng]
16:13 <andrewbogott> merging a patch that renames 'labswiki' to 'deploymentwiki' [releng]
09:21 <hashar> resetting git repository in /data/project/apache/conf to point to the betaclusterbranch of operations/mediawiki-config.git discarded all local hacks in the process [releng]
2014-08-27 §
23:03 <hashar> Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 [releng]
22:53 <hashar> removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 [releng]
22:48 <hashar> the IP is a known security audit. See Chris Steipp. [releng]
22:46 <hashar> blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP [releng]
22:37 <hashar> restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently [releng]
22:26 <bd808> when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) [releng]
22:19 <bd808> restarted varnish (again) on deployment-cache-text02 [releng]
22:10 <bd808> restarted varnish on deployment-cache-text02 [releng]