I recently received my tracker and am experiencing the exact same issue. No matter how I calibrate it the result is always very jittery, centred around or near the point I'm looking at but jumping around in an area of about 300px wide multiple times a second.
I'm also using a 1st gen 15" retina macbook pro with a 22" (maybe 24" I forget) external monitor. But the same problem occurs if I put the tracker under my laptop display and use it with the 15" display. I'm running OSX 10.9.4. Most of my testing has been done with server version 0.9.40, I recently upgraded to 0.9.41 but have only achieved a working calibration once (and the same problem occurred).
Sometimes I will try calibrating many times in a row and every time it will either fail or give a 1 star result, and sometimes I can reliably get 2-3 star calibrations. I have never achieved a calibration result higher than 3 stars. I normally wear contact lenses but I have tried using both glasses and no lenses at all, no lenses and contact lenses give similar results and glasses give worse results.
I'm a programmer with some computer vision experience, so if there are any troubleshooting techniques you would like me to do I'd be happy and able to do them. I've already looked at the raw output using Photo Booth. I do occasionally get corrupted horizontal lines flashing across the picture but they do not occur often enough to explain the constant jittering. Here is an image captured from photo booth using the tracker
https://www.evernote.com/shard/s9/sh/83 ... 414949f472. Is the noisiness and greenness normal?
Also note that accessing the tracker in photo booth causes the server to crash, leaving the LEDs on. The next time the server is started up it segfaults, and then leaves the lights off. The next time it works.