My take on a Python module

Development forum for the Python programming language.

My take on a Python module

Postby MSchulte » 07 Aug 2014, 11:57

https://github.com/djsmedes/thepyetribe

I've been working with these trackers for the better part of three months, and after a few different iterations, this is the best way I have come up with for communicating with it.

It uses new-style classes, i.e. inheriting from object, and Python's @property and @property.setter decorators to simplify access to the tracker's attributes.

It uses a HeartThread to send heartbeats. I specifically don't give the thread access to the socket, but instead pass in an anonymous lambda function that, when called, sends the heartbeat. Not terribly important in this case, but I think in general it's a good idea to compartmentalize, if only so that you are not tempted to write more complex and dangerous code.

It uses a ListenerThread to constantly receive data from the socket and put it into a Queue. We split on newlines so that each message the tracker sends is its own item in the Queue, even if they are sent so close together that they pop out of the same socket.recv call. This thread is also passed an anonymous lambda function, lambda: self.socket.recv(BUFSIZE), for reasons similar to the above.

It uses a ProcessorThread to take strings from the raw Queue, parse them into dicts, and put them into an appropriate "EyeTribeQueue" or write them to a file. The parsing is achieved in two stages: first, a string.replace call which turns the C error "-1.#IND" into "null" because Python's json module does not know what to do with that C error. If more things like this are observed, more processing will be done in this stage. The second stage is to call json.loads on the string. We then check a few keys in the dict for certain conditions, which determines whether we write the message to a file, put it in one of the EyeTribeQueues, or notify that certain conditions have changed (i.e. display index). Note that when in push mode, frame data will still have a request of "get," which I am not sure is accurately reflected on the documentation on this website. If that were to change, some changes would need to be made to the logic in this thread.

The EyeTribeQueue inherits from LifoQueue, but replaces the normal Queue.get() method with an error message. Instead, you use the new Queue.get_item(request [, values]) method, which will go through the Queue's underlying data structure (a list in this case, which is why we inherited from LifoQueue instead of just Queue) and pull out the message that has the specified request (and values if the request was "get").

There is only one place in the code where socket.send() is called, but multiple threads could conceivably all be trying to do this simultaneously, so we include a threading.Lock object there.

I would like to thank Edwin Dalmaijer for sharing his code at https://github.com/esdalmaijer/PyTribe. I got the idea of storing the most recent frame as a "current frame" object from him, and probably a lot of other things were inspired by his code as well.

Comments, questions, and critiques welcome!

Cheers,
Daniel Smedema

(MSchulte is not actually me, it's my boss, but he's the one with the account here)
MSchulte
 
Posts: 5
Joined: 10 Feb 2014, 18:13

Re: My take on a Python module

Postby MSchulte » 08 Aug 2014, 16:18

Just realized that my code has a sneaky bug in it. Line 211 reads:
Code: Select all
self._frame_file.write('{}\n'.format(msg))


But it should be:
Code: Select all
self._frame_file.write('{}\n'.format(raw_msg))


By recording msg instead of raw_msg to the file, I accidentally made it write a representation of a Python dict, rather than the JSON-formatted string given by the tracker. I will commit a fix in a few hours.

I discovered this after trying to parse a bunch of data (that we paid subjects to collect!) and getting repeated JSON errors. I got around it by simply using eval() instead of json.loads() on the saved data.

Cheers,
Daniel
MSchulte
 
Posts: 5
Joined: 10 Feb 2014, 18:13

Re: My take on a Python module

Postby esdalmaijer » 12 Aug 2014, 01:52

Looks very good! In return, I'm borrowing that queued responses idea of yours ;)
esdalmaijer
 
Posts: 5
Joined: 24 Mar 2014, 22:05

Re: My take on a Python module

Postby MSchulte » 13 Aug 2014, 16:04

Enjoy :D
MSchulte
 
Posts: 5
Joined: 10 Feb 2014, 18:13


Return to Python



cron