vsbabu.org

Lack of timeout hurts

As pretty much everyone used to the net knows by now, the MS SQL Server wreaked havoc yesterday. Indirectly, it also completely screwed up my blogroll and news aggregator Python scripts.

I used urllib module for fetching data from URLs. While the worm was having a swell time, most of the servers were very very slow to respond. This module doesn't implement timeouts. Since I run these scripts from Cron, all these jobs kept running forever.

Lesson learned: When you think of a possible bug, fix it at the earliest. I was lazy and kept postponing this and it hit finally My good hosts at Cube-Soft were kind enough to take down these cron jobs for me and notified me.

At the moment, I’ve decided to keep running these from command line instead of Cron.

I’ve not done much research on the options. But I’m fairly certain, I’ll have to use TimeoutSocket.py. Then get some enlightenment from Effbot’s excellent writeup about asyncore module. Perhaps even re-write these scripts to execute in multiple threads.

1. TimeoutSocket.py does not work for urllib.

   The good news is: Python 2.3 (currently only in the second alpha release but solid enough to use for our production web monitoring application) has socket timeouts (and for the Win32 users, SSL is compiled in).

   Posted by: Craig Allen on April 23, 2003 12:47 PM