I have several python crawler programs that need to run continuously, but each time they become unresponsive after running for two days. Each time, they are blocked at time.sleep(60), but the process continues. It's not dead. You can't exit the program even by typing Ctrl C on the console. You can only force quit in the task manager.
What is the reason for this? and how to solve this problem?
Thank you in advance, seniors!
巴扎黑2017-06-12 09:24:55
Problem analysis suggestions:
linux: You can try to track the process status through the strace command and analyze the output content, or analyze the various indicators of the process through the files in the corresponding directory of proc! Ctrl-C is invalid, the process is suspected Entering uninterruptible sleep, you can only know the details by looking at the actual data obtained
windows: Find the corresponding process through the task manager, right-click -> Create dump file, analyze the dump file to see if you can find out the cause.
Crawler program: Keep detailed logs of the crawling progress. Whether it is links or exceptions encountered, it is best to record them so that you can know the specific stop time and the progress of the crawling when it stopped. , next time you can focus on this progress first and then analyze it in depth.
PHP中文网2017-06-12 09:24:55
For Linux system, try using crontab to do scheduled tasks
For windows, use task scheduler