Almost got it

the crawler worked up 3800 listings before showing signs of hatred towards man kind. It began to error out when it was flipping categorys, but as far as i could tell, it did it fine before on all the sites, so i was totally unsure of what could have been happening.

So, i figured it was mysql spitting poop out its eating hole ( sure enough ). The querys were collecting in its cache.

usually, on a dedicated server, or something with root access, i could clear the cache in an instant, but the site is hosted at cirtex, so its got some sort of cache renewal rate ( probably every two or four hours ). So i have to wait it out

Blah

so probably 5-6k a day. That ain’t bad. Were looking at having the machine stop for a few hours every day so it can allow the mysql to clear, but that shouldent be a problem.

Anyways, time for a break

Share this post