Error Clean up – on to Adding more sites to the crawler

The exxxcavate crawler is having less errors. It was originally creating a bad slug with listing titles that contained “-” ( hyphen ) in it, simply by creating the slug with a “–” in it. Easy enough, i just added another str_replace call to it and it cleared right up

Also, some of the titles were ending up with hex characters in them ( %20 , %33 etc,… ) so i added a rawurldecode to the call and i think its cleared up. Some of the titles now have special chars like ( ! ) or ( : ) in them and they seem to parse fine, and get left out in the slug ( which is good ). One concern, there was a listing grabed last night that had nothing but special chars like “色戒做爱” and the slug was totally empty. Some of the japanese films sometimes are just japanese characters, which are considered special, which would all get blanked in the slug. I am going to check if the slug is empty and if so, create a mock slug ( like the post id ) so at least something is in that field. Otherwise, i don’t think the listing would show.

I am creating a form to easily check crawl vars for new sites. Hopefully i can have those all added today

Share this post