• Hi i went over this problem in the thread I psted below, after I managed to get my site to spider my php files, i came accroos the problem of not spidering deep enough:

    Old Thread:
    http://www.phpdig.net/showthread.php?s=&threadid=764&perpage=15&pagenumber=2

    It spiders 20 deep, and thats as far as it goes, but I have folders that contain over 60 php files, and each one needs to be spidered, so can i set the spider depth to about 70?

    so for these big folders, 20 of the php fiiles are spidered and the rest are not...

    can someone explain how to dig a little deeper :)

    cheers,

    Alex.


  • O right I see, i'll try that and see how it goes!

    cheers!


  • This is just a guess, but have you tried changing this code in config.php:define('SPIDER_MAX_LIMIT',20);
    to this?define('SPIDER_MAX_LIMIT',70);


  • Yeah cool!

    I did what we said before, i've made 1 php page that generates a huge list of links to every index page and it works but it seems to miss some of them,

    i only spidered 1 deep because every link was in the same documnet, perhaps if a spider a little deeper it might find the ones its missing out?

    alex.


  • DAM its not working!

    Its still spidering only 20 results, i think its because my php page are only generating 20 results for the page, when it reaches 20 results it creates a button that links to the next 20, the button will refresh the page and list results 20 - 40,

    you see all the links that are sopidered are created by php and don't exist anywhere else, is that why its not spidering anymore than 20 do you think?

    it sounds like its right.

    :)


  • Find the following in config.php:define('SPIDER_MAX_LIMIT',20);
    and change it to this:define('SPIDER_MAX_LIMIT',40);
    Re-spider and see what you get.


  • Hi. PhpDig follows links. If there are no links to the other pages, then PhpDig won't find them.


  • I set the depth to 20 and links to 0, but, still it is not spidering deep enough
    only certain documents in the homepage.


  • What search depth and links per values did you choose? If you left them both at the default of zero, you'll only get the home page crawled. Try changing the search depth to 5, for example, and see what happens. Depending on how many pages phpdig spiders you'll have to set that value even higher.

    Hope this helps. :)


  • Hi all

    I am trying to crawl my website, for some reason PHPDig is only crawling very partially my website, it does not go deaper than the homepage.

    Best Regards:
    Amjad Sanad







    • #If you have any other info about this subject , Please add it free.#
    Your name:
    E-mail:
    Telphone:

    Your comments:


    If you have any other info about Not Spidering Deep enough (strange site i have tho) , Please add it free.