veg oil forum down?

[Julian]

This is the current robots.txt file ...
====================================
# XMB 1.9.11 Search Engine Optimization File
#
# To help search engines avoid members-only areas of your board, you should
# move this file to the root path of your website.  If your board is
# not hosted from the root path, then you must also edit the file.
# For example, Disallow: /forum/buddy

User-agent: *
Disallow: /buddy
Disallow: /cp
Disallow: /memcp
Disallow: /misc.php?action=captchaimage
Disallow: /misc.php?action=list&desc=
Disallow: /misc.php?action=list&order=
Disallow: /misc.php?action=search&
Disallow: /post
Disallow: /search.php?
Disallow: /today.php?daysold=
Disallow: /u2u
Disallow: /viewthread.php?fid=

====================================

If I've understood the notes correctly, none of the instructions will be actioned as the forum's not the root. 

[Tony]

You're correct Julian, they are all root URL disallows so won't affect the forum as intended.

[Julian]

You're correct Julian,

Blimey! That's a first.

So would that have a significant effect on band width, each spider crawling everywhere?  I assume the VOD must have quite a sizable data base by now.

[Glycer-rides]

Spiders? I was thinking 'Earwigs' were maybe responsible?

[Tony]

You're correct Julian,

Blimey! That's a first.

So would that have a significant effect on band width, each spider crawling everywhere?  I assume the VOD must have quite a sizable data base by now.

It's possible.  But then you do want the site crawled for indexing with search engines.  The robots.txt should only really be used to prevent indexing of areas that shouldn't appear in search results (and some robots will ignore it anyway and crawl the lot).