[warning]people should protect their files better

or just make a no-robots file.

http://%7boption%7d

eh, still, there are some search engines that don't obey those.

no-robots files are rather useless these days, imho. Its not a standard, and its pretty easy to make a spider that will ignore those. Hell, even Google's ignore those. The only way to get Google spiders to stop hitting certain areas, or all of your page is to call them up and tell them to stop (They're actually really good about that sort of thing).

Most of those map files are rubbish tho stepp

so unless you either link to the files or to the folder, how do these bot-spiders even find the files anyways?

You can search for specific file extesions, they don't necessarily need to be linked in a page for google to find them.

when we did GTA:F we had passworprotected FTP, Every user had own folder, which was allso password protected (every one had own pw)..

Quote from csharp

when we did GTA:F we had passworprotected FTP, Every user had own folder, which was allso password protected (every one had own pw)..

That's awesome.

Quote from csharp

GTA:F

?

Quote from D3ads

?

Quote from Section_Ei8ht

no-robots files are rather useless these days, imho. Its not a standard, and its pretty easy to make a spider that will ignore those. Hell, even Google's ignore those. The only way to get Google spiders to stop hitting certain areas, or all of your page is to call them up and tell them to stop (They're actually really good about that sort of thing).

While you're right in saying that it's not a standard, I must point out that your claim of Google not obeying robots.txt is nonsense. Google does obey them; have a look in the official support database and you will see evidence of this, and Google even uses robots.txt on their own sites at times. Almost all respectable engines will obey robots.txt, only leaving the smaller ones that much fewer people use and various other crawlers such as those operated by Blog scourers. I'm not sure where exactly you got your facts, but they're wrong.

But, still, don't rely on a robot denial file if absolute privacy is vital. Instead, use a .htaccess password (Google for “htpasswd tool” for an easy-to-use utility for creating one) or something else that's more physical.

GTA:F = Grand Theft Auto: Finland.. but like allmost 90% of mods, this one died.. But atleast we made 0.1A (and 0.1B internal beta).. Damn we had great team, but we were lazy If someone wants 0.1b i can upload it (it just replaces couple of vehicles and gangs (every gang skin is made by me o//).. I think there were allso couple of weapons in 0.1b.. And yeah, it is to san andreas

Quote from Thrik

While you're right in saying that it's not a standard, I must point out that your claim of Google not obeying robots.txt is nonsense. Google does obey them; have a look in the official support database and you will see evidence of this, and Google even uses robots.txt on their own sites at times. Almost all respectable engines will obey robots.txt, only leaving the smaller ones that much fewer people use and various other crawlers such as those operated by Blog scourers. I'm not sure where exactly you got your facts, but they're wrong.

Yeah, they probably are. I'm basing this off of something that happend at work a few years ago. A client site kept getting thrown offline because the spider would poll every single page in the inventory (poorly written asp code). Now that I think about it, the sysadmin there was probably too stupid to config the norobots file correctly. But it wasnt my job to fix it, I'm just going off of what he told me (You know, one of those bestfriend's-sister's-boyfriend's-uncle's-cousin's stories )