vote up 2 vote down
star

Which is the right syntax to block all crawlers excepts Google AdSense and Google AdWords using a robots.txt? This is the content of my robots.txt file

User-agent: Mediapartners-Google
Allow: /
User-agent: Adsbot-Google
Allow: /
flag

2 Answers

vote up 3 vote down
check

Just add a 

User-agent: *
Disallow: /

after the user agents you want to allow.

For the sake of completeness, this is the final robots.txt file.

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /
User-agent: Adsbot-Google
Allow: /
link|flag
vote up 0 vote down

What purpose would this be for... Landing pages?

link|flag

Your Answer

Get an OpenID
or

Not the answer you're looking for? Browse other questions tagged or ask your own question.