What is a robots.txt file?

Discussion in 'Search Engine Optimization and Marketing (SEO/SEM)' started by business orientation inc, Jun 1, 2016.

  1. business orientation inc

    business orientation inc New Member

    Joined:
    Jun 1, 2016
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Bangalore
    hi friends,

    What is a robots.txt file?
     
  2. joomdev

    joomdev New Member

    Joined:
    May 27, 2016
    Messages:
    27
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    New Delhi
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door
     
  3. SWIFFJUSTUS

    SWIFFJUSTUS ⓿The Mad Methodologist⓿⎠ Elite VIP Member

    Joined:
    May 5, 2010
    Messages:
    1,408
    Likes Received:
    16
    Trophy Points:
    38
    Location:
    THE LAB
  4. JDIZZLE

    JDIZZLE Make Money Online

    Joined:
    Dec 15, 2010
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    6
    awwww shit i see my man swift is back.wassup mr swiff are you gonna start up again?
     
  5. SWIFFJUSTUS

    SWIFFJUSTUS ⓿The Mad Methodologist⓿⎠ Elite VIP Member

    Joined:
    May 5, 2010
    Messages:
    1,408
    Likes Received:
    16
    Trophy Points:
    38
    Location:
    THE LAB
    Lol Whats up Dizz,
    Yeah I'm thinking about it...
    Gotta a lot of new stuff..A LOT..
    Wanna touch basis with Doc first
    and see what the plans are for the Forum.
    Man I really Loved this Place!!
    Like to see it back to it's Former Glory.
    What I have in mind I think it's Possible!!
     
  6. JDIZZLE

    JDIZZLE Make Money Online

    Joined:
    Dec 15, 2010
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    6
    thats what im talking about. im ready and got a legion behind me this time;)
     
  7. daica85

    daica85 Member

    Joined:
    Oct 5, 2011
    Messages:
    260
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    eCoupon.io
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

    Robots exclusion standard - Wikipedia, the free encyclopedia
    https://en.wikipedia.org/wiki/Robots_exclusion_standard

    Hope this help.
     
  8. socialtechy1

    socialtechy1 New Member

    Joined:
    Jun 3, 2016
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
     
  9. hakireview

    hakireview New Member

    Joined:
    Sep 23, 2017
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    A robots.txt fіlе gives instructions tо web robots аbоut the pages tһе website owner ԁоеѕn’t wish to bе ‘crawled’. Fоr instance, if уоu didn’t want уоur images to bе listed by Google аnԁ other search engines, уоu’ԁ blосk them using уоur robots.txt file.

    You can gо to your wеbѕіtе and check іf you have а robots.txt file bу adding /robots.txt іmmеԁіаtеlу after your ԁоmаіn name in tһе address bar аt the top
     
  10. paricpo

    paricpo Member

    Joined:
    Apr 30, 2016
    Messages:
    185
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Germany
    A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want to be accessed by search engine crawlers.
     
  11. vinukum

    vinukum New Member

    Joined:
    Feb 3, 2018
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    A robots.txt is written to tell the 'bots which pages, or parts of the websites that must be indexed and which parts should not. A 'good' robot will follow these instructions
     
  12. michaeltim

    michaeltim New Member

    Joined:
    Jan 30, 2018
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    dubhai
    Robots.txt is a text file webmasters create to instruct web, how to crawl pages on their website.
    Basic format:

    User-agent: [user-agent name]
    Disallow: [URL string not to be crawled]

    The robots.txt file is part of the the robots exclusion protocol (REP).
     

Share This Page