ColdFusion searching robots.txt for specific page exception - coldfusion

Question

We're adding some functionality to our CMS whereby when a user creates a page, they can select an option to allow/disallow search engine indexing of that page.
If they select yes, then something like the following would apply:
<cfif request.variables.indexable eq 0>
<cffile
action = "append"
file = "C:\websites\robots.txt"
output = "Disallow: /blocked-page.cfm"
addNewLine = "yes">
<cfelse>
<!-- check if page already disallowed in robots.txt and remove line if it does --->
</cfif>
It's the <cfelse> clause I need help with.
What would be the best way to parse robots.txt to see if this page had already been disallowed? Would it be a cffile action="read", then do a find() on the read variable?
Actually, the check on whether the page has already been disallowed would probably go further up, to avoid double-adding.

Answer 1

You keep the list of pages in database and each page record has a indexable bit, right? If yes, simpler and more reliable approach would be to generate new robots.txt each time some page is added/deleted/changes indexable bit.
<!--- TODO: query for indexable pages ---->
<!--- lock the code to prevent concurrent changes --->
<cflock name="robots.txt" type="exclusive" timeout="30">
<!--- flush the file, or simply start with writing something --->
<cffile
action = "write"
file = "C:\websites\robots.txt"
output = "Sitemap: http://www.mywebsite.tld/sitemap.xml"
addNewLine = "yes">
<!--- append indexable entry to the file --->
<cfloop query="getPages">
<!--- we assume that page names are not entered by user (= safe names) --->
<cffile
action = "append"
file = "C:\websites\robots.txt"
output = "Disallow: /#getPages.name#.cfm"
addNewLine = "yes">
</cfloop>
</cflock>
Sample code is not tested, be aware of typos/bugs.

Answer 2

Using the Robots.txt files for this purpose is a bad idea. Robots.txt is not a security measure and you're handing "evildoers" a list of pages that you don't want indexed.
You're much better off using the robots meta tag, which will not provide anyone with a list of pages that you don't want indexed, and gives you greater control of the individual actions a robot can perform.
Using the meta tags, you would simply output the tags when generating the page as usual.

Answer 3

<!--- dummy page to block --->
<cfset request.pageToBlock = "/blocked-page.cfm" />
<!--- read in current robots.txt --->
<cffile action="read" file="#expandPath('robots.txt')#" variable="data" />
<!--- build a struct of all blocked pages --->
<cfset pages = {} />
<cfloop list="#data#" delimiters="#chr(10)#" index="i">
<cfset pages[listLast(i,' ')] = '' />
</cfloop>
<cfif request.variables.indexable eq 0>
<!--- If the page is not yet blocked add it --->
<cfif not structKeyExists(pages,pageToBlock)>
<cffile action="append" file="C:\websites\robots.txt"
output="Disallow: #request.pageToBLock#" addNewLine="yes" />
<!--- not sure if this is in a loop but if it is add it to the struct for nex iteration --->
<cfset pages[request.pageToBlock] = '' />
</cfif>
</cfif>
This should do it. Read in the file, loop over it and build a struct of the bloocked pages. Only add a new page if it's not already blocked.