Problem with robots.txt module for Orchard CMS and HTTP 404 Not found

Many modern CMS allow you to specify content of robots.txt manually, which then will be available for search engines by the regular URL: http://example.com/robots.txt. In Orchard you can use Robots module from online gallery for that. This module adds dynamic route for robots.txt file:

   1: public IEnumerable<RouteDescriptor> GetRoutes() {

   2:     return new[] {

   3:         new RouteDescriptor {   Priority = 5,

   4:                                 Route = new Route(

   5:                                     "robots.txt",

   6:                                     new RouteValueDictionary {

   7:                                         {"area", "SH.Robots"},

   8:                                         {"controller", "Robots"},

   9:                                         {"action", "Index"}

  10:                                     },

  11:                                     new RouteValueDictionary(),

  12:                                     new RouteValueDictionary {

  13:                                         {"area", "SH.Robots"}

  14:                                     },

  15:                                     new MvcRouteHandler())

  16:         },

  17:     };

  18: }

Content of robots.txt is stored in the CMS content database. You can change it from admin panel in Robots.txt section.

In some cases when you will install this module and enable robots.txt feature, you will get HTTP 404 Not found error when will try to access http://example.com/robots.txt. In Orchard by default all static files from the virtual folder will give 404 result. David Hayden described it in his post: Modifying Web.config to Serve Site.xml and Static Files with Orchard CMS.

However in our case robots.txt is not static file. It is dynamic ASP.Net MVC route. So why we can get 404 error? It can be caused if someone put real static robots.txt file to the virtual folder. In this case request will be handled by IIS without ASP.Net MVC (real files have priority over dynamic routes), and you will get 404 for robots.txt because as I said above in Orchard all static files give 404 by default. In order to fix the issue, remove static robots.txt from the route.