Saturday, December 8, 2012

Problem with robots.txt module for Orchard CMS and HTTP 404 Not found

Many modern CMS allow you to specify content of robots.txt manually, which then will be available for search engines by the regular URL: http://example.com/robots.txt. In Orchard you can use Robots module from online gallery for that. This module adds dynamic route for robots.txt file:

   1: public IEnumerable<RouteDescriptor> GetRoutes() {
   2:     return new[] {
   3:         new RouteDescriptor {   Priority = 5,
   4:                                 Route = new Route(
   5:                                     "robots.txt",
   6:                                     new RouteValueDictionary {
   7:                                         {"area", "SH.Robots"},
   8:                                         {"controller", "Robots"},
   9:                                         {"action", "Index"}
  10:                                     },
  11:                                     new RouteValueDictionary(),
  12:                                     new RouteValueDictionary {
  13:                                         {"area", "SH.Robots"}
  14:                                     },
  15:                                     new MvcRouteHandler())
  16:         },
  17:     };
  18: }

Content of robots.txt is stored in the CMS content database. You can change it from admin panel in Robots.txt section.

In some cases when you will install this module and enable robots.txt feature, you will get HTTP 404 Not found error when will try to access http://example.com/robots.txt. In Orchard by default all static files from the virtual folder will give 404 result. David Hayden described it in his post: Modifying Web.config to Serve Site.xml and Static Files with Orchard CMS.

However in our case robots.txt is not static file. It is dynamic ASP.Net MVC route. So why we can get 404 error? It can be caused if someone put real static robots.txt file to the virtual folder. In this case request will be handled by IIS without ASP.Net MVC (real files have priority over dynamic routes), and you will get 404 for robots.txt because as I said above in Orchard all static files give 404 by default. In order to fix the issue, remove static robots.txt from the route.

Several useful SEO rules via IIS URL rewrite module

In this post I would like to share with you several useful rules for IIS URL rewrite module, which can increase rating of your site for search engines. Before to add these rules to the web.config we need to install URL rewrite extension on the web server, otherwise site won’t work.

The first rule is redirect with 301 (Permanent redirect) http status code from the url without www prefix to the url with www. Very often sites are available by both urls, e.g. http://www.example.com and http://example.com. Search engines may treat these sites as separate (most of search engines may handle this case, but I would not suggest experimenting) and as content on them will be the same, search rank may be decreased. In order to fix this problem we need to setup permanent redirect from one address to another. It is important to use permanent redirect (301), because in this case search engine will know that url from which redirect was made is not used anymore and can be deleted from search index. Another way to perform redirect is to use 302 (Moved temporarily), but in this case page won’t be deleted from index. In theory it also can impact search rating, but I didn’t find exact evidences so if you know that please share it in comments.

In order to add the redirection rule to the site running on IIS we will use IIS URL rewrite module. You can add rules by several ways:

  • via UI – in IIS manager select site under the question and then select URL rewrite in the right panel (it will be added after installation of URL rewrite extension. Note that you need to restart IIS manager after installation. No need to make iisreset)
  • directly to the web.config using any text editor, e.g. notepad.

UI is just convenient interface for adding rules to the web.config. Result in both cases will be the same: rules will be added to the web.config of your site. In this post I will show how to add rules to web.config directly.

Rules are added to the <system.webServer> section under <rewrite>/<rules>. Redirection rule from no www to www will look like this:

   1: <rule name="redirect_from_nowwww_to_www" enabled="true" stopProcessing="true">
   2:   <match url=".*" />
   3:     <conditions>
   4:       <add input="{HTTP_HOST}" pattern="^example\.com$" />
   5:     </conditions>
   6:   <action type="Redirect" url="http://www.example.com/{R:0}"
   7: appendQueryString="true" redirectType="Permanent" />
   8: </rule>

Note that above I separated <action> to 2 lines (lines 6 and 7) in order to fit the article width, but in real example it should be single line. This rules tells to rewrite module that it should be applied to all urls (line 2) which have host “example.com” (line 4) and response should be redirected with 301 status code to “http://www.example.com” (lines 6-7). Here for matching urls we used regular expressions. {R:0} means back reference in term of regexp, which in this example has url part after the host. Plus we need to add possible query string in order to keep existing functionality working: see appendQueryString="true" attribute.

After that you can run fiddler and check that when you enter http://example.com in the browser, you will be redirected to http://www.example.com with 301 status. The rule will be also applied to all pages on the sites, including images and css.

Another useful rule is redirect from addresses without trailing slash to the addresses with slash: http://example.com/about –> http://example.com/about/. The reason here is the same: search engines may treat these url as different urls and if they will have the same content (in most cases they will), search rank can suffer. In order to add these redirection you can use the following url:

   1: <rule name="add_trailing_slash" stopProcessing="true">
   2:   <match url="(.*[^/])$" />
   3:   <conditions>
   4:     <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
   5:     <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
   6:   </conditions>
   7:   <action type="Redirect" redirectType="Permanent" url="{R:1}/" />
   8: </rule>

Here we used several conditions which mean that if physical file or folder are requested (matchType="IsFile" and matchType="IsDirectory"), the rule should not be applied (negate="true"). With this url you may have problems, e.g. if you use some CMS which allows to edit content of commonly used in SEO files robots.txt and sitemap.xml dynamically (i.e. if content is stored in the content database and routings are configured dynamically: http://example.com/robots.txt or http://example.com/sitemap.xml). In this case IIS won’t treat them as files and will add trailing slash: http://example.com/robots.txt/. Also it may cause problems on login view or administrative views if you use ASP.Net MVC (some actions won’t work). Solution is do disable this rule for these views. In order to avoid mentioned problems we need to add several additional conditions:

   1: <add input="{REQUEST_FILENAME}" matchType="Pattern" negate="true"
   2: pattern="robots\.txt" />
   3: <add input="{REQUEST_FILENAME}" matchType="Pattern" negate="true"
   4: pattern="sitemap\.xml" />
   5: <add input="{REQUEST_URI}" matchType="Pattern" negate="true" pattern="^/admin.*" />
   6: <add input="{REQUEST_URI}" matchType="Pattern" negate="true" pattern="^/users/.*" />
   7: <add input="{REQUEST_URI}" matchType="Pattern" negate="true"
   8: pattern="^/packaging/.*" />

This example contains conditions for administrative views for Orchard CMS. Here I excluded internal urls “/admin”, “/users”, “/packaging” from the rule (these urls are used in Orchard).

Note that also I used lowercase rule names with underscores, you may use more user friendly names for your rules. I used to this syntax :) That’s all what I wanted to write about in this article. Hope that it will be useful for you.

Sunday, December 2, 2012

Add Youtube widget to Sharepoint site using OTB Content editor web part

In the previous post I showed how to add Twitter widget to the Sharepoint site using only OTB Sharepoint features. In this article I will show how to add widget for another popular service – Youtube. The idea is the same – we will use Content editor web part and file with widget code uploaded to the document library. We need to use external file because Content editor will trim javascript code if you will put it to the web part directly.

First of all we need to find appropriate widget. From those widgets which I saw, the following was the most suitable: Youtube gadget. This is Google module which was developed by one of the Google employees (I found only “Mark L” name in the module’s metadata), but I’m not 100% sure in it, see below. Before to use it you should keep in mind the following disclaimer from Google which is shown in the bottom part of the page:

Much of the content in this directory was developed by other companies or by Google's users, not by Google. Google makes no promises or representations about its performance, quality, or content. Google doesn't charge for inclusion in this directory or accept payment for better placement.

Also I should warn you that I made conclusion that Mark L is Google’s employee only based on the metadata of the module which can be found here: http://www.google.com/ig/modules/youtube.xml as it contains the following field:

author_affiliation="Google"

I’m not sure does Google verifies this info or not, so use it at your own risk. Anyway if you will check module’s code (in the link above), you will see that all used js files are from Google.

Ok, first step is to specify Title and Channel:

image

After that click Get the code button below and copy html code from the text field:

image

Save the copied code to the local file with .txt extension, e.g. youtube.txt. After that perform the same steps which are described in the post about Twitter widget. I.e. upload file to the doclib, ensure that it is published with major version and approved, add Content editor web part to the page and put url of the uploaded file in the Content link web part property. After that you should see Youtube channel in your Sharepoint site:

image

Save the page and enjoy.

Saturday, December 1, 2012

Add Twitter widget to Sharepoint site using OTB Content editor web part

In this article I will show how to add Twitter widget to the Sharepoint site without programming using OTB Content editor web part. In Sharepoint 2010 you can’t just copy widget code to the web part, because it will trim it due to security reasons. You have to perform several additional steps, which will be shown below.

First of all you need to create your widget in the Twitter. Go to https://twitter.com/settings/widgets and click Create new:

image

In the opened window specify Twitter username (which tweets you want to add on your site) and domain name of the site where you will add this widget:

image

If you want to add this widget to several sites, specify several domain names separated by comma. Also here you may specify other settings for the widget, like height, theme (Light and Dark available at the moment) and some other settings which are shown on the picture above.

When you will specify all settings you need, click Create widget button below. In the opened window copy html code of the widget:

image

Save copied html code to the local file with .txt extension, e.g. twitter.txt. After that go to Sharepoint site where you want to add the widget and upload twitter.txt to some document library. When it will be done, copy url of the uploaded file (right click on the file and select Copy shortcut from context menu). If minor versions are turned on in the doclib settings, ensure that file is published with major version. Also if you use Content approval workflow, ensure that file is approved.

Go to the page where you want to add the widget and add Content editor web part to some web part zone. Then click Edit web part, and set specified copied url of the file to the Content link property:

image

Click Ok. After that tweets should be displayed on the page:

image

Don’t forget to save the page itself :).

Sunday, November 25, 2012

Host site on Windows Azure virtual machine

There are several actions which you need to do in order to host your public site on the Windows Azure virtual machine. First of all you need to create your VM instance in Azure panel. You may find many tutorials how to do that, so I won’t repeat it here.

Once you will do that, add Web server role in the Server manager:

image

If your site is running on ASP.Net 3.5 you have to select it as well as shown on the picture above. Then create site in IIS like you always do and assign host header bindings for you domain name.

Then go to Azure management panel, click on your virtual machine and select Endpoints tab above:

image

Here you need to add endpoint for http traffic. In order to do that click Add endpoint button below:

image

When you will create it use the following values: Protocol = TCP, Public port = 80, Private port = 80. At the end you should see the following endpoints in your panel:

image

After that you need to configure DNS A-record in the DNS editor of your hosting provider. For this you need to know public IP address by which your site will be available. It can be found in Azure panel. Click on your virtual machine and in the Dashboard tab you will find Quick glance panel. On this panel you will see Public virtual IP address:

image

This is exactly what you need to use for the A-record. According to information from MS (e.g. here: Overview of Managing Deployments in Windows Azure) virtual IP won’t be changed during service lifetime:

Throughout the lifetime of a deployment, the VIP assigned will not change, regardless of the operations on the deployment, including updates, reboots, and reimaging the OS. The VIP for a given deployment will persist until that deployment is deleted. When a customer swaps the VIP between a stage and production deployment in a single hosted service, both deployment VIPs are persisted. A VIP is associated with the deployment and not the hosted service. When a deployment is deleted, the VIP associated with that deployment will return to the pool and be re-assigned accordingly, even if the hosted service is not deleted. Windows Azure currently does not support a customer reserving a VIP outside of the lifetime of a deployment.

After these steps will be done, you need to wait until DNS servers will be updated with new IP. After that you will have your site running on the Windows Azure.

Fix “HTTP Error 500.19 - Internal Server Error” when install ASP.Net MVC 3 site on IIS 8

Some time ago I moved MVC 3 site to the Windows Server 8 with IIS 8. Right after installation I got the following error:

HTTP Error 500.19 - Internal Server Error
The requested page cannot be accessed because the related configuration data for the page is invalid.

Site worked properly on Windows Server 2008 R2 with IIS 7.5, so web.config was correct. The problem was in missing URL rewrite module for IIS. In the site I used several rewrite rules which are stored in web.config as you probably know. As URL rewrite module was not installed on the server, whole section with rewrite rules was not recognized and IIS shown above error. After installation of URL rewrite module, site became working.

This is not ASP.Net MVC specific problem. It may occur for general ASP.Net sites as well. So hope that this tip will help you if you will get the same problem.

Problem with context site in Sharepoint when call web service from javascript

If you need to call web service (here I will talk about old asmx web services) from javascript in ASP.Net or Sharepoint you may use auto-generated js proxy which is added to html output when you add ServiceReference to the ScriptManager instance for the current page:

   1: var scriptManager = ScriptManager.GetCurrent(this.Page);
   2: if (scriptManager == null)
   3: {
   4:     scriptManager = new ScriptManager();
   5:     this.Controls.Add(scriptManager);
   6: }
   7:  
   8: var referenceProxy = new ServiceReference();
   9: referenceProxy.Path = "/_layouts/test/test.asmx";
  10: referenceProxy.InlineScript = true;
  11:  
  12: scriptManager.Services.Add(referenceProxy);

Your web service’s class should be marked with [ScriptService] attribute, and all web methods which you want to use from javascript should be marked with [ScriptMethod] attribute:

   1: namespace MyNamespace
   2: {
   3:     [ScriptService]
   4:     [WebService(Namespace = "http://example.com/", Name = "Test")]
   5:     [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
   6:     [ToolboxItem(false)]
   7:     public class Test : WebService
   8:     {
   9:         [ScriptMethod]
  10:         [WebMethod]
  11:         public string Foo(string str)
  12:         {
  13:             // ...
  14:         }
  15:     }
  16: }

After that you can call web service from javascript like this:

   1: MyNamespace.Test.Foo("Hello, world!");

However in Sharepoint you may encounter with the problem when use this method. Check the line 9 in the code above which adds service reference to script manager.

   1: referenceProxy.Path = "/_layouts/test/test.asmx";

Path contains the url to the service. As you can see it is located in layouts folder, i.e. it can be called in context of any Sharepoint site. Also we specified that generated proxy should be added as inline script to the output html (referenceProxy.InlineScript = true). In this case we need to use relative url in the ServiceReference.Path property as it is said in the documentation:

If the InlineScript property is set to true, then you must use a relative path that points to the same Web application as the page that contains the ServiceReference instance.

In our example path is relative, so it should be Ok. However if this code runs not on the root site, but on some sub site (e.g. http://example.com/subsite) we will have problem: relative path which we specified will be combined with root url, i.e. web service will be executed in context of root site, but not in context of the sub site. I.e. full url will be the following: http://example.com/_layouts/test/test.asmx. It may cause different problems, e.g. if locale of subsite differs from locale of the root site your users may see incorrectly localized content.

In order to fix it first of all we need to comment the line which sets InlineScript to true. It will allow us to use absolute url in the Path property:

   1: string referenceProxyUrl = SPUrlUtility.CombineUrl(SPContext.Current.Web.Url,
   2:     "/_layouts/test/test.asmx");
   3:  
   4: var referenceProxy = new ServiceReference();
   5: referenceProxy.Path = referenceProxyUrl;
   6: //referenceProxy.InlineScript = true;
   7:  
   8: scriptManager.Services.Add(referenceProxy);

These changes will have the following effect. Proxy will be added as external js file via <script> tag. Src attribute will contain absolute url:

   1: <script src="http://example.com/subsite/_layouts/test/test.asmx/jsdebug" type="text/javascript"></script>

(it added “/jsdebug” to the absolute url to the asmx). As you can see js proxy is loaded from context of the correct sub site now. So problem is solved? Unfortunately no. When you will check SPContext.Current.Web.Url property in the debugger of the web method, you will see that code still runs in context of the root site (it will contain http://example.com url, instead of http://example.com/subsite). And this is regardless of the site in which content js proxy was loaded.

In order to fix it we need to perform one extra step. Open the url which is specified in the src script. You will see the code of the proxy. It contain line which we are interesting in:

   1: MyNamespace.Test.set_path("/_layouts/test/test.asmx");

set_path is internal method generated on the client side only, i.e. there is no such method in our web service. As you can see it still contains relative url. This is exactly what we need. Before to use web service proxy in the javascript, we need to override relative url by absolute. We can get absolute url on server side as shown above using standard SPUrlUtility.CombineUrl() method and then pass it to javascript. After that call set_path with passed value:

   1: MyNamespace.Test.set_path(absoluteUrlFromServerSide);
   2: MyNamespace.Test.Foo("Hello, world!");

And it fixes the problem finally. Now the code of asmx web service will be executed in the context of the sub site. Hope that it will help you. E.g. you will encounter with this problem if will use framework for loading web parts asynchronously which I wrote above in the following post: Create asynchronous web parts for Sharepoint.

PS. However the described solution doesn’t fix problem with different locales which I used as example :) Some time ago I wrote how Sharepoint sets locale of the current thread using language of requested SPWeb: see this post. But because of some reason it didn’t happen in this case. So I fixed problem with locale manually:

   1: namespace MyNamespace
   2: {
   3:     [ScriptService]
   4:     [WebService(Namespace = "http://example.com/", Name = "Test")]
   5:     [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
   6:     [ToolboxItem(false)]
   7:     public class Test : WebService
   8:     {
   9:         [ScriptMethod]
  10:         [WebMethod]
  11:         public string Foo(string str)
  12:         {
  13:             this.ensureCorrectLocale();
  14:             // ...
  15:         }
  16:  
  17:         private void ensureCorrectLocale()
  18:         {
  19:             if (Thread.CurrentThread.CurrentUICulture.LCID !=
  20: SPContext.Current.Web.UICulture.LCID)
  21:             {
  22:                 Thread.CurrentThread.CurrentUICulture =
  23: SPContext.Current.Web.UICulture;
  24:             }
  25:             if (Thread.CurrentThread.CurrentCulture.LCID !=
  26: SPContext.Current.Web.Locale.LCID)
  27:             {
  28:                 Thread.CurrentThread.CurrentCulture =
  29: SPContext.Current.Web.Locale;
  30:             }
  31:         }
  32:     }
  33: }

After that current thread’s locale became correct.