config file “configuration file is not well-formed xml”

I was fiddling with the config file on the web server, but I had a good reason.  That is, until I broke it.  Let me start from the beginning.

I am using load-balancing for IIS.  It is very cool, except for a few things.  One of them is the viewstate in ASP.NET pages.  Here is why: By default, IIS will recycle the app pools to refresh the memory, etc. (a good thing).  Also, every time an app pool (in IIS) starts-up/recycles, it generates a new encryption key pair for encrypting/encoding the viewstate. It is also a good idea, because fresher keys are harder to crack.  The down-side to refreshing the keys is that, when it happens, the old keys become invalid.  If somebody is still using them.  Oh well.

So, imagine that you are a user and you are on a page and it is working good.  A minute or two goes-by and you click next and the page just errors out.  You wonder what happened.  Was it something that you did? The error shows up in .net as “Validation of viewstate MAC failed”.  It means that a user just sent a request to your server and that request had a viewstate that cannot be decrypted with the current key.

If you have load-balanced servers, the problem is even worse because every additional server will have its own encryption keys (for Viewstate) and each server will have its own start-up/recycle intervals.  This means two or three MORE times that the users will see this kind of error. 

Just in-case this isn’t bad enough, the same error also occurs when a user gets load-balanced to another server.  Since each server generates their own keys and does not share them.  Imagine you are a user.  You get page 1 from server 1 with key abc123, you click next, the load balancer sees that server 1 is really busy, so it sends you to server 2.  Server 2 has different viewstate keys (efg456):  So, you get a MAC error.  Once again, you (the user) don’t realize that you just got load-balanced and you have no idea what just went wrong. 

Since, every decent programmer wants to fix all errors, I was not able to rest until this was fixed.  The solution to this mess is to generate a permanent set of encryption keys for IIS and manually set those on each server. This article on MSDN covers it very well: http://msdn.microsoft.com/en-us/library/ms998288.aspx

To generate some good machineKey values, I downloaded some free code on CodeProject.com.  http://www.codeproject.com/Articles/16645/ASP-NET-machineKey-Generator

The Break

I could have applied the machineKey information in each web.config file, but I thought it might be easier (for other hosted sites, on the same server), if I just set the machineKey in the machine.config, effectively applying it to all apps on the server.  Then it is a no-brainer for the other teams that need to deploy onto those servers. 

The machine.config file is typically located in the folder named C:\windows\Microsoft .NET\Framework\4.0.*\config\ (32-bit) and C:\windows\Microsoft .NET\Framework64\4.0.*\config\.  Of course, you will also need to set it in the config files for Framework*\2.0.*\… as well, depending on which level of .net you are using.

As soon as I made the change and hit save, the site went down.  OMG!  Panic or don’t panic?!

The first thing I checked was that the syntax (on the only line that I changed) was correct.  I compared the new machineKey to the format that MSDN (and all other documentation) recommended:

<machineKey
validationKey=”21F090935F6E49C2C797F69BBAAD8402ABD2E…
  …AD972A119482D15A4127461DB1DC347C1A63AE5F1CCFAACFF”
decryptionKey=”ABAA84D7EC4BB56D75D217CECFFB962880…9B”
validation=”SHA1″
decryption=”AES”
/>

(of course, I didn’t use that EXACT key, and neither should you.  Please generate your own keys!)

I started looking in the NT System error logs.  They indicated that the app pool had crashed because of a 503 error on the site.  The message said “configuration file is not well-formed xml“.  Wait a sec.  That is not true!  Um, as you can see, it IS well-formed XML.  But arguing with an event log is pointless.

It was a real head-scratcher, because all of the documentation implied that I was doing everything right. 

The Solution

It turns out that IIS will only accept the machineKey tag if it is an encapsulating tag instead of a singular tag.  So I changed the tag to look like this (instead of the previous example) and it worked.

<machineKey
validationKey=”21F090935F6E49C2C797F69BBAAD8402ABD2E…
  …AD972A119482D15A4127461DB1DC347C1A63AE5F1CCFAACFF”
decryptionKey=”ABAA84D7EC4BB56D75D217CECFFB962880…9B”
validation=”SHA1″
decryption=”AES”>
</machineKey>   <– That fixed it!

I restated the IIS pool and my app was happy once again.  There were no more MAC errors when the load-balancer shuffled me around or the pools cycled.

Advertisements

About Tim Golisch

I'm a geek. I do geeky things.
This entry was posted in ASP.NET, Errors, Programming. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s