Nov 18
How *not* to hash passwords
icon1 Darrell Mozingo | icon2 Misc. | icon4 November 18th, 2010| icon3No Comments »

We were stupid back in the day (OK, a year or two, but who’s counting?). When we started our latest project it was a given that we’d be hashing passwords for storage. The most obvious and easiest way to do it was the good ‘ol (password + hash).GetHashCode(). Done and done. We moved on to the next feature and never gave it a second thought.

As it turns out though, using GetHashCode() for password hashing purposes is, well, pretty stupid and irresponsible. GetHashCode() was never intended to be stable across .NET versions or even architectures (x86 vs x64), and apparently the framework spec documents call this out. In fact, its results have changed slightly between .NET 3.5 and 4.0, which is what we were just upgrading to when I noticed this. Similar changes aparently occurred between 1.1 and 2.0 too.

For example, the GetHashCode() hash of the string “password” from .NET 3.5 is -733234769, while the hash from that exact same string in .NET 4.0 is -231203086. Scary, huh?

In light of that, we switched to using the SHA512Managed class to generate our hashes. Switching our code over wasn’t an issue (DRY for the win!), but having to email our customers to enter new passwords and security questions, which we also hashed the same way, wasn’t exactly fun. Not knowing their passwords apparently does have a downside! Here’s how we’re generating our hash codes now:

private const string _passwordSalt = "some_long_random_string";
 
public static string CalculateSaltedHash(string text)
{
	var inputBytes = Encoding.UTF8.GetBytes(text + _passwordSalt);
	var hash = new SHA512Managed().ComputeHash(inputBytes);
 
	return Convert.ToBase64String(hash);
}

Yay? Nay?

Nov 12

In the first post I gave a quick overview of what our deployment script does and why you’d want one. This post will go over some of the pre-deployment steps we take. Most all of this code will probably be pretty self explanatory, but I know just having something to work off of is a huge boost to starting your own, so here ya go.

function modify_web_config_for_production($webConfig)
{
	echo "Modifying $webConfig for production deployment."
 
	$xml = [xml](Get-Content $webConfig)
	$root = $xml.get_DocumentElement();
 
	$root."system.web".compilation.debug = "false"
 
	$xml.Save($webConfig)
}

Given the path to a web.config file, this function switches off the debug flag (and any other changes you’d need). Being a dynamic language, you can access XML keys quite easily. You’ll need the quotes around system.web since there’s the dot in the name though. Also, if you need access to any of the app.settings keys, you can use something like: $xml.selectSingleNode('//appSettings/add[@key="WhateverYourKeyIs"]').value = "false".

function precompile_site($siteToPreCompile, $compiledSite)
{
	echo "Precompiling $siteToPreCompile."
 
	$virtual_directory = "/"
 
	exec { & $tools_aspnetCompiler -nologo -errorstack -fixednames -d -u -v $virtual_directory -p "$siteToPreCompile" $compiledSite }
}

This little beauty precompiles the site (located in the $siteToPreCompile directory, with the results output to the $compiledSite directory) using the ASP.NET compiler. I prefer to copy the actual compiler executable into the project folder even though it’s installed with the Framework. Not sure why. Anyway, $tools_aspnetCompiler can either point locally, or to C:\Windows\Microsoft.NET\Framework\vwhatever\aspnet_compiler.exe. You can also configure the options being passed into the compiler to suit your needs.

function execute_with_secure_share($share, [scriptblock]$command)
{
	try
	{
		echo "Mapping share $share"
		exec { & net use $share /user:$security_full_user $security_password }
 
		& $command
	}
	finally
	{
		echo "Unmapping share $share"
		exec { & net use $share /delete }
	}
}

This is more of a helper method that executes a given script block (think of it as an Action or anonymous code block in C#) while the given share is mapped with some known username and password. This is used to copy out the site, create backups, etc. I’ll leave the $security_full_user & $security_password variable declarations out, if you don’t mind! We just put them in plain text in the build script (I know, *gasp!*).

properties {
	$share_web = "wwwroot"
	$servers_production = @("server1", "server2")
	$live_backup_share = "\\server\LiveSiteBackups"
 
	$number_of_live_backups_to_keep = 10
}
 
function archive_current_live_site
{
	$current_datetime = Get-Date -Format MM_dd_yyyy-hh_mm_tt
	$one_of_the_production_servers = $servers_production[0]
 
	$web_share_path = "\\$one_of_the_production_servers\$share_web"
 
	echo "Archiving $web_share_path"
 
	$full_backup_path = "$web_share_path\*"
	$full_archive_file = "$live_backup_share\$current_datetime.zip"
 
	execute_with_secure_share $web_share_path {
		execute_with_secure_share $live_backup_share {
			exec { & $tools_7zip a $full_archive_file $full_backup_path } 
		}
	}
}
 
function delete_extra_live_site_backups
{
	execute_with_secure_share $live_backup_share {
		$current_backups = Get-ChildItem $live_backup_share -Filter "*.zip" | sort -Property LastWriteTime
		$current_backups_count = $current_backups.Count
 
		echo "Found $current_backups_count live backups out there, and we're aiming to keep only $number_of_live_backups_to_keep."
 
		$number_of_backups_to_kill = ($current_backups_count - $number_of_live_backups_to_keep);
 
		for ($i = 0; $i -lt $number_of_backups_to_kill; $i++)
		{
			$file_to_delete = $current_backups[$i]
			$extra_backup = "$live_backup_share\$file_to_delete"
 
			echo "Removing old backup file: $extra_backup"
			delete_file $extra_backup
		}
	}
}

These pair of methods create a backup of the current live site and make sure we’re only keeping a set number of those backups from previous runs, to keep storage and maintenance in check. Nothing too complicated. To create the backup, we just farm out to 7-Zip to compress the directory, which is ran withing nested execute_with_secure_share calls from above, which map the web server file share and backup file share. Likewise, the second method just gets a count of zip files in the storage directory and deletes the oldest ones in there until the total count gets to a specified count.

Conclusion

That’s the basics for what we do pre-deployment. Again, not really that complicated, but it can give you a starting point for your script. I’ll go over our actual deployment steps in the next post, then follow that up with some post-deployment goodness. I know, you can’t wait.

Sep 24

Pushing to production with a script? Crazy talk, right? Well, maybe not. Sure, there are lots of insane corporate setups out there where a script might not completely work, but for the vast majority of companies out there, this is totally within the realm of possibility. Maybe you want to save some steps when you’re deploying, or maybe you want to stop forgetting some of those crazy “best practices” people always talk about (building in release mode? turning off debugging in the web.config? pre-compiling sites?). Whatever the reason, a deployment script is a great solution.

What it does

Our current deployment script will:

  1. Get a completely clean copy of the code base from source control
  2. Build it in release mode
  3. Run all unit tests, slow/integration tests, and UI tests
  4. Switch off debugging in the web.config
  5. Pre-compile the site with the ASP.NET compiler
  6. Archive the current live site to a backup server, just in case (keeping a few previous versions as well)
  7. Deploy the latest copy of our 3rd party tools to each server
  8. XCopy deploy the site to each server in our cluster (taking down IIS first and letting our load balancer get users off that server)
  9. Visit the site with a browser to do all the first time pre-load reflection stuff (NHibernate, AutoMapper, StructureMap, etc)

    1. It’ll actually change its local DNS hosts file to make sure its looking at each server in the cluster too, so that each one is “primed”
  10. Make sure our error emails are working by visiting a test URL that throws an exception (therefore creating an error email), then logging into a special email account and making sure that email was received

OK, so this script takes a while to run (with all the tests taking up a majority of the time), but we gain a lot. A single click in TeamCity kicks the whole thing off, and we’re guaranteed little to nothing is broken in the system thanks to all the tests (unit, integration, and UI), that there’s backup copies if something does happen, and that everything is compiled/configured for production so we’re not missing any easy performance gains. I’d say that’s a win.

How it’s ran

We don’t have have this running in any automatic fashion, but instead run the target by hand from our build server whenever we’re ready. Our build server lets us easily schedule the “build” whenever we need to, though, so we can schedule it late at night so we don’t disrupt access. Our test servers are also being setup right now, so we’ll probably continuously deploy to those when they’re ready (twice a day? every check-in?).

Fail safes

There honestly aren’t a whole lot. As we come across certain failures we’ll add checks to keep them from cropping back up, but I didn’t want to complicate the script with all sorts of edge case checking if it’ll never need to worry about them. You need to apply the KISS and YAGNI principals to your build scripts just like your code. We do a few operations in try/catch blocks to make sure servers are kept down if they’re not deployed to correctly, or our 3rd party tools get cleaned up properly, etc., but not many.

I’m sure that’ll unsettle many of you, but a script like this is going to be highly customized to your environment, and your environment might have a higher chance of certain parts of the script failing, so you’ll need to guard against that. I’d highly recommend starting simple and only building out as situations arise though.

Build server bonuses

We use TeamCity as our build server, so I can’t speak about the others (CC.NET, Hudson, etc) and how much or little of the following benefits they offer.

The two biggest benefits we get, virtually for free, with using TeamCity to run our deployment script include:

  • Auditing – you can see who’s ran the script, and when
  • Tracking – it’ll show you, for each run, which changes were included in that deployment down to a diff of each file and source control comments
    • It’ll also show which changes are pending for the next run: Pending Changes in TeamCity
    • We don’t use a bug tracker that’s supported by TeamCity, but theoretically it can link right to each bug fix that’s included in each deployment

What’s next?

I’m going to show off parts of our build script and how we handle different situations in the next blog post(s). I’m not sure how many it’ll take or how deep I’ll go since much of it is situation specific, but I’ll definitely get you started on the road with your own.

As a heads up, this will all be written in PowerShell. We’ve moved our build script setup to it and it’s what made this deployment scenario possible.

Conclusion

Manual deployment sucks. Automated deployment rocks.

If there’s any way you can script your deployment (or even parts of it), I’d recommend it in a heart beat. It’s a bit bumpy getting it up and running, I won’t lie, but it’s a huge help once it’s stable and working. I’ll take a look at some of the basic pre-deployment steps we take in the next post.

Sep 2
Comments are just missed refactorings
icon1 Darrell Mozingo | icon2 Musings | icon4 September 2nd, 2010| icon32 Comments »

Ah comments, those delectable little nuggets of static information that routinely get out of sync with your code’s intention. Actually, I’m sort of excited when I run into them though. See, to me they represent missed refactorings that are usually pretty easy to implement.

Say you run across a piece of code like this:

public void ProcessOrder(Order order)
{
	// If the order is on hold, figure out why and get the next availability date.
	if (order.Status == OrderStatus.OnHold && order.OrderDate < DateTime.Now)
	{
		// <imagine complex logic to check reason here...>
		order.OnHoldReason = (result of complex logic above);
 
		// <imagine complex logic to get the next available date here...>
		order.NextAvailableDate = (result of complex logic above);
	}
}

So it’s usually not this obvious in real code bases, but almost every time I’ve run into a grouping of comments in this arrangement it could be boiled down into something this simple. Can you see the refactoring potential? It’s pretty easy – extracting some variables and methods based upon the text in the comment:

public void ProcessOrder(Order order)
{
	var orderIsOnHold = (order.Status == OrderStatus.OnHold && order.OrderDate < DateTime.Now);
 
	if (orderIsOnHold)
	{
		order.OnHoldReason = getOnHoldReason(order)
		order.NextAvailableDate = getNextAvailableDate(order);
	}
}
 
public string getOnHoldReason(Order order)
{
	return // <imagine complex logic to check reason here...>
}
 
public DateTime getNextAvailableDate(Order order)
{
	return // <imagine complex logic to get the next available date here...>
}

Like I said, it’s not much and it’s certainly not hard (a few keystrokes in Visual Studio, with or without ReSharper), but it moves the comments from sitting idly above the code, not participating, to being true first class citizens in the program in the form of variable and method names. It doesn’t guarantee they’ll be updated with the code if the logic changes in the future, but it gives them a lot better shot at it. I mean, there aren’t really developers out there lazy enough to update the intention of a variable or method and not change the name, right? Right?

Aug 26
Controlling IIS7 remotely with PowerShell
icon1 Darrell Mozingo | icon2 Misc. | icon4 August 26th, 2010| icon31 Comment »

Our deployment script needed to do some basic IIS administrative tasks remotely on a Windows 2008 (non-R2) server, which runs IIS7, recently. Finding the information and fiddling around with it took me a good day and a half, so I thought I’d post the steps here to help someone else (more than likely myself) in the future:

  1. Download the Windows Management Framework Core package for your setup
  2. If your machine is something older than Windows 7 or Server 2008 R2, you’ll need to get the PowerShell IIS7 Snap-In
  3. If your workstation/build server and target web servers happen to be on different Windows domains, you’ll need to run this one time on each client machine:
    Set-Item WSMan:\localhost\Client\TrustedHosts *
  4. Run this command once on each server:
    winrm quickconfig
  5. You’ll need to load the PowerShell Snap-In once on each client, which differs depending on which version of Windows you’re running. Anything older than Windows 7 or Server 2008 R2:
    Add-PSSnapin WebAdministration

    Windows 7 and Server 2008 R2 run:

    Load-Module WebAdministration
  6. Check if it’s working properly by running this command on any version of Windows (you should see the IIS7 Snap-In listed):
    Get-Module -ListAvailable
  7. Get credentials for accessing the remote server
  8. Start running some remote commands on your web servers:
    Invoke-Command -ComputerName "webserver_computerName" -Credential $credentials_from_last_step -ScriptBlock { Add-PSSnapin WebAdministration; Stop-Website IIS_Site_Name }

It’s not really that hard once you get the proper packages installed and the permissions worked out, and since it’s so powerful and useful for scripting purposes it’s well worth the trouble. The available commands are awesome for use in automated deployment scripts.

You can learn more about the PowerShell Snap-In provider here, and at its download site here.

Aug 19
Clever vs Explicit
icon1 Darrell Mozingo | icon2 Musings | icon4 August 19th, 2010| icon31 Comment »

When we all start out developing, either through classes in high school/college or slowly on our own time, we inevitably want to write code thats super clever. Like, use-3-lines-to-express-what-used-to-take-20-lines type of clever. We see less code and we’re pleased. All is good and well with the world.

Until you look at that code a few months down the road and wounder what the hell you were smoking, and it takes you almost as long to decipher it again as it did to write it in the first place. All of a sudden saving those few extra lines of code don’t seem so smart, huh?

The simple fact of the matter is you spend more time maintaining code than you do writing it in the first place, so being more explicit in your code always trumps being clever to save a few lines. Always.

There’s the obvious places where people get clever, like algorithms or loops, but there’s plenty of other places too. Places where I wouldn’t really call it “being clever”, or at least I’m sure the original authors never thought they were trying to be clever when they wrote it. It was probably just quicker to write it in a certain way. For example, take this code:

var mappedEmployees = new List<EmployeeViewModel>();
 
foreach (var employee in _employeeRepository.All().Where(x => x.IsRetired == false && x.Salary > 100000))
{
	mappedEmployees.Add(_employeeMapper.Map(employee));
}
 
return View(mappedEmployees);

It’s not really hard to read, but it’s not really easy either. It might take you an extra second or two to figure out what’s going on when you first look at (even if you wrote it a few months ago), but multiply that by how many places you see code like this and how often you go back in to modify it (for new features, bugs, whatever). It adds up, quick. Written more explicitly, it might look something like this:

var mappedEmployees = new List<EmployeeViewModel>();
var nonRetiredHighEarningEmployees = _employeeRepository.All().Where(x => x.IsRetired == false && x.Salary > 100000);
 
foreach (var nonRetiredHighEarningEmployee in nonRetiredHighEarningEmployees)
{
	var mappedEmployee = _employeeMapper.Map(nonRetiredHighEarningEmployee);
	mappedEmployees.Add(mappedEmployee);
}
 
return View(mappedEmployees);

You might call it verbose, but I’d say it’s a net gain. Each line is doing one thing. Yon can step through and read it without mentally pulling pieces apart. None of this “OK, that’s the mapped object call there, and its return is going into the collection there, and the whole thing is looping through that query there”. Things are given names and methods aren’t nested inside each other.

Always be on the lookout for “clever” areas in your code. Be explicit. Try to stick to each line doing one thing so there’s no hidden surprises.

May 21
OCP in action
icon1 Darrell Mozingo | icon2 Design Principles | icon4 May 21st, 2010| icon3No Comments »

What is OCP?

The Open/Closed Principle is the O in the nearly infamous SOLID set of design principles. The gist of the principle is that any given class should be open for extension, but closed for modification. In more practical terms, you should be able to make most classes do something new or different without actually changing a single line of code within the class itself.

That’s simple! Right? Ummm, sure. How, exactly, is a class supposed to be able to do more stuff without being modified? Well, there’s lots of ways, if you think about it. I mean, that’s sort of what inheritance and polymorphism were made for. There’s also other pretty neat ways, too. One of them shows the true beauty of inversion of control, and dependency injection frameworks.

Lets say you have a collection of customers. You need to loop through these people, running a few different checks on them (only preferred customers can carry a negative balance, credit cards on file aren’t expired, etc), and adding any resulting errors from those checks to another collection for display later. Nothing too terribly complicated.

Solving without considering OCP

First we have to load up the customers and send them on through:

private static void Main(string[] args)
{
	var customers = get_customers_from_somewhere();
 
	var check_runner = new Check_runner();
	var warnings = check_runner.run_checks_on(customers);
 
	foreach (var warning in warnings)
	{
		Console.WriteLine(warning);
	}
 
	Console.ReadLine();
}
 
private static IEnumerable<Customer> get_customers_from_somewhere()
{						// database, webservice, whatever.
	return new[]
	       	{
	       		new Customer
	       			{
					name = "Joe Smith",
	       				credit_card = new Credit_card { is_valid = true },
	       				balance = 100,
	       				is_preferred = true
	       			},
	       		new Customer
	       			{
					name = "Nathan Hawes",
	       				credit_card = new Credit_card { is_valid = false },
	       				balance = 0,
	       				is_preferred = true
	       			},
	       		new Customer
	       			{
					name = "Melinda Plunkett",
	       				credit_card = new Credit_card { is_valid = true },
	       				balance = -100,
	       				is_preferred = false
	       			}
	       	};
}

The class running all the checks:

public class Check_runner
{
	private static readonly IList<string> _warnings = new List<string>();
 
	public IEnumerable<string> run_checks_on(IEnumerable<Customer> customers)
	{
		foreach (var customer in customers)
		{
			check_that_only_preferred_customer_can_have_a_negative_balance(customer);
			check_that_on_file_credit_card_is_not_expired(customer);
			// additional checks in the future...
		}
 
		return _warnings;
	}
 
	private static void check_that_on_file_credit_card_is_not_expired(Customer customer)
	{
		if (customer.credit_card.is_valid)
		{
			return;
		}
 
		_warnings.Add("Credit card expired for customer: " + customer.name);
	}
 
	private static void check_that_only_preferred_customer_can_have_a_negative_balance(Customer customer)
	{
		if (customer.is_preferred || customer.balance >= 0)
		{
			return;
		}
 
		_warnings.Add("Negative balance for non preferred customer: " + customer.name);
	}
}

Pretty standard. Loop through the customers, calling a separate private method for each check you need to preform, adding a message for that check’s error to a collection that ultimately gets returned to the caller for display.

The problems with the first approach

At first blush, this way of running the checks might seem very simple and understandable, but it starts to break down for a few different reasons:

  • New checks could get pretty complicated, requiring access to other expensive objects (repositories, web services, file I/O, etc). Even if only one check needed a certain dependency, the whole Check_runner class is now burdened with that dependency.
  • Every new check requires you to open up the Check_runner class and making a modification. Opening a class and modifying it? That’s pretty much the definition of an OCP violation. Modifying a class always introduces the possibility for regression bugs. No matter how small the possibility or bug, they’re there.
  • Testing this thing as it gets larger and larger is going to be a pain in the rear. It’ll also get much harder, especially if outside dependencies are brought in (having to setup multiple dependencies when the one check you’re testing doesn’t even use them isn’t fun, or clear to read later).

One possible solution

There’s a few different ways to go about fixing this. My suggestion would be to break each check into its own individual class, with the Check_runner taking them all in and looping through them, running each as it goes. It sounds a little more black-magicy than it really is. I’m going to show all the code first, then go over the benefits of an approach like this later on. Lets start by defining an interface for these check classes:

public interface ICustomer_check
{
	string buildWarningFor(Customer customer);
	bool failsFor(Customer customer);
}

Now we can define a single check, which knows when it fails for a given customer, and how to build a warning message for that failure. The check classes for the two checks that are ran above would be a simple conversion of the existing code:

public class Negative_balance_check : ICustomer_check
{
	public string buildWarningFor(Customer customer)
	{
		return "Negative balance for non preferred customer: " + customer.name;
	}
 
	public bool failsFor(Customer customer)
	{
		return (!customer.is_preferred && customer.balance < 0);
	}
}
 
public class Expired_credit_card_check : ICustomer_check
{
	public string buildWarningFor(Customer customer)
	{
		return "Credit card expired for customer: " + customer.name;
	}
 
	public bool failsFor(Customer customer)
	{
		return (customer.credit_card.is_valid == false);
	}
}

Now the Check_runner just has to loop through all of the ICustomer_check implementations and run them:

public class Check_runner
{
	private readonly IEnumerable<ICustomer_check> _customer_checks;
 
	public Check_runner(IEnumerable<ICustomer_check> customerChecks)
	{
		_customer_checks = customerChecks;
	}
 
	public IEnumerable<string> run_checks_on(IEnumerable<Customer> customers)
	{
		var warnings = new List<string>();
 
		foreach (var customer in customers)
		{
			foreach (var check in _customer_checks)
			{
				if (check.failsFor(customer))
				{
					warnings.Add(check.buildWarningFor(customer));
				}
			}
		}
 
		return warnings;
	}
}

Again, pretty simple and focused. Where does that enumeration of ICustomer_check implementations come from though? The missing key: our dependency injection framework. I’ll use StructureMap for this example. After downloading that and referencing the assembly, we’ll modify our main method to set it up:

private static void Main(string[] args)
{
	ObjectFactory.Initialize(y => y.Scan(x =>
	                                     {
	                                     	x.TheCallingAssembly();
						x.AddAllTypesOf<ICustomer_check>();
	                                     }));
 
	var customers = get_customers_from_somewhere();
 
	var check_runner = ObjectFactory.GetInstance<Check_runner>();
	var warnings = check_runner.run_checks_on(customers);
 
	foreach (var warning in warnings)
	{
		Console.WriteLine(warning);
	}
 
	Console.ReadLine();
}

We fire up StructureMap, telling it to scan the calling assembly and find all implementations of ICustomer_check. When we ask for an instance of Check_runner, StructureMap knows to provide all the implementations it found of ICustomer_check to Check_runner‘s constructor argument in a list. Since this is the outer most edge of the application, we’ll interact with the dependency injection framework here instead of inside Check_runner.

Benefits

So perhaps other than the StructureMap related code (if you don’t already know the basics of it), nothing I’ve done here has really complicated the system. It’s still a few primitive classes working together in a fairly obvious way. What benefits do we gain from these changes though?

  • Each piece of the system now has a single, specific responsibility. You can look at each check and quickly figure out what its purpose is. The runner simply takes in all the checks and runs them (funny how its name now follows its responsibility too).
  • The check classes can now take in their own dependencies. Need an ICustomerRepository or ICustomerAccountService for something? List it in the constructor. Each check is getting pulled from the container, so their dependencies will get filled as well. Checks will also only take on what each one needs, as opposed to requiring dependencies they might not have before.
  • With decreased responsibilities, each piece now becomes much easier to test. Supply a list of dummy checks and dummy customer to make sure the runner is doing its job. Same for the checks themselves. In fact, too many tests for a class is a smell that class is doing too much in the first place.
  • The point of the article: no more OCP violations! Future requirements for different kinds of checks now become almost mind numbingly easy. Slap in a new class and implement ICustomer_check. That’s it – the container will take care of the rest. Virtually no possibility of introducing a regression bug and messing up one of the other checks by adding a new one.

Conclusion

One thing to remember when looking for OCP violations in your code base is that “closed for modifications” should be taken within context. Fixing bugs, adding complete new features, etc, will obviously require modifications to something. You’re not going to create every class in your system and never touch them again. Within reason, you should apply the Open-Closed Principle to your code as much as possible. It makes it simpler to understand on the micro and macro level once your familiar with some of the more common patterns, and it helps reduce the possibility of introducing bugs from future additions.

The code from this post is available here.

Apr 27

I finally got around to implementing help screens on our site recently. We needed a system that would enable our domain peeps to update the help text directly with no intervention from us, along with being easy to implement and maintain on our end. I ended up using flat HTML files and a jQuery modal dialog (Colorbox), which has support for asynchronously loading those HTML files from disk when needed. The one thing we didn’t want to do with this solution was give our domain peeps production server access or the responsibility of keeping those HTML files up to date on the servers – I could only imagine the chaos that’d ensue from that.

Solution: use our build script & build server to handle it for us.

We gave our domain peeps commit access to the repository – thankfully we’re still on SVN, as I’m sure their heads will explode when we switch to a DVCS. This provides nice versioning and accountability features if someone messes up (imagine that), and gives us a hook for the build server. All help files are contained in a hierarchy under a folder that’s appropriately named HelpFiles. I checked out just that folder from the source tree on their machines and gave them a quick commit/update spiel. We created empty HTML files for them, and they went about their way filling them all in.

Now on to the more interesting part, our build script. As I’ve mentioned previously, we’re using psake. Here’s the relevant properties and task:

properties {
	$scm_hidden_dir = ".svn";
 
	$executing_directory = new-object System.IO.DirectoryInfo $pwd
	$base_dir = $executing_directory.Parent.FullName
 
	$source_dir = "$base_dir\src"
	$build_dir = "$base_dir\build"
	$build_tools_dir = "$build_dir\tools"
 
	$share_web = "wwwroot"
	$servers_production = @("server1", "server2")
 
	$security_user = "user_with_write_access"
	$security_password = "pa55w0rd"
 
	$tools_robocopy = "$build_tools_dir\robocopy\robocopy.exe"
 
	$help_folder = "HelpFiles"
	$help_local_dir = "$source_dir\$project_name.Web\$help_folder"
	$deployTarget_help = "$project_name\$help_folder"
}
 
task publish_help {
	foreach ($server in $servers_production)
	{
		$full_server_share = "\\$server\$share_web"
 
		exec { & net use $full_server_share /user:$security_user $security_password }
 
		& $tools_robocopy $help_local_dir $full_server_share\$deployTarget_help /xd $scm_hidden_dir /fp /r:2 /mir
 
		# See page 33 of the help file in the tool's folder for exit code explaination.
		if ($lastexitcode -gt 3)
		{
			Exit $lastexitcode
		}
 
		exec { & net use $full_server_share /delete }
	}
}

There’s an array of production server names, which we iterate over and use the net command built into Windows to map its wwwroot share using a different username & password than the current user (this allows the build server to run as an unprivileged user but still access needed resources).

Then we use the surprisingly awesome Robocopy tool from Microsoft, which is basically xcopy on steroids, to copy out the help files themselves. The xd flag is excluding the hidden .svn folders, the fp flag is displaying full path names in the output (for display in the build output from TeamCity later), the r flag is telling it to only retry failed file twice (as opposed to the default million times!), and the mir flag is telling it to mirror the source directory tree to the destination, including empty folders and removing dead files.

We can’t use psake’s built in exec function to run Robocopy, as exec only checks for non-zero return codes before considerng it a failure. Of course, just to be different, Robocopy only fails if its return code is above 3 (1 = one or more files copied successfully, 2 = extra files or folders detected, and there is no 3). So we check the return code ourselves and exit if Robocopy failed. We then delete the share, effectively making the machine forget the username/password associated with it.

With that done, we created a new build configuration in TeamCity and had it check the repository for changes only to the help file directory by adding +:src/Project.Web/HelpFiles/** to the Trigger Patterns field on the Build Triggers configuration step.

That’s pretty much it. Our domain peeps have been pretty receptive to it so far, and they love being able to edit the help files, commit, and see them live only a minute or two later. We loved not having to pull all that text from the database on each page load and not having to create editing/viewing/versioning/etc tools around such a process. It’s a win-win.

Apr 3
2010 Goals – April Update
icon1 Darrell Mozingo | icon2 Goals | icon4 April 3rd, 2010| icon3No Comments »

First quarter’s down. In full disclosure fashion, here’s how I’m doing so far:

Books

  1. Code Complete – Steve McConnellNot started.
  2. Patterns of Enterprise Application Architecture – Martin FowlerNot started.
  3. Applying Domain-Driven Design and Patterns – Jimmy Nilsson
  4. Working Effectively With Legacy Code – Michael FeathersNot started.

Tools/Techniques/Processes @ Work

  • Move from SVN to Git (and learn more about Git in the process). – Learning about Git and bugging our sys admin about setting up Git, but haven’t actually started yet.
  • Move from CC.NET to Team City for our build server.
  • Build a more robust build script and management process – including production deployment and database migration scenarios. – Well underway.

Involvement

  • Develop an idea I was given for an open source project and get it live to see what happens. – In progress.
  • At least 24 blog posts (I’m not going to say 2 per month as I’m getting married this summer and I’m certain I won’t be able to maintain a schedule around it). – 4 down (ouch!).
  • At least 3 feature/patch submissions to open source projects. – Not started.

Coding

  • Get a version 1 out there on at least 1 of the 3 product ideas I have floating around. – Not started.
  • Keep working on a good working knowledge of Ruby & Ruby on Rails (and use it to build the product mentioned above). – Playing and reading.

So-so. Lets see about next quarter.

Apr 2

A while back I wrote a small series on creating a basic build script and setting up a build server (part 1, part 2, and part 3). I used NAnt and CruiseControl.NET in that series, but alluded to a few other options for each. I recently got around to switching our build script from NAnt to psake, which is written in PowerShell, and switching our build server from CruiseControl.NET to JetBrain’s TeamCity. I’ll give a quick overview of our new build script here, which I’ll use to build on in future posts showing a few of the more interesting things that suddenly became much easier with this setup, and in a few cases, possible at all.

To start with, you’ll want to make sure you have the latest versions of PowerShell (2.0) and psake. Here’s the basics of our build script:

$ErrorActionPreference = 'Stop'
 
Include ".\functions_general.ps1"
 
properties {
	$project_name = "MainApplication"
	$build_config = "Debug"
}
 
properties { # Directories
	$scm_hidden_dir = ".svn";
 
	$executing_directory = new-object System.IO.DirectoryInfo $pwd
	$base_dir = $executing_directory.Parent.FullName
 
	$source_dir = "$base_dir\src"
 
	$build_dir = "$base_dir\build"
	$tools_dir = "$base_dir\tools"
	$build_tools_dir = "$build_dir\tools"
 
	$build_artifacts_dir = "$build_dir\artifacts"
	$build_output_dir = "$build_artifacts_dir\output"
	$build_reports_dir = "$build_artifacts_dir\reports"
 
	$transient_folders = @($build_artifacts_dir, $build_output_dir, $build_reports_dir)
}
 
properties { # Executables
	$tools_nunit = "$tools_dir\nunit\nunit-console-x86.exe"
	$tools_gallio = "$tools_dir\Gallio\Gallio.Echo.exe"
	$tools_coverage = "$build_tools_dir\ncover\ncover.console.exe"
	$tools_coverageExplorer = "$build_tools_dir\ncover_explorer\NCoverExplorer.Console.exe"
}
 
properties { # Files
	$solution_file = "$source_dir\$project_name.sln"
 
	$output_unitTests_dll = "$build_output_dir\$project_name.UnitTests.dll"
	$output_unitTests_xml = "$build_reports_dir\UnitTestResults.xml"
	$output_coverage_xml = "$build_reports_dir\NCover.xml"
	$output_coverage_log = "$build_reports_dir\NCover.log"
	$output_coverageExplorer_xml = "$build_reports_dir\NCoverExplorer.xml"
	$output_coverageExplorer_html = "$build_reports_dir\NCover.html"
}
 
properties { # Skip coverage attributes
	$skipCoverage_general = "Testing.SkipTestCoverageAttribute"
}
 
task default -depends unit_test_coverage
 
task clean {
	$transient_folders | ForEach-Object { delete_directory $_ }
	$transient_folders | ForEach-Object { create_directory $_ }
}
 
task compile -depends clean {
	exec { msbuild $solution_file /p:Configuration=$build_config /p:OutDir=""$build_output_dir\\"" /consoleloggerparameters:ErrorsOnly }
}
 
task unit_test_coverage -depends compile {
	exec { & $tools_coverage $tools_nunit $output_unitTests_dll /nologo /xml=$output_unitTests_xml //reg //ea $skipCoverage_general //l $output_coverage_log //x "$output_coverage_xml" //a $project_name }
	exec { & $tools_coverageExplorer $output_coverage_xml /xml:$output_coverageExplorer_xml /html:$output_coverageExplorer_html /project:$project_name /report:ModuleClassFunctionSummary /failMinimum }
}

As the second line alludes to, you can break functions out into separate files and include them back into the main one. Here’s functions_general.ps1:

function delete_directory($directory_name)
{
	Remove-Item -Force -Recurse $directory_name -ErrorAction SilentlyContinue
}
 
function create_directory($directory_name)
{
	New-Item $directory_name -ItemType Directory | Out-Null
}

This script will build our project and run the unit tests, producing a coverage report we can display later inside Team City. Much of this maps loosely one to one against the NAnt version discussed in my past series, and there’s plenty of articles/posts online explaining this stuff in much more detail than I can here. Note that all the pieces that can “fail” the script are wrapped in exec, which will execute the code block (i.e. lambda/anonymous delegate) and basically alert the build server if it fails. Not too difficult, at least for now 🙂

As for getting this to work with Team City, if you specify the runner as a command line and point it at a batch file with these contents:

@echo off
cls
powershell -Command "& { Set-ExecutionPolicy Unrestricted; Import-Module .\build\tools\psake\psake.psm1; $psake.use_exit_on_error = $true; Invoke-psake '.\build\build.ps1' %*; Remove-Module psake}"

You’ll be golden. This batch allows the build server to run the script (perhaps setting unrestricted execution isn’t the smartest from a security standpoint, but oh well), sets up the psake environment, tells psake to raise its warnings in a way that TeamCity can pick up on, executes your build script, and tears down the psake environment. Looks a little complicated, but it’s just a bunch of smaller commands strung together on one line, and you shouldn’t have to look at it again.

« Previous Entries Next Entries »