The first post gave a quick overview of what our deployment script does and why you’d want one, the second post went over pre-deployment steps, and the third post in this series covered the actual site’s deployment. This post will go over a few of the post-deployment steps we take after publishing our site. Like the last posts, most all of this code will probably be pretty self explanatory.
Preloading
We make heavy use of StructureMap, NHibernate (w/Fluent NHibernate), and AutoMapper in our system, and those guys have some heavy reflection startup costs. Since it’s all done when the app domain starts, we hit each server in our farm to “pre-load” the site for us because that first visit takes a good 30-40 seconds because of those tools.
Since the servers are in a farm, we can’t just go to the site’s URL as we’d only get one box - even multiple loads aren’t guaranteed to move you around to them all. To make sure we’re looking at each server, we fiddle with the build server’s hosts file and point it at each web server. We don’t do parallel builds on our build server, so we thankfully don’t have any issues with other build scripts getting tripped up, but you may want to consider that if it’s applicable to your situation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
| properties {
$hosts_file = "C:\Windows\System32\drivers\etc\hosts"
$servers_production = @( "server1", "server2" )
$servers_production_ip = @{ "server1" = "192.168.1.1"; "server2" = "192.168.1.2" }
}
function setup_hosts_file_for($server, $url)
{
$server_ip = $servers_production_ip[$server]
echo "Setting hosts file to use $server_ip ($server) for $url."
"$server_ip $url" | Out-File $hosts_file -Encoding Ascii
}
function remove_hosts_file_entries
{
echo "Removing all hosts file entries and reverting to a clean file."
"127.0.0.1 localhost" | Out-File $hosts_file -Encoding Ascii
}
function make_sure_we_are_pointing_at($server, $url)
{
$expected_server_ip = $servers_production_ip[$server]
$ping_output = & ping -n 1 $url
$ip_pinged = ($ping_output | Select-String "\[(.*)\]" | Select -ExpandProperty Matches).Groups[1].Value
if ($ip_pinged -ne $expected_server_ip)
{
throw "The site's IP is supposed to be $expected_server_ip, but it's $ip_pinged (for $url). Hosts file problem?"
}
echo "Correctly pointing at $ip_pinged for $url."
}
function stop_dns_caching
{
& net stop dnscache
}
function start_dns_caching
{
& net start dnscache
} |
The hosts file allows you to point any request for, say, www.asdf.com on your machine to whatever IP you want. So if you wanted to preload www.asdf.com for server1, you can put “192.168.1.1 www.asdf.com” in your hosts file, and you’ll always hit that machine. Your load balancing setup might not allow this though. There’s also a method that’ll ping the given URL to make sure it’s going to the proper server, throwing up if it isn’t. The last two methods start/stop the DNS Caching service in Windows, just to help make sure we’re looking at the correct IP for a given URL.
With that setup, we can easily manipulate IE through COM to pull up the site:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
| properties {
$live_site_text_in_title = "Our cool site"
$times_to_try_preloading_sites = 50
}
function fire_up_ie
{
return New-Object -Com InternetExplorer.Application
}
function preload_url_on_server($server, $url)
{
setup_hosts_file_for $server $url
make_sure_we_are_pointing_at $server $url
$current_attempt_count = 0
$3_seconds = 3
$ie = fire_up_ie
$ie.navigate($url)
echo "Pulling up $url in the browser."
while ($current_attempt_count -lt $times_to_try_preloading_sites)
{
pause_for $3_seconds
$document = $ie.document
if ($document -ne $null -and $document.readyState -eq "Complete" -and $document.title -match $live_site_text_in_title)
{
$time_taken = ($current_attempt_count + 1) * $3_seconds
echo "Preloaded $url on $server in about $time_taken seconds."
break
}
$current_attempt_count++
}
$ie.quit()
if ($current_attempt_count -ge $times_to_try_preloading_sites)
{
throw "$url (on $server) couldn't be preloaded after a pretty long ass wait. WTF?"
}
} |
Working with IE’s COM interface is pretty painless in PowerShell. Dynamic languages FTW, aye? We just fire up IE, browse to the URL (which should be pointing to the given server only), and keep checking on IE’s progress until the page is fully loaded the title contains some piece of text we expected it to. Simple and to the point.
The first snippet in Part 3 of this series showed how we deployed the site. You can see there where we temporarily stop the DNS Caching service, then pre-load the site on each server are deploying to it, then reset the hosts file and start the DNS Caching service again.
Testing Error Email Generation
We have some basic code to email exceptions out if our app hits an exception. Nothing fancy. To test our error emails are getting sent OK, I created an obscure URL in the application that’ll just generate a TestErrorEmailException. When our error handler sees that exception, all it does it send the generated error email to a buildserver@domain.com address rather than the normal one. The build script then logs into it’s special GMail accont and checks for the email. This is bar far the chunckiest part of the build script:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
| properties {
$email_url = "mail.ourdomain.com"
$error_generation_path = "/SomeObscurePath/GenerateTestErrorEmail/?subject="
$max_email_check_attemps = 100
}
function wait_for_browser_to_finish($ie)
{
while ($ie.busy -eq $true) {
pause_for 1 #second
}
}
function generate_test_error_emails_on($server, $base_url, $error_email_subject)
{
setup_hosts_file_for $server $base_url
make_sure_we_are_pointing_at $server $base_url
$error_url = $base_url + $error_generation_path
$full_error_url = $error_url + $error_email_subject
$ie = fire_up_ie
$ie.navigate($full_error_url)
echo "Generating test error email from $full_error_url."
wait_for_browser_to_finish $ie
$ie.quit()
}
function ensure_error_emails_are_working_on($server, $base_url)
{
echo "Ensuring error emails are getting sent out correctly on $server."
$current_datetime = Get-Date -Format MM_dd_yyyy-hh_mm_tt
$error_email_subject = "Error_" + $server + "_$current_datetime"
generate_test_error_emails_on $server $base_url $error_email_subject
check_email_was_sent $error_email_subject
}
function check_email_was_sent($expected_email_subject)
{
echo "Pulling up $email_url in the browser."
$ie = fire_up_ie
$ie.navigate($email_url )
wait_for_browser_to_finish $ie
logout_of_email $ie
echo "Logging in to email."
$ie.document.getElementById("email").value = $security_user
$ie.document.getElementById("passwd").value = $security_password
$ie.document.getElementById("signin").click()
wait_for_browser_to_finish $ie
echo "Looking for test error email."
$test_error_email = $null
for ($i = 1; $i -le $max_email_check_attemps; $i++)
{
echo "Attempt #$i checking for the test error email."
$test_error_email = get_link_containing_text $ie $expected_email_subject
if ($test_error_email -ne $null)
{
echo "Found the test error email."
break
}
pause_for 10 #seconds
echo "Refreshing the page after a pause."
click_link_with_text $ie "Refresh"
}
if ($test_error_email -eq $null)
{
$ie.quit()
throw "Test error email was never received after $max_email_check_attemps attempts. Problem?"
}
echo "Pulling up the test error email."
$ie.navigate($test_error_email.href)
wait_for_browser_to_finish $ie
echo "Deleting test error email."
click_link_with_text $ie "Delete"
logout_of_email $ie
$ie.quit()
}
function logout_of_email($ie)
{
$signout_link = get_link_with_text $ie "Sign out"
if ($signout_link -ne $null)
{
echo "Signing out of email."
$ie.navigate($signout_link.href)
wait_for_browser_to_finish $ie
}
}
function click_link_with_text($ie, $text)
{
$link = get_link_with_text $ie $text
$there_are_multiple_links_with_that_text = ($link.length -gt 1)
if ($there_are_multiple_links_with_that_text)
{
$ie.navigate($link[0].href)
}
else
{
$ie.navigate($link.href)
}
wait_for_browser_to_finish $ie
}
function get_link_with_text($ie, $text)
{
return $ie.document.getElementsByTagName("a") | where { $_.innerText -eq $text }
}
function get_link_containing_text($ie, $text)
{
return $ie.document.getElementsByTagName("a") | where { $_.innerText -match $text }
} |
It seriously looks worse than it really is, and most of it is due to navigating around GMail’s interface. So we hit the obscure URL in our app, pass it a subject line for the error email, wait a bit, then log into GMail and check for an email with that same subject line. If we don’t find the email after a waiting period, we blow up the script. Simple as that.
If you know an easier way to do this, I’m all ears!
Conclusion
The two biggest things we do after deploying our site is, for each individual server in the farm, load it up so all the first time reflection stuff can get taken care of and make sure any errors on the site are getting emailed out correctly. While controlling IE through its COM interface is a lot cleaner and easier with PowerShell, there’s still some code for navigating around GMail’s site. Obviously if you use a different setup for your email, you’ll either have to control a different app or access the SMTP server directly.
Unfortunately, the biggest piece for both of these things being helpful is if you can navigate to each server in the farm. If your network setup prevents that, it’s not going to do you much good unless you keep clearing your cookies and revisiting the site a bunch of times in hopes you’ll get each server, or something crazy like that.
So while most of this code is straight forward, I hope it’ll give you a starting point for your deployment script. Like I said in the beginning: it’s a bit painful to initially setup (both creating it and testing it), but we’ve found huge value from having it in place. It’s obviously not as easy as Capistrano, but, meh, it works. Another option for .NET is Web Deploy, a relatively new tool from Microsoft. I haven’t had time to get too deep into it, but it may help for your situation.
Good luck!
Darrell Mozingo | 
December 22nd, 2010| 
