A common question I see out there is if the RPC Client Access Service (including Client Access Service Arrays) can access databases in other sites. The answer is, yes. Let’s take a look at a couple scenarios.

Scenario #1 – Full Site Failure

Let’s say you have a Client Access Server Array called array.domain.com.  Primary Site goes down.  As a part of the manual site switchover process, you must update the DNS records in your Primary Site to point to the CAS infrastructure at your DR Site.  One out of several DNS records you change will include the CAS Array. You change array.domain.com to point to DRSiteCAS instead of PrimarySiteCAS.  The client (after the DNS record flushes – recommended for TTL value to be 5 minutes for DNS records in site resilient solutions) will then start to connect to the DRSiteCAS which will then access the database in the DR Site.

Scenario #2 – Server Failure(s) in Primary Site and Disabling Automatic Activation for Databases and Servers

In the case where all database copies go down in the Primary Site, your databases can automatically failover to the DR Site as long as you allow automatic activation on the DR Servers (yes, you can turn off automatic activation on databases and servers) and as long as you still have Majority for your Quorum. In this scenario, the RPC Client Access (and array) can access the mailbox databases that are mounted in the DR Site.

Automatic Activation

As I just eluded to above, it is possible to turn off automatic activation on databases and servers. There is something called Database Activation Policy.  Let’s say you wanted to disable a specific database from being considered in the Automatic Activation Process.

You can use the following command to prevent the database from being considered in the Automatic Activation Process:

This example resumes the copy of the database DB1 on the server MBX2 for automatic activation:

This is also possible to do at the mailbox server level using the Set-MailboxServer cmdlet.  You can use the following command to prevent any databases on a specific mailbox server from being considered in the Automatic Activation Process:

This example resumes all database copies on the mailbox server “MailboxServer” for automatic activation:

Example

Let’s say we have 6 DAG Servers with 4 in the Primary Site and 2 in the DR Site with no modifications to the Automatic Activation Policy (DAG Servers in the DR Site can automatically mount databases).  Let’s say, we currently have a lack of funds for storage which prohibit the ability to have mailbox database copies on all servers.  So PrimarySiteMBX01 and PrimarySiteMBX02 in the Primary Site are mirrored in terms of mailbox database copies.  PrimarySiteMBX03 and PrimarySiteMBX04 in the Primary Site are mirrored in terms of database copies.  PrimarySiteMBX01 and PrimarySiteMBX02 are mirrored with SecondarySitMBX0102 in the DR Site and PrimarySiteMBX03 and PrimarySiteMBX04 are mirrored with SecondarySiteMBX0304 in the DR Site.

To make it a bit more clear, the following image shows database distribution.  You can see there are 6 nodes and 3 copies of each database.

Should PrimarySiteMBX01 and PrimarySiteMBX02 go down (as illustrated below), SecondarySiteMBX0102 can automatically mount the database because majority is still there for quorum.  In this case, the RPC Client Access Array in the Primary Site will still successfully be able to provide mailbox access to the databases mounted on SecondarySiteMBX0102 in the DR Site.  This is one of the nice things I like about Exchange 2010 High Availability, is that if your DAGs go down, you can allow the copy in the DR Site to automatically activate (provided the Database Activation Policy as described above allows it to automatically mount) whereas in Exchange 2007, you had to manually activate any SCR copy.

Exchange 2007 and Exchange 2010 Clusters both use Majority Node Set Clustering.  This means that 50% of your votes (server votes and/or 1 file share witness) need to be up and running.  With DAGs, if you have an odd number of DAG nodes in the same DAG (Cluster), you have an odd number of votes so you don’t have a witness.  If you have an even number of DAGs nodes, you will have a file share witness in case half of your nodes go down, you have a witness who will act as that extra +1 number.

So in this scenario, we have 6 votes from the servers plus 1 witness from the file share witness totaling 7 votes.  This means we can have up to 3 servers fail and our cluster will still be online.  This is because if you are in the scenario where we 7 votes, if 3 go down that leaves us with 4 votes which satisfies the 50% + 1 majority rule. Because of this, we still have majority and our quorum and cluster are still fully operational.

Now when exactly would we have to do a manual switchover?  Well, there’s a couple cases.  The first would be if your Primary Datacenter has a complete outage.  This may be due to power failure, environmental disaster, etc…  The other is because all Primary Datacenter DAG members go down or just enough servers go down (again, 50% + 1 voters must be up which means if we lose more than 3 machines (includes FSW), our entire cluster goes offline.  In this case, you’ll have to do a manual datacenter switchover.  You’ll move over all services to the secondary datacenter including changing the RPC Client Access Server FQDN to point to the single CAS Server or the standby VIP that publishes RPC across multiple Secondary Datacenter CAS Servers.

Yes, this is old news and there’s about 462 blog entries (ok, that’s a made up number, but there are a lot) about how to force Communicator 2007 R2 to do an Address Book (Galcontacts.db) update.  These blog entries will talk about the July 2009 update for Communicator 2007 R2 and how it introduced a random delay of 0-60 minutes for Communicator 2007 R2 to download an updated GalContacts.db to prevent the network from getting hammered by so many clients downloading an updated GalContacts.db all at the same time.  And yes, these blog entries also talk about a registry entry you can create called GalDownloadInitialDelay and creating a Dword set to 0 in order to force Communicator to do an instant update.

Some blog articles that talk about this include:

http://www.tincupsandstring.com/2009/12/01/forcing-address-book-download/

http://www.markc.me.uk/MarkC/Blog/Entries/2009/12/17_Force_Downloading_the_Address_Book_in_OCS.html

Now I’m sure you are asking yourself why I am creating this entry?  Is it just to repeat information that’s already out there?  Of course not!

So, Communicator 2007 R2 is a 32-bit (x86) application.  That registry entry works perfectly fine on x86 systems.  But, if you are running on a x64 system, it won’t.  Why?  Well, because when you run x86 applications on a x64 based system, it utilizes a system in Windows called Windows on Windows (WOW64).  WOW64 has its own section within the registry called Wow6432Node.

So let’s say we take the registry key for our Communicator x86 (Communicator x64 not available) and run it on an x86 system.  The following registry key works fine:

But let’s say we have an x64 system.  The above registry key will not work.  We need to utilize the WOW6432Node part of the registry.  The following registry key works for x64 systems:

Please make sure you back up your registry before making changes as making changes to the registry can be harmful to your system if not done properly.

Exchange 2010 introduces a new moderation feature.  This moderation feature can be used to perform moderation on both messages submitted to a Distribution Group and/or requests to join a group through the self-service options provided by the Exchange Control Panel (ECP).

The Issue

For both the Distribution Group Mail Flow Message Moderation as well as the Membership Approval Moderation were both grayed out for us.  I did a dump of these two groups by running the following two commands:

Distribution Group Mail Flow Message Moderation

Membership Approval Moderation

The Fix

After taking a look at both .txt files, I noticed that the user that was specified under ManagedBy no longer exists.  I changed the ManagedBy to an existing user and all the Moderation feature options lit up and were no longer grayed out.

Keep in mind, you absolutely will need to have a Manager specified in order for the moderation features to light up.  This makes sense with the Membership Approval tab as it only allows you to use Group Managers.  For the Message Moderation, it would seem that Microsoft should not gray everything out by default when you don’t have a Group Manager specified since it does allow you to specify specific people for moderation instead of using the Group Manager.

You can use the following command to modify all your groups in bulk:

Update (1/20/2010) - I ran into another issue today where the moderation options were grayed out even with a valid group manager.  I changed the manager and changed it back and the moderation features were lit up.

Update (2/17/2010) – I forgot to update this with another issue after my 1/20/2010 update.  After updating the Group Manager for a group, the Address Book service in OCS seems to consider this group as an entirely new group.  This means that if people had this group added in OCS and you modified the Group Manager, that group becomes void in Communicator and stops expanding.  I had to end up deleting the GalContacts.db and let a client download a new GalContacts.db.  The client had to delete the group from Communicator and then delete it in order for the group to start working again.

OCS 2007 R2 Standard Edition Front Ends utilize SQL 2005 Express with SP2 for storing its databases.  Unfortunately, with SQL Express, you will have to backup using SQL Server Management Studio or find an automated way.  This article will detail the steps I utilize to make backing up easier and automated. For information on how to back up OCS, please see the Backup and Restoration Guide here.

The following data will ultimately need to be backed up:

1. Global Config
2. Pool Config
3. Machine Config
4. SQL Databases
5. Standard Edition File Shares

The first command specifies the /level to be global and pool.  The second command specifies the /level to be machine.  What we will do is create a batch file (.bat) and place both commands in this .bat and have them run against the server every 6pm using scheduled tasks.

Our Servername is SHUD-OCSFE01.  The folder to store the backups is C:\OCSBackup.  We’ll also be running the batch file from the C:\OCSBackup.  Because the folder which contains lcscmde.exe is not a part of the system variables, we’ll have to specify the entire path for lcscmd.exe. Taking this information into consideration, our two commands for our batch file will be:

After executing this .bat file, we can see the two files have been created.

SQL Databases

The following is the list of SQL Databases that an OCS Standard Edition Front End uses:

Because we are utilizing SQL Express, we will have to find some other method other than a backup agent to automate the backups. Much of the SQL Backup information is provided by the SQLDBATips Blog.  The following article I utilized is located here.

Create a file with the extension of sql in our OCSBackup folder.  Also, create a new folder called C:\Reports for script reporting. I created a file C:\OCSBackup\ocssqlbackup.sql with the following text:

exec expressmaint
@database      = ‘ALL_USER’,
@optype        = ‘DB’,
@backupfldr    = ‘c:\ocsbackup’,
@reportfldr    = ‘c:\reports’,
@verify        = 1,
@dbretainunit  = ‘days’,
@dbretainval   = 1,
@rptretainunit = ‘weeks’,
@rptretainval  = 1,
@report        = 1

exec expressmaint
@database      = ‘ALL_USER’,
@optype        = ‘LOG’,
@backupfldr    = ‘c:\ocsbackup’,
@reportfldr    = ‘c:\reports’,
@verify        = 0,
@dbretainunit  = ‘days’,
@dbretainval   = 1,
@rptretainunit = ‘days’,
@rptretainval  = 1,
@report        = 1

All of our OCS Databases are User Databases, not System Databases.  We can see this using SQL Server Management Studio which is not installed by default but can be downloaded from here.

Note: Keep in mind that we’re not using the default SQL Express instance of SQLExpress.  The OCS Front End Standard install will create and utilize an instance of RTC.

We now have our .SQL file created.  We’ll go ahead and create a new .bat file called ocssqlbackup.bat.  This batch file will run the following command:

This won’t work just yet.  You can see in the .SQL file, it’s calling the stored procedure “expressmaint.”  We need to create this stored procedure within SQL.  SQLDBATips has the vbscript code in order to do that here.  You take this code and save it as storemaint.sql.  Then run the following code:

Note: The website that shows these instructions specify the -S.\ as -S.\SQLExpress.  Again, we’re not using the SQLExpress instance, but rather the RTC instance.

You can delete the expressmain.sql file now.  This is a permanent change in our instance and we won’t need to run the expressmain.sql script again.

We should now be able to run our SQL backup batch file as our .sql command that specifies our databases and logs has been created and our batch file to call sqlcmd.exe to execute our .sql file has been created.

We can see our ocssqlbackup.bat file successfully runs and creates backups of our databases.

Scheduled Tasks

We obviously want to keep backing up our databases every night in case something goes wrong.  We’ll create two scheduled tasks.  One that runs ocsbackup.bat for our global, pool, and machine specific information.  And the other that runs our SQL Backups.

I am launching the Task Scheduler from Server Manager (I am using Server 2008 but you can access Task Scheduler on Windows 2003 by going to Control Panel).

Create a Basic Task and give it a name.  We’ll name this OCS Backup.  Click Next to Continue.

Specify how often you want the task to run.  I typically run it Daily. Utilize whatever method works best for your organization. Click Next to Continue.

Choose what time the Daily Task will run.  Again, choose whatever time works best for your organization. Click Next to Continue.

We’ll want to run the script.  Because of this, choose “Start a program.” Click Next to Continue.

Specify the path to our batch file. Click Next to Continue. Review the Settings and then Click Finish.

You can then forcefully run the Scheduled Task to ensure it runs.

Now don’t forget to create the second scheduled task to run the batch file for SQL Backups!

Your OCSBackup folder should look something like this after your scheduled tasks run and your data is backed up.

Backing up your data to a remote Backup Server

Now what good is having all this data backed up onto the OCS File System if OCS crashes?  No good!  We’ll still want to take your backup system and back up all these files including the OCS Standard Edition File Shares.  Now keep in mind that you will want to back up all of these files at some time after your batch files are set to run in Scheduled Tasks.  For example, my Scheduled Tasks are set to run at 8pm.  The batch files do not take long to run.  You can have your backup set to run at 8:30pm or 9:00pm.  Be sure to test and validate this is working as intended and you are getting successful backups.

The Standard Edition File Shares you will want to backup include:

So to sum it up, you will want back up all the above file locations and your OCSBackup folder.  Backing up your Reports folder is optional. But again, keep in mind you will want to run this file level backup after all your Scheduled Tasks are successfully run.

After recently doing an Exchange 2007 SP2 Upgrade and Rollup 1 for SP2, I encountered an issue with Impersonation.  I’m assuming this happened due to the SP2 Upgrade, not the Rollup, but be cognizant about the issue either way.  For this upgrade, Geomant Message Waiting Indicator (MWI) for Exchange 2007 Unified Messaging was being utilized.  Geomant MWI utilized Impersonation in order to light up the telephone when the user receives a voicemail.  This stopped working after the SP2/RU1 was applied to the system.

The issue

The error message we saw was as follows

2009-12-17 22:56:37,744 ERROR Com.Geomant.Exchange12.MWIService.ExchConnector – Failed to create search folder by using WebService for user: CN=Lastname\, Firstname,OU=OUHERE,DC=domain,DC=tld. Reason: The server to which the application is connected cannot impersonate the requested user due to insufficient permission. [1488]

2009-12-17 22:56:37,760 ERROR Com.Geomant.Exchange12.MWIService.ExchConnector – Failed to subscribe the user: CN=Lastname\, Firstname,OU=OUHERE,DC=domain,DC=tld for Exchange events.  [1488]

The fix

Run the commands to grant the service account impersonation rights as it previously had.  Please refer to the documentation previously used for granting impersonation. In the case of MWI, the following two commands were run:

Welcome to Part 4 of this article series. In Part 1, we started off by discussing the goal of this lab. That goal is how to deploy a two-node Exchange 2010 RTM Database Availability Group (DAG) on Windows Server 2008 R2. We then prepared our Operating System with the Exchange 2010 Prerequisites which includes software prerequisites as well as modification to hardware configuration such as our Network Interface Cards (NIC)s.  In Part 2, we went over the installation of one of our Exchange 2010 Servers which will include the Mailbox, Client Access, as well as Hub Transport Server Roles. In Part 3, we went over the creation and configuration of our Database Availability Group (DAG).  We then added our first Node to our DAG.

In this Part, I will be adding the second node to the DAG and then create a  database which will then be synchronized to the second DAG node.

Part 1

Part 2

Part 3

Part 4

Adding the second Node to our DAG

Well, let’s go ahead and add our first node to the DAG.  Go into the EMC > Organization Configuration > Mailbox > Database Availability Group > Right-Click our DAG > Manage Database availability Group Membership.

Add the second Node.  Click Manage to Continue.

Our second node has successfully been added.

But… what exactly was done during this behind the scenes when this second node was added to the DAG?  The following occurs (from Technet documentation):

• The server is joined to Windows Failover Cluster for the DAG.
• The quorum model is automatically adjusted:
• A Node Majority quorum model is used for DAGs with an odd number of members.
• A Node and File Share Majority quorum is used for DAGs with an even number of members.
• The witness directory and share are automatically created by Exchange when needed.
• The server is added to DAG object in Active Directory.
• The cluster database is updated with info on mounted databases.

First of all,we can see the DAG has been joined to the Windows Failover Cluster.

Second of all,we can see the Quorum Model has been adjusted to Node Majority and File Share Witness because we have an even number of nodes.

We can also see the FSW is set to the location we specified when creating our DAG (SHUD-OCSFE01 with a path of C:\ShudnowDAG) and that there is Quorum data in this location.

Adding Database Replicas

Well, let’s go ahead and create a new database and replicate it.  Go into the EMC > Organization Configuration > Mailbox > Database Management.

We can see there’s currently two databases that were created during the installation on our Exchange Mailbox Servers; one for the first node and one for the second node.

We can’t delete these databases because they contain some arbitration mailboxes.  Arbitration mailboxes are special mailboxes that are used to manage approval workflows.  For example, moderated e-mails.  We can see these arbitration mailboxes and what mailbox databases they belong to by running the following command:

Create a new Database.  I will create a new mailbox database with the name, LABDatabase01.  I will then also mount the database The two commands I will use to do this are:

Let’s add a Mailbox Database Copy to our second DAG node so we have redundant databases.  Database Management > Select the new Database > Right-Click and Choose “Add Mailbox Database Copy.”

Choose the second server for the server that will obtain our Database Copy.  Click Add to Continue.

We should then see a successful copy being added to our second DAG Node.

To verify, in the EMC, click on the LABDatabase01 and we should see a Mounted copy and a Healthy copy below.

To do a switchover, you can right-click on the copied database and choose Activate Database Copy.

DAGs Networks

Go into the EMC > Organization Configuration > Mailbox > Database Availability Group.  At the bottom, you will see the Networks.  You can see both are enabled for Replication.  Exchange 2010 always uses the last recently used replication network.  You can leave both enabled to Replication or you can disable the MAPI Network from having Replication enabled.  This will force all replication to go over your dedicated replication network. Keep in mind, when you do this, your MAPI Network can still do replication.  It will only do replication when there are no dedicated replication networks available.  For example, if the dedicated replicated network were to go down due to some switch but your MAPI network was available, replication would begin to utilize the MAPI network.

If you were in a situation where you were adding a 3rd node to the DAG and it was in a different subnet, you will need to add an IP Address for that subnet so the Network Name resource can come online for that subnet.  So let’s say we now added a 3rd DAG node that was on the 172.16.0.0/12 subnet.  Remember our Set-DatabaseAvailabilityGroup cmdlet with the -DatabaseAvailabilityGroupIpAddresseses switch?  In this case, let’s say 172.16.2.154 was going to be our DAG IP for that subnet.  We would have to add that IP to the switch above.  But that switch is not additive, so we would have to run the following command:

As you can see, I specified both 192.168.1.154 in addition to 172.16.2.154.

What happens is if the DAG fails over to the second DAG node, the DAG will keep the 192.168.1.154 address.  But if it fails over to the 3rd node, it will use the 172.16.2.154.  Again, this command has nothing to do with the replication networks, only the MAPI Networks.  And again, it’s only so the Network Name resource can come online which is a cluster dependency.  No clients will connect to this Network Name resource and Exchange has multiple mechanisms to connect to Exchange.

Summary

Well folks, that is all for Part 4 of this article as well as the article series. Thanks for reading!

Welcome to Part 3 of this article series. In Part 1, we started off by discussing the goal of this lab. That goal is how to deploy a two-node Exchange 2010 RTM Database Availability Group (DAG) on Windows Server 2008 R2. We then prepared our Operating System with the Exchange 2010 Prerequisites which includes software prerequisites as well as modification to hardware configuration such as our Network Interface Cards (NIC)s.  In Part 2, we went over the installation of one of our Exchange 2010 Servers which included the Mailbox, Client Access, as well as Hub Transport Server Roles.

In this Part, I will go over the creation and configuration of our Database Availability Group (DAG).  We will then add our first Node to our DAG.

Part 1

Part 2

Part 3

Part 4

File Share Witness

We will be using a File Share Witness (FSW) on a non-Exchange Server.  This will go on our member server, SHUD-OCSFE01.

We will need to go onto our Member Server and add the “Exchange Trusted Subsystem” group to our Local Administrators Group.  If you do not do this, you will get an Access Denied error message. Unlike Exchange 2007, we do not have to pre-create the FSW.  We tell Exchange 2010 where the FSW will be located, and because the Exchange Trusted Subsystem is added to the non-Exchange box, it will have the permissions necessary to create, modify, and maintain the FSW.

It is still recommended to place the FSW on a Hub Transport Server.  In fact, if you don’t specify the FSW location (Witness Server and Witness Directory), Exchange 2010 will automatically go out and look for a Hub Transport Server and choose a location on its own.  Alternatively, you can specify the Witness Directory and not the Server; in which case Exchange 2010 will automatically choose an Exchange 2010 Server (non-DAG Hub Transport Server preferred) on its own but use the Directory you specified.

Creating the DAG using the EMC and assigning a Static IP

Open the Exchange Management Console (EMC) and go into Organization Management > Mailbox.  Click on Database Availability Group. As you can see, it’s currently empty.

Right-Click on the empty space and choose ” New Database Availability Group.”

Let’s give our DAG a name.  For purposes of this lab, I used the name ShudnowDAG.  As stated, we want SHUD-OCSFE01 to be our witness server and our directory will be C:\ShudnowDAG.  Click New to Create our DAG.

The DAG is successfully created.  At this time, an empty objecting representing the DAG with the name you specified and an object class of msExchMDBAvailabilityGroup is created in Active Directory. You can see the object in either ADSIEdit or LDP.  The DN for this object is:

The completion page will show a warning  informing you that the server that contains the FSW is not an Exchange Server.  We know this already as it was our intention to do this all along.

When the GUI is used to create a DAG, it uses DHCP for the Network Name cluster resource.  If we want to specify a static IP for our DAG, we need to use the Set-DatabaseAvailabilityGroup cmdlet.  The Set-DatabaseAvailabilityGroup cmdlet has a switch called -DatabaseAvailabilityGroupIpAddresseses.  This switch will and should never contain IP Addresses for your replication network.  This -DatabaseAvailabilityGroupIPAddresses switch is ONLY for your MAPI Network subnets so the Network Name resource can come online due to a dependency in the Failover Clustering Services among some other Exchange related functions.

Among some other Exchange related functions eh?  I knew you would ask! Read on…

The Network Name resource is what is used to update the password for the Cluster Name Object (CNO) which is the DAG$ computer account.  The Network Name resource does not have to necessarily be online for Exchange to operate properly, but if it’s not, the DAG$ computer account will expire.  Obviously this would not be a good thing and will cause bad things to happen.

There are also parts of the code that will attempt to connect to the Network Name resource (such as DAG member modifications), but if that fails, those pieces of code will fall back to the servername once the network timeout occurs.

Exchange also utilizes the Possible Owners of the Network Name resource for moving the Primary Active Manager (PAM) which is the server that has control of the default Cluster Group which essentially monitors database status and makes the decisions on what server mounts which database.  For more information about the Active Manager, click here.

So moving on… as you can see, our DAG is using DHCP which is denoted by the <> characters.

So taking a look at our first node, we see our MAPI Network is on the 192.168.1.0 subnet due to the IP Address being 192.168.1.152/24.  Our second node is on the same subnet.

We currently have 192.168.1.154 free so we will use that static IP for our DAG.  It’s not absolutely necessary to use a static IP, but if you feel the need to use a static, feel free.

Now that we have our static IP chosen, let’s run the following command:

We now see that our DAG has the following static IP configured.

Creating the DAG using the EMS and assigning a Static IP

Using the EMS is much faster.  Instead of doing all the above, all you need to do is run the following command:

See? Much faster than using the EMC.  This will definitely be the method I am going to be using in the future to create a DAG when using a static IP instead of DHCP.

Adding the first Node to our DAG

Well, let’s go ahead and add our first node to the DAG.  Go into the EMC > Organization Configuration > Mailbox > Database Availability Group > Right-Click our DAG > Manage Database availability Group Membership.

Add the first Node.  Click Manage to Continue.

Our first node has successfully been added.

But… what exactly was done during this behind the scenes when this first node was added to the DAG?  The following occurs (from Technet documentation):

• The Windows Failover Clustering component is installed, if it is not already installed.
• A failover cluster is created using the name of the DAG.
• A cluster network object (CNO) is created in default computers container.
• The name and IP address of the DAG is registered as a Host (A) record in DNS.
• The server is added to DAG object in Active Directory.
• The cluster database is updated with information on the databases that are mounted on the added server.

First of all,we can see the DAG has been registered in DNS.

Second of all, we can see the DAG’s Cluster Network Object (CNO) has been created in AD.

Third of all, we can see the cluster has been formed.  As you can see, there’s no CMS/Virtual Server in the Services and applications.  This is because Exchange 2010 is not a cluster aware application.  Exchange 2010 only utilizes Windows Failover Clustering Services for heartbeat information and cluster networks.

Finally, we can see that the cluster is currently set to Node Majority.  When we add our second node, the cluster will be switched to Node Majority with File Share Witness since we’ll have an even number of Exchange Nodes and will need a 3rd node/share to act as our witness.  Because of this, we won’t see any FSW data inside of FSW share until our second node is added to the DAG.

Summary

Well folks, that is all for Part 3 of this article. For Part 4, I will be adding the second node to the DAG and then create a  database which will then be synchronized to the second DAG node.

Welcome to Part 2 of this article series. In Part 1, we started off by discussing the goal of this lab. That goal is how to deploy a two-node Exchange 2010 RTM Database Availability Group (DAG) on Windows Server 2008 R2. We then prepared our Operating System with the Exchange 2010 Prerequisites which includes software prerequisites as well as modification to hardware configuration such as our Network Interface Cards (NIC)s.

In this Part, I will go over the installation of one of our Exchange 2010 Servers which will include the Mailbox, Client Access, as well as Hub Transport Server Roles.

Part 1

Part 2

Part 3

Part 4

Installation

With Exchange 2010, we still have the setup.com for unattended mode installations using the Command Line Interface (CLI) as well as setup.exe for attended mode installations using the Graphical User Interface (GUI). We’ll be using the GUI for purposes of this lab.

After running setup.exe, we’ll be presented with the following screen:

We can see that the first two steps are already taken care of.  If you recall from Part 1, we used PowerShell to take care of the prerequisite installations.  So, let’s proceed to Step 2 and choose our language.  For me, it will be English.

When clicking on the language option, we get a couple choices.

If you choose the first option, Install all languages from the language bundle, you will be provided with an option to download the language pack or use an already downloaded language pack.  For purposes of this lab, we’ll choose the second option as we’ll only be using English.

It’s finally time to choose Step 4 and Install Exchange!

So let’s go ahead and choose Step 4 and let’s begin installing Exchange.

After some initializing, we’re provided with the Installer GUI.  The first page, as you guessed it, an Introduction Page.  Read the Introduce Page and Click Next to Continue.

You are now provided with the License Agreement.  After reading the agreement, select “I accept the terms in the license agreement.” Click Next to Continue.

You are now provided with the Error Reporting page.  I like to choose Yes for this option.  The reason why is when you call into Premiere Support Services (PSS), they will have some error reporting information from your servers that may assist with the troubleshooting/fixing of your server.  Choose whichever option best fits your needs.  Click Next to Continue.

You are now provided with the Installation Type.  Previously, in Exchange 2007 CCR/SCC, you could only install the Mailbox Server role.  Now, with DAGs, you can have HUB/CAS/MBX/UM all on the same server.  We’ll be choosing the Typical Exchange Server Installation for this lab which includes HUB/CAS/MBX as well as the Exchange Management Tools.  A nice tip to note is that you can have both the Exchange 2007 Management Tools as well as the Exchange 2010 Management Tools installed on the same box.  Click Next to Continue.

You are now provided with Client Settings.  If you have Outlook 2003 or Microsoft Entourage, click Yes.  This creates a Public Folder Database and modifies some Exchange options such as OAB Distribution for Public Folders to provide support for these clients.  As a side note, there was an msexchangeteam.com blog post that stated that Entourage is getting updated to support Exchange Web Services (EWS) so in the future, you may only have to do this for Outlook 2003 clients and not Entourage.  For purposes of this lab, we will not be using Entourage or Outlook 2003.  Click Next to Continue.

Your first server is most likely going to be an internet facing CAS server.  Because of this, I specified our Internet-facing FQDN.  This will modify your -ExternalURL parameters for this Exchange 2010 CAS.  Pretty nifty and an installation option I very much welcome.  Click Next to Continue.

You are now provided with the Customer Experience Improvement Program.  I always like to join these things to provide information to Microsoft to help make the product better. Click Next to Continue.

Finally, it’s time for some Readiness Checks.  We can see that the organization will need to be prepared with a /PrepareAD which will prepare the schema, forest, and domain.  Make sure the person running this installation is a part of the Enterprise Admins and Schema Admins in order to update the schema.

We also see that we need the Filter Pack.  I didn’t include this in Part 1 as Microsoft updates their Setup Prerequisite files and this (links/files/requirement)  may change in the future.  So go to the link here to download the filter pack.  Make sure you download and install the x64 version.  You can install the Filterpack while the Exchange setup is still running.  Once the Filterpack is finished installing, Click Install in the Exchange Setup.

So now you’re presented with the installation.  It only took 10-15 minutes for the install to complete.  Pretty fast!  Click Finish to Finish.

Summary

Well folks, that is all for Part 2 of this article. For Part 3, I will go through the DAG creation process (including File Share Witness on a non-Exchange Server) as well as add our first node to the DAG.

Now that Exchange Server 2010 is RTM and Server 2008 R2 is RTM, I thought it would be nice to create a multi-part article on how to use create a Database Availability Group (DAG) on two Exchange Server 2010 RTM nodes utilizing Server 2008 R2 as their Operating System. This article is to guide you through the entire process from preparing Server 2008 R2 for Exchange 2010 RTM, installing Exchange 2010 RTM, creating databases, creating a DAG, adding our nodes to the DAG, and then replicating our databases between both servers.

Part 1

Part 2

Part 3

Part 4

Lab Setup

Guest Virtual Machines

One Server 2008 R2 Enterprise (Standard can be used) RTM x64 Domain Controller.

Two Server 2008 R2 Enterprise (Enterprise Required) RTM x64 (x64 required) Member Servers where Exchange 2010 RTM will be installed with the Mailbox, Client Access Server, and Hub Transport Server roles.

One Server 2008 Enterprise (Standard can be used) RTM x64  server that will be our File Share Witness (FSW) Server.  This box will not serve any other purpose in this lab other than FSW.

Assumptions

• You have a domain that contains at least one Server 2003 SP2 Domain Controller (DC).
• You have configured the IP settings accordingly for all servers to be on the same subnet which includes the public NICs for both Failover Cluster nodes. I have provided the IP scheme of my lab below, but this will vary depending on your needs and VMware configuration.

Computer Names

DAG Node 1 – SHUD-EXC01

DAG Node 2 – SHUD-EXC02

Domain Controller – SHUD-DC01

FSW Server – SHUD-OCSFE01

Configuration of  Exchange 2010 DAG Nodes

Processor: 4

Memory: 1024MB

Network Type - MAPI NIC (MAPI Network)

Network Type - Replication NIC (Replication Network)

Virtual Disk Type – System Volume (C:\): 50GB Dynamic

Storage Note: In a real-world environment, depending on the needs of the business and environment, it is best practice to install your database and logs on separate disks/spindles; both of which are separate from the spindles that the C:\ partition utilize. We will be installing Exchange 2010 RTM databases/logs on the same disks/spindles for simplicity sakes for this lab.  While Exchange 2010 databases move a lot of the IO for databases to sequential IO, there’s still quite a bit of Random IO occurring and is still recommended to place the database and logs on separate spindles.

Network Note: A single NIC DAG is supported.  It is still recommended to have at least one dedicated replication network.  If using only a single NIC, it is recommended for this network to be redundant as well as gigabit.

Configuration of  Domain Controller

Processor: 4

Memory: 512MB

Network Type - External NIC

Virtual Disk Type – System Volume (C:\): 50GB Dynamic

IP Addressing Scheme (Corporate Subnet otherwise known as a MAPI Network to Exchange 2010 DAGs)

IP Address – 192.168.1.x

Subnet Mask – 255.255.255.0

Default Gateway – 192.168.1.1

DNS Server – 192.168.1.150 (IP Address of the Domain Controller/DNS Server)

IP Addressing Scheme (Heartbeat Subnet otherwise known as a Replication Network to Exchange 2010 DAGs)

IP Address – 10.10.10.x

Default Gateway – 10.10.10.x

Subnet Mask – 255.255.255.0

LAB Architecture

Some notes about this architecture:

• Exchange 2010 DAGs remove the limitation of requiring Mailbox Only Role Servers as existed with Exchange 2007 Clustered Servers
• Exchange 2010 is no longer Cluster Aware and only utilizes very few pieces of the Failover Cluster Services such as Cluster Heartbeat and Cluster Networks.  More on this in an upcoming part.
• UM is supported on these two DAG nodes but is recommended to be installed on separate servers
• For HTTP publishing, ISA can be utilized.  For RPC Client Access Server publishing (which ISA cannot due as it publishes HTTP traffic only) with CAS Servers on the DAG nodes, you must use a hardware load balancer due to a Windows limitation preventing you from using Windows NLB and Clustering Services on the same Windows box.  Alternatively, you can deploy two dedicated CAS Servers and utilize Windows NLB to load balance your RPC Client Access Server traffic.
• Two node DAG requires a witness that is not on a server within the DAG.  Unlike Exchange 2007, Exchange 2010 automatically takes care of FSW creation; though you do have to specify the location of the FSW. It is recommended to specify the FSW to be created on a Hub Transport Server.  Alternatively, you can put the witness on a non-Exchange Server after some prerequisites have been completed.  I will be deploying the FSW on a member server (which happens to be my OCS Server in my lab) and will display the prerequisite process for achieving this.

Preparation of Exchange 2010 RTM DAG Nodes

Network Interface Card (NIC) Configuration

First thing we will want to do is configure the IP Configuration of both the MAPI NIC and the Replication NIC.

We will want to rename our MAPI NIC connection to MAPI and our Replication NIC connection to Replication. To do so, go to Start > Right-Click Network > Properties.

Once in the Control Panel, Choose Change Adapter Settings.

Now you will be presented with the Network Connections window. This is where you can modify the network properties for each NIC in your server. For your Internal Corporate Connection which is also your MAPI Network, rename your Local Area Connection to Internal. Likewise, for your Private Heartbeat Connection which is also your Replication Network, rename your Local Area Connection to Replication. After you have done this, it will look something similar to the following:

Network Interface Card (NIC) Configuration

First thing we will want to do is configure the IP Configuration of both the MAPI and Replication NIC.

Part of the assumptions earlier in this article as that you have a properly configured TCP/IP Network where all nodes are properly connected to the TCP/IP Network. Because of this, I will skip the Public TCP/IP Configuration and proceed to configuring the Private Heartbeat NIC.

Important: When configuring the MAPI NIC, you can leave IPv6 enabled if you are using Server 2008 R2.  There is an issue with Server 2008 (still exists in SP2) that prevents IPv6 from listening on port 6004 that prevents Outlook Anywhere from working. You can read more about that here. Again, Server 2008 R2 does not have this issue.  So if you happen to be installing Exchange 2010 on Server 2008, disable IPv6 as discussed below.  If using Server 2008 R2, feel free to leave IPv6 enabled.

Note: You can, if you’d like, disable File and Printer Sharing for Microsoft Networks.  In Exchange 2007 SP1, Microsoft provided the ability to allow for continuous replication to occur over the private network.  Because Exchange 2007 utilizes SMB for log shipping, it is required to have the File and Printer Sharing enabled.  Exchange 2010 no longer utilizes SMB and now utilizes TCP.  More on this later in an upcoming Part.

In addition to disabling IPv6 from the NIC Properties, I would follow these instructions here to fully disable IPv6 on your Exchange 2010 system as disabling it on the NIC itself doesn’t fully disable IPv6.  While the article is based on Exchange 2007, it’s a Windows based modification and will apply to a system running Exchange 2010 as well.

Double-Click or Right-Click > Properties on the Replication NIC to begin configuration.

Uncheck the following:

• Internet Protocol Version 6 (TCP /IPv6) – Disable IPv6 in the registry as well as noted above.

Select Internet-Protocol Version 4 (TCP /IPv4) and press the Properties button. For NodeA, the only TCP/IP configuration we will need, is the IP Address and Subnet Mask. NodeA’s IP configuration will be 10.10.10.152/24 while NodeB’s IP configuration will be 10.10.10.153/24.

Go into the Advanced NIC configuration settings by clicking the Advanced button. From there, you will navigate to DNS tab and de-select “Register this connection’s addresses in DNS.”

Select the WINS tab and de-select “Enable LMHOSTS lookup” and configure the NetBIOS setting to “Disable NetBIOS over TCP/IP.”

Once you are done configuring the Advanced settings, press OK three times and you will be back at the Network Connections screen. From here, choose Advanced and select Advanced Settings…

You will be presented with the Binding Order for your current NICs. Ensure that the MAPI NIC is on top by selecting MAPI and pressing the green up arrow key on the right-hand side of the dialog.

Exchange 2010 Operating System Prerequisites

Server 2008 SP2 and Server 2008 R2 prerequisites are quite different.  Because our servers are going to be deployed on Server 2008 R2, we will follow the guidance for deploying on Server 2008 R2.  You can see the prerequisite requirements here.

We will be doing our prerequsite installations via PowerShell.  You can open PowerShell by going to Start > Run > PowerShell.

You will first have to import the module for ServerManager.  Afterwards, depending on the roles that are installed, different prerequisites are required.  Because we are going to be installing HUB/CAS/MBX, the command we would run is the following:

Note: The installation documentation does not have you include Failover-Clustering in the above command.  I add it anyways since we’ll be using it for the DAG.  I you don’t add it in the above command, you can just add it below when you enable the NetTcpPortSharing.  If you don’t add it below, when you add the first node to the DAG, Failover Clustering will be installed anyways.  I like to install it beforehand though.

Finally, we’ll want to modify the NetTcpPortSharing service to start automatically.

Summary

Well folks, that is all for Part 1 of this article. For Part 2, I will go through the installation process of one our DAG nodes that will contain the Client Access Server, Hub Transport Server, as well as Mailbox Server roles.

Exchange has an Online Maintenance task that runs every night.  In the Exchange Management Console (EMC),  go to Organization Configuration > Mailbox > Database Management Tab > Right-Click our Database > Properties >  Maintenace Tab. We can then see our Maintenance Schedule.

In Exchange 2010, this will appear as:

As you can see, in Exchange 2010, there is a new option that is enabled by default.  This option is the “Enable background database maintenance (24 x 7 ESE scanning).  This option is not Online Defragmentation, but is rather Database Checksumming.  More on this later…

In Online Maintenance, there’s several tasks that run such as dumpster cleanup, purging mailboxes based on retention, etc…  You can see a full list of these tasks here.  When these eleven tasks successfully finish, an Online Defragmentation (OLD) process runs.  Microsoft explains OLD as, “The intention for online defragmentation is to free up pages in the database by compacting records onto the fewest number of pages possible, thus reducing the amount of I/O necessary. The ESE database engine does this by taking the database metadata, which is the information in the database that describes tables in the database, and for each table, visiting each page in the table, and attempting to move records onto logically ordered pages.”

There is also a process called Online Maintenance Database Checksumming.  Checksumming checks the integrity of the database by looking through every database page since there was no guarantee OLD would successfully look through every page to ensure there is no corruption.  This process is entirely sequential and doesn’t cause a performance problem on the database.  The issues with this method in Exchange 2007 RTM is that this process ran at the end of Online Maintenace and because of that, resiliency is effected as these processes temporary suspend continous replication.  In Exchange 2007 SP1, Microsoft provided registry keys to allow you to run background checksumming.  You can read more about these processes and the registry keys at the bottom of this article here.

In Exchange editions prior to Exchange 2010, we can monitor OLD by checking out the available 70x Event IDs in the Event Viewer’s Application Log.  Similarly, you can verify the amount of whitespace that has been created in the database by viewing the 1221 Event ID.   The list of Event IDs for Exchange versions prior to Exchange 2010 is as follows:

• 700 – Starting
• 701 – Completed
• 702 – Resuming
• 703 – Completed Resumed Pass
• 704 – Interrupted and Terminated
• 1221 – Whitespace Amount

This has all changed quite a bit in Exchange 2010.  OLD2 is the new version of Online Defragmentation and no longer occurs at the end of the Online Maintenance Schedule.  Instead, it runs 24 x 7 on a database. It is throttled so it does not negatively affect performance.  You cannot modify OLD2 to run as OLD did in earlier versions of Exchange.  OLD2 is not configurable. Because of this, the need to troll the above Event IDs is no more.  Instead of trolling 70x Event IDs, Exchange 2010 will only notify you if something goes wrong with Online Maintenance.  That way all the 70x error codes do not appear as spam.  If you see a 70x in Exchange 2010, you know there is a problem. Keep in mind though, that this is all in regards to Mailbox Databases.  Queue Databases still have 70x Event IDs.

If you need to check available whitespace, you can now do this via the Exchange Management Shell (EMS).  If your database is called Database1, the command would be:

Note: The -Status switch is required when you need to contact the database directly for the following pieces of information:

• BackupInProgress
• Mounted
• OnlineMaintenanceInProgress
• Available free space in the database root

As stated, OLD2 is throttled and doesn’t negatively affect performance.  If you have an interest in monitoring performance of OLD2, you can do so by using the following perfmon counter set: MSExchange Database -> Defragmentation Tasks.

In Exchange 2010, there are two ways you can configure Online Database Scanning (checksumming).  The first is the default option shown in the first image in this article.  By default, it runs as a 24 x 7 process on the Active Database.   You can uncheck this option which will then revert Online Database Scanning so that it runs after all online maintenance tasks are completed.  Because most online maintenance tasks complete within an hour, this process works reasonably fine for smaller databases (under 500GB).  Microsoft now supports up to 2TB databases.  Anything larger than 500GB should definitely have the default set which is to run this process 24 x 7 to ensure it completes.  Exchange 2010 was designed with the mindset that Online Database Scanning should complete at least once every three days.  If it does not, Exchange 2010 will provide a warning event in the Event Logs.

Thanks to Matt Gossage, Program Manager for Storage at Microsoft for providing much of this information. You can see Matt Gossage’s level 300 webcast on Exchange 2010 Storage here.