Mirroring iSCSI Volumes with Dynamic Disks (Windows Server 2008R2/2012)

Problem

I was recently tasked with copying a raw iSCSI LUN from one SAN to another within a window of an evenings downtime.

No problem I initially thought, I can present both LUNs to a single server and run an XCOPY/ROBOCOPY job to copy all of the data across. I later found there to be two problems with this plan:

• A copy job containing 7TB of data would not complete in one evening.
• There was a fairly large amount of files with long file names sporadically nestled in the file system which would cause the copy job to fail and require babysitting.

Solution

A colleague suggested that I could overcome the above issues by mirroring dynamic disks and then breaking the mirror once the synchronization had finished. That way users can still access the files via the source data whilst the disks are syncing and as it's done on a block level we no longer have to worry about any files with long files spoiling our fun.

First present both disks to a single server and convert the source to a dynamic disk:

Right click the source volume and add a mirror:

Choose your target disk:

Accept that the target disk will be converted to a dynamic disk:

Wait until the disks have finished syncing and show healthy:

Right click one of the mirrored volumes and choose 'Break Mirrored Volume':

Offline the source disk and change the drive letter to resume that place of the source (in my example drive E:):

Disconnect the source disk and start using your new volume:

At the time of writing this I could not find this procedure documented online, so I hope it has been helpful to any of you reading.

451 4.4.0 Primary target IP address responded with: "451 5.7.3 Must issue a STARTTLS commnd first" Office 365 Hybrid

If you have an Office 365 hybrid configuration you may experience issues sending emails between on premise and cloud users (in either direction).

The Exchange 2013 (or 2010) on premises queue viewer may show:

'451 4.4.0 Primary target IP address responded with: "451 5.7.3 STARTTLS is required to send mail." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was xxx.xxx.xxx.xxx'

The Office 365 Message Trace Console shows the delivery status of 'None'
 


Office 365 Message Trace

 

The errors suggest the TLS connection cannot be made but a TLS certificate IS present and during the Hybrid Connection Wizard the required connectors are automatically created so should not require an additional configuration.

When an email is sent between on premise & cloud (Office 365) users of your SSO domain it is sent across one of the automatically created send connectors. These connectors are secured using TLS.

So, assuming you have ruled out all the normal stuff its now time to get baffled. We know the on premise server can send and receive external email. We also know that the Office 365 service can send and receive email. It is just the email between the two services that does not work.

I was banging my head against a wall for ages until I used Telnet to connect from my on premise Exchange server to Microsoft cloud gateway.

What I got is shown below:

This is not correct. As you can see the server has not recognised the "ehlo" statement and the banner does not "look right"...

A bit of digging around the firewall I noticed that packets were being dropped when TLS was attempted.

The firewall is a Cisco PIX 515. I disabled ESMTP inspection but that made no difference so I discounted this as the cause.

After a lot more digging around and raging I remembered that the PIX was behind another Cisco firewall - this time an ASA 5510. So I accessed this device and sure enough this edge firewall was also inspecting and dropping TLS over SMTP.

Once both firewall were configured not to inspect ESMTP the default configuration that was set by the Hybrid Configuration Wizard started working straight away.

The commands to disable ESMTP inspection are:

pix(config)#policy-map global_policy
pix(config-pmap)#class inspection_default
pix(config-pmap-c)#no inspect esmtp
pix(config-pmap-c)#exit
pix(config-pmap)#exit

Or you can do this via the ASDM GUI as shown.

Now telnet the cloud server and you should see a correct banner:

Error 1024, 1036, 11729 & 1035 occur during exchange 2010 sp2 rollup1 Installation

Issue:

Exchange roll up installation fails with numerous error codes.

When installing Exchange 2010 Post SP2 Rollup1 you will most likely face this issue halfway through the installation.

Followed by a terminal failure message with generic web link!

And errors 1024, 1036, 11729 & 1035 will be shown in the Windows Event Log.

Solution:

Use an ELEVATED command prompt to install this (or any other) Exchange 2010 updates.

To use the command prompot as administrator simply right click the Command Prompt icon and select "Run as Administrator"

Then navigate to the location of the update file and run the command directly from the shell. You should now be able to install this patch without issue.

Disable StartTLS on EX2010 Send Connectors - Quick Fix

In your Exchange Hub Transport Server event viewer you may see a repeated TransportService Error 12014 on a regular basis.

This is typically because you do not have a certificate installed on the HT server with a FQDN that matches the FQDN entered into the send connector or the local computer FQDN if the send connector FQDN is empty.

When you check the properties of the Send Connector you see that TLS is NOT required but you still get the error.

This is because the receiving server has requested TLS so your exchange server tries to start TLS but this fails and the message is sent without TLS in the normal fashion. Everything works but you see the annoying errors!

Even when you have disabled TLS on a send connector this does not stop the receiving server from requesting TLS. If you examine the send connector using the EMS - get-sendconnector [connectorname] | fl you will probably see an entry called "IgnoreSTARTTLS" set to false.

This is the default and allows the Exchange server to provide TLS when requested. But if the following are true then the solution is simple:

• If you dont have a certificate installed and bound to the SMTP service
• If your certificate does not have the correct names
• You just dont want to provide or use TLS ever

Solution:

Use the Exchange Management Shell (EMS) and change the IgnoreSTARTTLS value to $true as shown below:

Open the EMS

1. get-SendConnector -identity [send connector name] | set-SendConnector -IgnoreSTARTTLS: $true
2. Restart the Microsoft Exchange Transport Service (restart-service msexchangetransport).

Important Note:

If you have configured your server to Require TLS this procedure will not work and you will need to sort your certificates out.

Server 2008 R2 Slow Shut Down

If you experience very slow shutdowns after deploying a 2008 R2 member server in to a 2003 domain you might want to check the Page file at Shutdown option. If the option is enabled it can take 25 minutes or so to shutdown the server.

In order to disable this setting follow the below steps:

Select Start > run and type secpol.msc
Select Local Policies > Security Options > Shutdown: clear virtual memory pagefile

You should then find that server shutsdown in normal time

Veeam 5 - File is larger than the maximum size supported by datastore

When Veeam is used to back up a virtual machine a snapshot is created while the backup takes place.


By default VMware snapshots are stored on the same datastore as the virtual machine configuration file (the .vmx file).


So, if the VM that you want to back up has additional VMDK files on other datastores and the additional VMDK file is larger than the block size of the datastore where the configuration file is stored you will not be able to back it up with Veeam and you will get this error.

File <unspecified filename> is larger than the maximum size supported by datastore <unspecified datastore>


The datastore where the snapshot will be created does not have a large enough block size to hold the snapshot of the large VMDK so the operation fails.


To work around this you have two options:

1. Storage VMotion the configuration file only to the datastore that holds the biggest VMDK for the machine in question.
2. Edit the VMX file and reboot the VM.

If you have a licence for storage VMotion then simply do that as its simple and does not require a reboot, remember to use the advanced options for SVMotion to just move the configuration file.

If you don't have a SVMotion licence you will have to power off the VM and perform an offline SVMotion of the configuration file only.

Alternatively you can download the .vmx file using the VI Client and edit the "workingDir" entry to reflect a datastore with a large enough block size to accommodate the snapshot.

This VMware article describes the process in more detail: 

Remember that the .VMX file is read by the VM on startup so you will have to reboot it for the changes to take place, but if you have to reboot it why not just offline SVMotion the configuration file - it's much easier this way!

Once this is all done Veeam should back up your large VM again.

Exchange 2010 MAPI not working - Only Outlook Anywhere works!

Recently I had an issue during an EX2003 - EX2010 migration where MAPI appeared not to work but Outlook Anywhere did!!

The symptoms were this:

An outlook profile would refuse to connect first time but upon a restart of Outlook you would get a standard username/password dialog box. Once the credentials were entered Outlook would connect and the user would be happy. A quick look at the connection status showed that Outlook was connected over HTTPS ie. Outlook Anywhere - But this had not been configured for the client so how was this happening?

One look at the Outlook profile showed that Outlook Anywhere was indeed enabled, indeed if you disabled this it would immediately re apply itself next time Outlook was launched.

All the ususal suspects were examined in turn as we thought it must be a GPO or dodgy Outlook .PRF file, the onsite guys even thought it was some new Exchange 2010 security feature that forces all connection to be made by HTTPS!!


The Exchange CAS array was behind a hardware load balancer so this was bypassed but the result was the same. So that ruled out the load balancer...So we thought :-)

Basically I had made a schoolboy error and had neglected to add the RPC Endpoint Mapper (TCP:135) to the Exchange Virtual IP. The result was that Exchange treated all client connections as remote and using a bit of autodiscover wizardry it established a HTTPS connection and modified the Outlook profile to reflect a HTTPS connection.

Closer examination of the load balancer showed the error and once RPC traffic was added to the load balancer front end everything started working properly.

But why did it still fail when I bypassed the loadbalancer? Simple - I had a peaky hosts file directing the CAS array name to the load balancer so changing the CAS array 'A' record had no effect!

Moral of the story is to check the simple things first I guess!