Update Rollup 9 for Microsoft Exchange Server 2007 SP 1

hi, few hours ago was released the rollup 9 for Microsoft Exchange Server 2007 Service Pack 1

http://support.microsoft.com/kb/970162

Download

Download the Exchange2007-KB970162-EN package now.

Issues that the update rollup fixes

Update Rollup 9 for Exchange Server 2007 SP1 fixes the issues that are described in the following Microsoft Knowledge Base articles:

• 943073 (http://support.microsoft.com/kb/943073/ ) An image attachment appears as a red "X" when you send an RTF e-mail message from an Exchange Server 2007 organization to an external recipient

• 945877 (http://support.microsoft.com/kb/945877/ ) The "eseutil /k" command takes a long time to verify the checksum of transaction logs in Exchange Server 2007 Service Pack 1

• 947662 (http://support.microsoft.com/kb/947662/ ) The transport rule "when the Subject field or the body of the message contains text patterns" does not work accurately on an Exchange Server 2007 Service Pack 1-based computer

• 954739 (http://support.microsoft.com/kb/954739/ ) The Exchange Impersonation feature does not work if a cross-forest topology has only a one-way trust relationship between forests in Exchange Server 2007 Service Pack 1

• 957137 (http://support.microsoft.com/kb/957137/ ) The reseed process is unsuccessful on the CCR passive node after you restore one full backup and two or more differential backups to the CCR active node in Exchange Server 2007 Service Pack 1

• 957374 (http://support.microsoft.com/kb/957374/ ) The Microsoft Exchange Replication service on a Standby Continuous Replication (SCR) target server continually crashes when you enable SCR for a storage group on an Exchange Server 2007 Service Pack 1-based computer

• 959559 (http://support.microsoft.com/kb/959559/ ) Transaction log files grow unexpectedly in an Exchange Server 2007 Service Pack 1 mailbox server on a computer that is running Windows Server 2008

• 961124 (http://support.microsoft.com/kb/961124/ ) Some messages are stuck in the Outbox folder or the Drafts folder on a computer that is running Exchange Server 2007 Service Pack 1

• 961544 (http://support.microsoft.com/kb/961544/ ) Mobile users whose location is set to New Zealand cannot synchronize an exceptional occurrence after the daylight saving time (DST) update that is described in KB 951072 is installed on an Exchange 2007 Service Pack 1 Client Access server (CAS)

• 961551 (http://support.microsoft.com/kb/961551/ ) An error message is returned when you run the Get-Recipient command in the Exchange Management Shell that uses a Windows 7 domain controller

• 963679 (http://support.microsoft.com/kb/963679/ ) The Update-Recipient command does not update specified domain controller parameters when you use Identity Lifecycle Manager (ILM) 2007 to migrate mail users to mailbox users in Exchange Server 2007 Service Pack 1

• 967479 (http://support.microsoft.com/kb/967479/ ) Entourage clients cannot synchronize with mailboxes that are located on a computer that is running Exchange 2007 Service Pack 1 and Windows Server 2008

• 967525 (http://support.microsoft.com/kb/967525/ ) Error 4 is returned when you synchronize a supported list of contact properties by using Exchange ActiveSync in Exchange Server 2007 Service Pack 1

• 967605 (http://support.microsoft.com/kb/967605/ ) A non-delivery report (NDR) is returned when a user sends an e-mail message to an X.400 address that includes the slash field separator in Exchange Server 2007 Service Pack 1

• 967676 (http://support.microsoft.com/kb/967676/ ) E-mail address properties of contacts changed through Exchange Web Services (EWS) are not updated in Outlook or Outlook Web Access (OWA) in Exchange Server 2007 Service Pack 1

• 967739 (http://support.microsoft.com/kb/967739/ ) If a sender requests a delivery receipt in an e-mail message, a delivery status notification (DSN) message is returned that has a blank subject in the body even though the original message contains a subject in Exchange Server 2007 Service Pack 1

• 968081 (http://support.microsoft.com/kb/968081/ ) Monthly recurring meetings are declined if the "Schedule only during working hours" option is enabled in Exchange Server 2007 Service Pack 1

• 968106 (http://support.microsoft.com/kb/968106/ ) Outlook clients are directed to global catalogs from the wrong domain if you are using a split session configuration to enable Outlook clients to access their mailboxes through an RPC/HTTP proxy server in Exchange Server 2007 Service Pack 1

• 968111 (http://support.microsoft.com/kb/968111/ ) Event ID 4999 is logged when an administrator deletes a mailbox store on an Exchange Server 2007 Service Pack 1-based server

• 968205 (http://support.microsoft.com/kb/968205/ ) The Microsoft Exchange Information Store service crashes every time that a specific database is mounted on a computer that is running Exchange Server 2007 Service Pack 1

• 968224 (http://support.microsoft.com/kb/968224/ ) You still receive unexpected error messages when you run the Test-OwaConnectivity command or the Test-ActiveSyncConnectivity command after you apply hotfix KB954213 on an Exchange 2007 Service Pack 1-based server

• 968322 (http://support.microsoft.com/kb/968322/ ) An HTTP 500 error message is returned when you send a message that has a large attachment by using Outlook Web Access (OWA) with S/MIME installed in Exchange Server 2007 Service Pack 1

• 968350 (http://support.microsoft.com/kb/968350/ ) When you change the location field of a recurring calendar item to empty in Exchange Server 2007 Service Pack 1, the location field is set to the default value of the recurring series if this recurring item is synchronized on a Windows Mobile device

• 968621 (http://support.microsoft.com/kb/968621/ ) The Microsoft Exchange Information Store service crashes when you use a Data Protection Manager (DPM) 2007 server to perform a snapshot backup for an Exchange Server 2007 Service Pack 1 server

• 968626 (http://support.microsoft.com/kb/968626/ ) Event ID 1009 is logged when you use an application to access a shared mailbox by using the POP3 protocol in Exchange Server 2007 Service Pack 1

• 968651 (http://support.microsoft.com/kb/968651/ ) Exchange Server 2007 Service Pack 1 servers continue to contact a domain controller even after you exclude it by using the Set-ExchangeServer command

• 968715 (http://support.microsoft.com/kb/968715/ ) Both public logons and private logons that connect to a Client Access server (CAS) proxy are processed as private logons on an Exchange Server 2007 Service Pack 1-based server

• 969054 (http://support.microsoft.com/kb/969054/ ) Error message after an Exchange Server 2007 Service Pack 1 user replies to a message that has more than 300 recipients in Outlook Web Access (OWA): "Microsoft Exchange issued an unexpected response (500)"

• 969089 (http://support.microsoft.com/kb/969089/ ) Some databases are not mounted on the target server after you use the Move-ClusteredMailboxServer command to transfer a clustered mailbox server (CMS) to an available passive cluster node in Exchange Server 2007 Service Pack 1

• 969129 (http://support.microsoft.com/kb/969129/ ) HTML e-mail messages that have a charset META tag that differs from the MIME charset tag are garbled when they are processed through disclaimer rules in Exchange Server 2007 Service Pack 1

• 969324 (http://support.microsoft.com/kb/969324/ ) Outlook crashes when you try to use Outlook to view e-mail messages that are arranged by subject in Exchange Server 2007 Service Pack 1

• 969436 (http://support.microsoft.com/kb/969436/ ) You cannot log on to a hidden mailbox by using Base64 authentication for IMAP4 or for POP3 in an Exchange Server 2007 Service Pack 1 environment

• 969838 (http://support.microsoft.com/kb/969838/ ) An error message is returned when a user tries to change a recurring appointment in Office Outlook Web Access that was created in Outlook 2007 in Exchange Server 2007 Service Pack 1

• 969911 (http://support.microsoft.com/kb/969911/ ) Mailboxes do not follow E-mail Lifecycle (ELC) configuration or storage limitation policies in Exchange Server 2007 Service Pack 1

• 969943 (http://support.microsoft.com/kb/969943/ ) Memory leaks occur in the Powershell.exe process when you run the Get-MailboxStatistics command and the Get-PublicFolderStatistics command in Exchange Server 2007 Service Pack 1

• 969969 (http://support.microsoft.com/kb/969969/ ) Error message when an Exchange Server 2007 Service Pack 1 user tries to delete a calendar item in OWA: "Outlook Web Access has encountered a Web browsing error"

• 970028 (http://support.microsoft.com/kb/970028/ ) The Store.exe process crashes when you use a WebDAV application to connect to Exchange Server 2007 Service Pack 1

• 970086 (http://support.microsoft.com/kb/970086/ ) Exchange Server 2007 Service Pack 1 crashes when the Extensible Storage Engine (ESE) version store is out of memory on a computer that is running Windows Server 2008

• 970277 (http://support.microsoft.com/kb/970277/ ) The System Attendant (SA) resource is not brought online or offline during a failover in an Exchange 2007 Service Pack 1 cluster environment

• 970444 (http://support.microsoft.com/kb/970444/ ) A move operation between an Exchange Server 2003-based server and an Exchange Server 2007 Service Pack 1-based server fails if the SimpleDisplayName attribute of a mailbox in the Exchange Server 2003-based server contains a single quotation mark

• 970515 (http://support.microsoft.com/kb/970515/ ) You receive an error message when you try to use the "New-Mailbox" command to create more than 1000 users who have the same “mailNickname” attribute (alias) in Exchange Server 2007 Service Pack 1

• 970526 (http://support.microsoft.com/kb/970526/ ) The EdgeTransport.exe process on a computer that is running Exchange Server 2007 Service Pack 1 crashes when a MIME message that contains iCAL items for a recurring meeting has more than 999 occurrences

• 970725 (http://support.microsoft.com/kb/970725/ ) Public folder replication messages stay in the local delivery queue and cause an Exchange Server 2007 Service Pack 1 database to grow quickly

• 970993 (http://support.microsoft.com/kb/970993/ ) Error message when a user tries to perform an address book search by using Outlook Web Access in an Exchange Server 2007 Service Pack 1 environment: “The item that you attempted to access appears to be corrupted and cannot be accessed.”

How to collect per request Performance Stats for IIS on Exchange 2007

Ever had a time where you were trying to troubleshoot an IIS Performance related issue on Exchange 2007 and the built-in performance counters were not giving you the data that you needed to gain insight in to the problem? I know I have run in to these before and they are not always the easiest to track as we cannot see latencies at a per request level easily.

As part of the default installation of Exchange 2007, you may have also seen IIS log entries similar to the following, but didn’t know what the appended IIS data meant.

/owa/ev.owa oeh=1&ns=DatePicker&ev=GetFreeBusy&m=2009-04-01T00%3a00%3a00&fId=LgAAAADBC0ggZ4mHTKllH8Mc0937AQBmBiNCEaM7R53LcWBj0I1aAAAAAACrAAAC&prfltncy=98&prfrpccnt=6&prfrpcltncy=78&prfldpcnt=0&prfldpltncy=0&prfavlcnt=0&prfavlltncy=0

The information I am calling out in this IIS Log request is prfltncy, prfrpccnt, prfrpcltncy, prfldpcnt, prfldpltncy, prfavlltncy. These entries are specific to latency entries at the end of each call that is being made. There may only be a handful of these throughout the logs by default.

Luckily, there is a way to enable additional per request user tracing in to the IIS logs to help you with troubleshooting these performance type problems. This tracing will allow you to see per request latencies for OWA, RPC and Availability requests.

To enable this additional logging, you would do the following:

Go to "Program Files\Microsoft\Exchange Server\ClientAccess\OWA". Edit web.config in Notepad. Add the following line of text under appSettings:


After saving the web.config file, you should start seeing entries in the IIS logs similar to the above, but here is another log example:

/owa/default.aspx modurl=7&prfltncy=84212&prfrpccnt=37&prfrpcltncy=84011&prfldpcnt=9&prfldpltncy=30&prfavlcnt=0&prfavlltncy=0

In the above request, we can see that the RPC latencies are high (prfltncy=84212&prfrpccnt=37&prfrpcltncy=84011) , so this was most likely a bottleneck between the CAS and the backend Mailbox server. Now wasn’t that easy to determine where the potential bottleneck might lie?

Per Request Tracing Legend
Prfltncy - Overall Performance Latencies for this request
Prfrpccnt - RPC request count
Prfrpcltncy - RPC Latencies
Prfldpcnt - LDAP request count
Prfldpltncy – LDAP Latencies
Prfavlltncy - Availability Latencies

If you break one of these log requests down, here is the way you would look at this based on the first request example above. (Note: This was a call to get Free/Busy Data for a specific time period)

* prfltncy=98 - Overall Performance Latency for the request
* prfrpccnt=6&prfrpcltncy=78 - 6 RPC requests with a latency of 78ms
* prfldpcnt=0&prfldpltncy=0 - 0 LDAP requests with a latency of 0ms
* prfavlcnt=0&prfavlltncy=0 - 0 Availability requests with a latency of 0ms

You can use any log parser (ie.logparser.exe) of your choice to get further information, but this should help you understand some of the latencies down to a per request level.

I hope this helps in your performance troubleshooting…..

Troubleshooting Exchange 2007 Store Log/Database growth issues

One of the common issues we see in support is excessive Database and/or Transaction log growth problems. If you have ever run in to one of these issues, you will find that they are not always easy to troubleshoot as there are many tools that are needed to help understand where the problem might be coming from. Customers have asked why does the Server allow these type of operations to occur in the first place and why is the Exchange Server not resilient to this? That is not always an easy question to answer as there as so many variables as to why this may occur in the first place ranging from faulty Outlook Add-ins, Custom or 3rd party applications, corrupted rules, corrupted messages, online maintenance not running long enough to properly maintain your database, and the list goes on and on.

Once an Outlook client has created a profile to the Exchange server, they pretty much have full reign to do whatever actions they want within that MAPI profile. This of course, will be controlled mostly by your Organizations mailbox and message size limits and some of the Client throttling or backoff features that are new to Exchange 2007.

Since I have dealt with these type problems in great detail, I thought it would be helpful to share some troubleshooting steps with you that may help you collect, detect and mitigate these problems when and if you should see them.

General Troubleshooting

1. Use Exchange User Monitor (Exmon) server side to determine if a specific user is causing the log growth problems.
   
   
   
   
   • Sort on CPU (%) and look at the top 5 users that are consuming the most amount of CPU inside the Store process. Check the Log Bytes column to verify for this log growth for a potential user.
   • If that does not show a possible user, sort on the Log Bytes column to look for any possible users that could be attributing to the log growth
   • If it appears that the user in Exmon is a ?, then this is representative of a HUB/Transport related problem generating the logs. Query the message tracking logs using the Message Tracking Log tool in the Exchange Management Consoles Toolbox to check for any large messages that might be running through the system. See step 5.9 for a Powershell script to accomplish the same task.

2. If suspected user is found via Exmon, then do one of the following:

   1. Disable the users AD account temporarily

   2. Kill their TCP connection with TCPView

   3. Call the client to have them close Outlook in the condition state for immediate relief.

3. If closing the client down seems to stop the log growth issue, then we need to do the following to see if this is OST or Outlook profile related:

   1. Have the user launch Outlook while holding down the control key which will prompt if you would like to run Outlook in safe mode. If launching Outlook in safe mode resolves the log growth issue, then concentrate on what add-ins could be attributing to this problem.

   2. If you can gain access to the users machine, then do one of the following:

      1. Launch Outlook to confirm the log file growth issue on the server.

      2. If log growth is confirmed, do one of the following

         1. Check users Outbox for any messages.

            1. If user is running in Cached mode, set the Outlook client to Work Offline. Doing this will help stop the message being sent in the outbox and sometimes causes the message to NDR.

            2. If user is running in Online Mode, then try moving the message to another folder to prevent Outlook or the HUB server from processing the message.

            3. After each one of the steps above, check the Exchange server to see if log growth has ceased

         2. Call Microsoft Product Support to enable debug logging of the Outlook client to determine possible root cause.

      3. Follow the Running Process Explorer instructions in the below article to dump out dlls that are running within the Outlook Process. Name the file username.txt. This helps check for any 3rd party Outlook Add-ins that may be causing the excessive log growth.
         
         
         
         970920 Using Process Explorer to List dlls Running Under the Outlook.exe Process
         
         http://support.microsoft.com/kb/970920

      4. Check the Sync Issues folder for any errors that might be occurring

   3. Let’s attempt to narrow this down further to see if the problem is truly in the OST or something possibly Outlook Profile related:

      1. Run ScanPST against the users OST file to check for possible corruption.

      2. With the Outlook client shut down, rename the users OST file to something else and then launch Outlook to recreate a new OST file. If the problem does not occur, we know the problem is within the OST itself.

      3. If renaming the OST causes the problem to recur again, then recreate the users profile to see if this might be profile related.

4. Ask Questions:

   1. Is the user using any type of mobile device?

   2. Question the end user if at all possible to understand what they might have been doing at the time the problem started occurring. It’s possible that a user imported a lot of data from a PST file which could cause log growth server side or there was some other erratic behavior that they were seeing based on a user action.

5. If Exmon does not provide the data that is necessary to get root cause, then do the following:

   1. Check current queues against all HUB Transport Servers for stuck or queued messages
      
      
      
      get-exchangeserver | where {$_.IsHubTransportServer -eq "true"} | Get-Queue | where {$_.Deliverytype –eq “MapiDelivery”} | Select-Object Identity, NextHopDomain, Status, MessageCount | export-csv HubQueues.csv
      
      
      
      Review queues for any that are in retry or have a lot of messages queued.
      
      
      
      Export out message sizes in MB in all Hub Transport queues to see if any large messages are being sent through the queues.
      
      
      
      get-exchangeserver | where {$_.ishubtransportserver -eq "true"} | get-message –resultsize unlimited | Select-Object Identity,Subject,status,LastError,RetryCount,queue,@{Name="Message Size MB";expression={$_.size.toMB()}} | sort-object -property size –descending | export-csv HubMessages.csv
      
      
      
      Export out message sizes in Bytes in all Hub Transport queues.
      
      
      
      get-exchangeserver | where {$_.ishubtransportserver -eq "true"} | get-message –resultsize unlimited | Select-Object Identity,Subject,status,LastError,RetryCount,queue,size | sort-object -property size –descending | export-csv HubMessages.csv

   2. Check Users Outbox for any large, looping, or stranded messages that might be affecting overall Log Growth.

      get-mailbox -ResultSize Unlimited| Get-MailboxFolderStatistics -folderscope Outbox | Sort-Object Foldersize -Descending | select-object identity,name,foldertype,itemsinfolder,@{Name="FolderSize MB";expression={$_.folderSize.toMB()}} | export-csv OutboxItems.csv
      
      
      
      Note: This does not get information for users that are running in cached mode.

   3. Utilize the MSExchangeIS Client\Jet Log Record Bytes/sec and MSExchangeIS Client\RPC Operations/sec Perfmon counters to see if there is a particular client protocol that may be generating excessive logs. If a particular protocol mechanism if found to be higher than other protocols for a sustained period of time, then possibly shut down the service hosting the protocol. For example, if Exchange Outlook Web Access is the protocol generating potential log growth, then stopping the World Wide Web Service (W3SVC) to confirm that log growth stops. If log growth stops, then collecting IIS logs from the CAS/MBX Exchange servers involved will help provide insight in to what action the user was performing that was causing this occur.

   4. Run the following command from the Management shell to export out current user operation rates:
      
      
      
      To export to CSV File:
      
      
      
      get-logonstatistics |select-object username,Windows2000account,identity,messagingoperationcount,otheroperationcount,progressoperationcount,streamoperationcount,tableoperationcount,totaloperationcount | where {$_.totaloperationcount -gt 1000} | sort-object totaloperationcount -descending| export-csv LogonStats.csv
      
      
      
      To view realtime data:
      
      
      
      get-logonstatistics |select-object username,Windows2000account,identity,messagingoperationcount,otheroperationcount,progressoperationcount,streamoperationcount,tableoperationcount,totaloperationcount | where {$_.totaloperationcount -gt 1000} | sort-object totaloperationcount -descending| ft
      
      
      
      Key things to look for:
      
      In the below example, the Administrator account was storming the testuser account with email.
      
      You will notice that there are 2 users that are active here, one is the Administrator submitting all of the messages and then you will notice that the Windows2000Account references a HUB server referencing an Identity of testuser. The HUB server also has *no* UserName either, so that is a giveaway right there. This can give you a better understanding of what parties are involved in these high rates of operations
      
      
      
      UserName : Administrator
      
      Windows2000Account : DOMAIN\Administrator
      
      Identity : /o=First Organization/ou=First Administrative Group/cn=Recipients/cn=Administrator
      
      MessagingOperationCount : 1724
      
      OtherOperationCount : 384
      
      ProgressOperationCount : 0
      
      StreamOperationCount : 0
      
      TableOperationCount : 576
      
      TotalOperationCount : 2684
      
      
      
      UserName :
      
      Windows2000Account : DOMAIN\E12-HUB$
      
      Identity : /o= First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=testuser
      
      MessagingOperationCount : 630
      
      OtherOperationCount : 361
      
      ProgressOperationCount : 0
      
      StreamOperationCount : 0
      
      TableOperationCount : 0
      
      TotalOperationCount : 1091

   5. Enable Perfmon/Perfwiz logging on the server. Collect data through the problem times and then review for any irregular activities. You can grab some pre-canned Perfmon import files at http://blogs.technet.com/mikelag/archive/2008/05/02/perfwiz-replacement-for-exchange-2007.aspx to make collecting this data easier.

   6. Run ExTRA (Exchange Troubleshooting Assistant) via the Toolbox in the Exchange Management Console to look for any possible Functions (via FCL Logging) that may be consuming Excessive times within the store process. This needs to be launched during the problem period. http://blogs.technet.com/mikelag/archive/2008/08/21/using-extra-to-find-long-running-transactions-inside-store.aspx shows how to use FCL logging only, but it would be best to include Perfmon, Exmon, and FCL logging via this tool to capture the most amount of data.

   7. Dump the store process during the time of the log growth. (Use this as a last measure once all prior activities have been exhausted and prior to calling Microsoft for assistance. These issues are sometimes intermittent, and the quicker you can obtain any data from the server, the better as this will help provide Microsoft with information on what the underlying cause might be.)

      1. Download the Current Release version of the Windows debuggers from http://www.microsoft.com/whdc/devtools/debugging/install64bit.mspx and select a custom installation and change the directory to install the debuggers to c:\debuggers and finish the installation.

      2. Open the command prompt and change in to the c:\Debuggers directory

      3. Type cscript adplus.vbs –hang –pn store –quiet –o d:\DebugData. Note: -o switch signifies the location in which you want to store the debug data that has sufficient drive space. Important: Once this has launched, a minimized CDB window will open. Please wait for this to complete and do not close this window as this will disappear once the dump has completed.

      4. Wait 2 minutes and perform the same dump operation again.

      5. Open a case with Microsoft Product Support Services to get this data looked at.

   8. Collect a portion of Store transaction log files (100 would be good) during the problem period and parse them following the directions in http://blogs.msdn.com/scottos/archive/2007/11/07/remix-using-powershell-to-parse-ese-transaction-logs.aspx to look for possible patterns such as high pattern counts for IPM.Appointment. This will give you a high level overview if something is looping or a high rate of messages being sent. Note: This tool may or may not provide any benefit depending on the data that is stored in the log files, but sometimes will show data that is MIME encoded that will help with your investigation

   9. Export out Message tracking log data from affected MBX server
      
      
      
      Method 1
      
      Download the attached ExLogGrowthCollector.zip file to this post and extract to the MBX server that experienced the issue. Run ExLogGrowthCollector.ps1 from the Exchange Management Shell. Enter in the MBX server name that you would like to trace, the Start and End times and click on the Collect Logs button.
      
      
      
      
      
      
      
      Note: What this script does is to export out all mail traffic to/from the specified mailbox server across all HUB servers between the times specified. This helps provide insight in to any large or looping messages that might have been sent that could have caused the log growth issue.
      
      
      
      Method 2
      
      Copy/Paste the following data in to notepad, save as msgtrackexport.ps1 and then run this on the affected Mailbox Server. Open in Excel for review. This is similar to the GUI version, but requires manual editing to get it to work.

      #Export Tracking Log data from affected server specifying Start/End Times
      
      
      
      Write-host "Script to export out Mailbox Tracking Log Information"
      
      Write-Host "#####################################################"
      
      Write-Host
      
      $server = Read-Host "Enter Mailbox server Name"
      
      $start = Read-host "Enter start date and time in the format of MM/DD/YYYY hh:mmAM"
      
      $end = Read-host "Enter send date and time in the format of MM/DD/YYYY hh:mmPM"
      
      $fqdn = $(get-exchangeserver $server).fqdn
      
      Write-Host "Writing data out to csv file..... "
      
      Get-ExchangeServer | where {$_.IsHubTransportServer -eq "True" -or $_.name -eq "$server"} | Get-MessageTrackingLog -ResultSize Unlimited -Start $start -End $end | where {$_.ServerHostname -eq $server -or $_.clienthostname -eq $server -or $_.clienthostname -eq $fqdn} | sort-object totalbytes -Descending | export-csv MsgTrack.csv -NoType
      
      Write-Host "Completed!! You can now open the MsgTrack.csv file in Excel for review"

      
      
      Method 3
      
      You can also use the Process Tracking Log Tool at http://msexchangeteam.com/archive/2008/02/07/448082.aspx to provide some very useful reports.

   10. Save off a copy of the application/system logs from the affected server and review them for any events that could attribute to this problem

   11. Enable IIS extended logging for CAS and MB server roles to add the sc-bytes and cs-bytes fields to track large messages being sent via IIS protocols and to also track usage patterns.

Proactive monitoring and mitigation efforts

1. Increase Diagnostics Logging for the following objects depending on what stores are being affected:
   
   
   
   
   • MSExchangeIS\Mailbox\Rules
   • MSExchangeIS\PublicFolders\Rules
     
     
     
     
2. Enable Client Side monitoring per http://technet.microsoft.com/en-us/library/cc540465.aspx
   
   
   
   
3. Create a monitoring plan using MOM/SCOM to alert when the amount of Log Bytes being written hit a specific threshold and then alert the messaging team for further action. There are thresholds that are a part of the Exchange 2007 Management Pack that could help alert to these type situations before the problem gets to a point of taking a database offline. Here are 2 examples of this.
   
   
   
   ESE Log Byte Write/sec MOM threshold
   
   Warning Event
   
   http://technet.microsoft.com/en-us/library/bb218522.aspx
   
   
   
   Error Event
   
   http://technet.microsoft.com/en-us/library/bb218733.aspx
   
   
   
   If an alert is raised, then perform an operation to start collecting data.

4. Ensure http://support.microsoft.com/kb/958701 is installed at a minimum for each Outlook 2003 client to address known log/database growth issues for users streaming data to the information store that have exceeded message size limits. This fix also addresses a problem where clients could copy a message to their inbox from a PST that during the sync process could exceed mailbox limits, thus causing excessive log growth problems on the server.
   
   
   
   These hotfixes make use of the PR_PROHIBIT_SEND_QUOTA and PR_MAX_SUBMIT_MESSAGE_SIZE which is referenced in http://support.microsoft.com/kb/894795

   Additional Outlook Log Growth fixes:
   
   http://support.microsoft.com/kb/957142
   
   http://support.microsoft.com/kb/936184

5. Implement minimum Outlook Client versions that can connect to the Exchange server via the Disable MAPI clients registry key server side. See http://technet.microsoft.com/en-us/library/bb266970.aspx for more information.
   
   
   
   To disable clients less than Outlook 2003 SP2, use the following entries on an Exchange 2007 server
   
   "-5.9.9;7.0.0-11.6568.6567"

   Setting this to exclude Outlook client versions less than Outlook 2003 SP2 will help protect against stream issues to the store. Reason being is that Outlook 2003 SP2 and later understand the new quota properties that were introduced in to the store in http://support.microsoft.com/kb/894795. Older clients have no idea what these new properties are, so if a user sent a 600MB attachment on a message, it would stream the entire message to the store generating excessive log files and then get NDR’ed once the message size limits were checked. With SP2 installed, the Outlook client will first check to see if the attachment size is over the set quota for the organization and immediately stop the send with a warning message on the client and prevent the stream from being sent to the server.

   Allowing any clients older than SP2 to connect to the store is leaving the Exchange servers open for a growth issue.
   
   

6. If Entourage clients are being utilized, then implement the MaxRequestEntityAllowed property in http://support.microsoft.com/kb/935848 to address a known issue where sending a message over the size limit could potentially create log growth for a database.
   
   
   
   
7. Check to ensure File Level Antivirus exclusions are set correctly for both files and processes per http://technet.microsoft.com/en-us/library/bb332342.aspx
   
   
   
   
8. Enable Content Conversion tracing on all HUB servers per http://technet.microsoft.com/en-us/library/bb397226.aspx . This will help log any failed conversion attempts that may be causing the log growth problem to occur.
   
   
   
   
9. If POP3 or IMAP4 clients are connecting to specific servers, then implementing Protocol Logging for each on the servers that may be making use of these protocols will help log data to a log file where these protocols are causing excessive log growth spurts. See http://technet.microsoft.com/en-us/library/aa997690.aspx on how to enable this logging.
   
   
   
   
10. Ensure Online maintenance is completing a pass for each database within the past week or two. Query Application event logs for the ESE events series 700 through 704 to clarify. If log growth issues occur during online maintenance periods, this could be normal as Exchange shuffles data around in the database. We just need to ensure that we keep this part in mind during these log growth problems.
    
    
    
    
11. Check for any excessive ExCDO warning events related to appointments in the application log on the server. (Examples are 8230 or 8264 events). http://support.microsoft.com/kb/947014 is just one example of this issue. If recurrence meeting events are found, then try to regenerate calendar data server side via a process called POOF. See http://blogs.msdn.com/stephen_griffin/archive/2007/02/21/poof-your-calender-really.aspx for more information on what this is.
    
    
    
    Event Type: Warning
    
    Event Source: EXCDO
    
    Event Category: General
    
    Event ID: 8230
    
    Description: An inconsistency was detected in username@domain.com: /Calendar/ .EML. The calendar is being repaired. If other errors occur with this calendar, please view the calendar using Microsoft Outlook Web Access. If a problem persists, please recreate the calendar or the containing mailbox.

    Event Type: Warning
    
    Event ID : 8264
    
    Category : General
    
    Source : EXCDO
    
    Type : Warning
    
    Message : The recurring appointment expansion in mailbox has taken too long. The free/busy information for this calendar may be inaccurate. This may be the result of many very old recurring appointments. To correct this, please remove them or change their start date to a more recent date.
    
    
    
    Important: If 8230 events are consistently seen on an Exchange server, have the user delete/recreate that appointment to remove any corruption

12. Add additional store logging per http://support.microsoft.com/kb/254606 to add more performance counter data to be collected with Perfmon. This will allow us to utilize counters such as ImportDeleteOpRate and SaveChangesMessageOpRates which allows us to see what these common log growth rates are.
    
    
    
    
13. Recommend forcing end dates on recurring meetings. This can be done through the usage of the registry key DisableRecurNoEnd (DWORD).
    
    
    
    For Outlook 2003:
    
    http://support.microsoft.com/kb/952144
    
    HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Preferences
    
    
    
    For Outlook 2007:
    
    http://support.microsoft.com/kb/955449
    
    HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\Preferences
    
    Value: 1 to Enable, 0 to Disable
    
    

14. Implement LimitEmbeddingDepth on the Exchange servers as outlined in KB 833607 to prevent log growth due to recursion looping. Note: This article states this if for Exchange 2000-2003, but the key is also still valid in Exchange 2007 per source code

Known Issues

Exchange Server



SP1 Release Update 9 fixes

• 959559 - Transaction log files grow unexpectedly in an Exchange Server 2007 Service Pack 1 mailbox server on a computer that is running Windows Server 2008

• 925252 - The Store.exe process uses almost 100 percent of CPU resources, and the size of the public folder store increases quickly in Exchange Server 2007
• 961124 - Some messages are stuck in the Outbox folder or the Drafts folder on a computer that is running Exchange Server 2007 Service Pack 1
  
  970725 - Public folder replication messages stay in the local delivery queue and cause an Exchange Server 2007 Service Pack 1 database to grow quickly

SP1 Release Update 8 fixes

• 960775 - You receive a "Message too large for this recipient" NDR that has the original message attached after you restrict the Maximum Message Send Size value in Exchange Server 2007

SP1 Release Update 7 fixes

• 957124 - You do not receive an NDR message even though your meeting request cannot be sent successfully to a recipient
• 960775 - You receive a "Message too large for this recipient" NDR that has the original message attached after you restrict the Maximum Message Send Size value in Exchange Server 2007

SP1 Release Update 1 fixes

• 947014 - An Exchange Server 2007 mailbox server randomly generates many transaction logs in an Exchange Server 2007 Service Pack 1 environment
• 943371 - Event IDs 8206, 8213, and 8199 are logged in an Exchange Server 2007 environment
  
  

Outlook 2007

• 970944 – Installing this hotfix package addresses and issue where log files are generated unexpectedly when a user is running Outlook 2007 in the cached Exchange mode and sends an e-mail message to the recipients who have a corrupted e-mail address and/or e-mail address
  
  

Outlook 2003

• 958701 - Description of the Outlook 2003 Post-Service Pack 3 hotfix package (Engmui.msp, Olkintl.msp, Outlook.msp): October 28, 2008
• 936184 - Description of the Outlook 2003 post-Service Pack 3 hotfix package: December 14, 2007
• 897247 - Description of the Microsoft Office Outlook 2003 post-Service Pack 1 hotfix package: May 2, 2005
  
  

Entourage

• 935848 - Various performance issues occur when you use Entourage for Mac to send large e-mail messages to an Exchange 2007 server
  
  

Windows 2008

• 955612 - The "LCMapString" function may return incorrect mapping results for some languages in Windows Server 2008 and in Windows Vista

Downloadable contents for Office SharePoint Server 2007

Microsoft is providing lots of books, white papers and articles on MOSS configuration and deployment for free. We as a MOSS consultant use these resources in most of the projects as a baseline for architecture. Below link contains a complete list of all downloadable contents available on MOSS, this list is continuously updating by Microsoft for covering new versions and service packs;

http://technet.microsoft.com/en-us/library/cc262788.aspx

Public Folder Mail Enabling issue

Last week i faced an issue. whenever i mail enable any public folder and then double clicked it in exchange management console for setting email address, i am getting following error

“The mail proxy for this folder can not be found. This may be due to replication delays. The mail enabled pages will not be shown.”

After some time following event is logged in the event viewer;

Event Type: Warning
Event Source: MSExchangeIS
Event Category: General
Event ID: 9543
Date: XXXX
Time: XXXX
User: N/A
Computer: XXXX
Description:
Unable to create Public Folder proxy object for folder "XXXX" in the Active Directory.

After some search i found one KB (http://support.microsoft.com/kb/327841) suggesting following fix;

“This issue may occur if the Microsoft Exchange System Attendant service is not set to start under the local system account.”

I checked System attended service and found that it is configured under local system account which is a correct configuration. After lots of search i found nothing and all searches are directing towards the same article.

Because all peoples are pointing towards the same system attended service so i thought why shouldn’t i restart the service and see. I restarted the service and then again checked one recently mail enabled folder but the issue is same then i mail disabled it and then re-enabled it and guess what the issue gone! i it seems that there was some issue and system attended service is not communicating properly with active directory which has been solved after restarting the service.

Changing Exchange server Queues and Logs directory Location

When you install Exchange 2007 Edge or Hub Transport Role, by default Exchange setup configures Queues, Replay, Pickup and Transport logs directory on C:\Program Files\Microsoft\Exchange Server Folder. In an enterprise environment where you have high volume of emails coming in and out it is good to have separate drive for these folders . Following are the steps for changing the these folders default location

Old Queues, Replay, Pickup and Transport logs directory: C:\Program Files\Microsoft\Exchange Server

New Queues, Replay, Pickup and Transport logs directory: F:\Exchange

Configuration steps for Queue Directory:

1. Create the following directory: F:\Exchange\Queue

Permissions Required for the Directory:

Administrator: Full Control

System: Full Control

Network Service: Full Control

2. Open the following file by using Notepad:

C:\Program Files\Microsoft\Exchange Server\Bin\EdgeTransport.exe.config

3. .Modify the following line in the section:

And

4. Save and close the EdgeTransport.exe.config file.

5.Restart the Microsoft Exchange Transport service.

(Make sure you have also added the same set of accounts with the same permissions on F:\Exchange folder otherwise Microsoft exchange Transport service will not start.)

Configuration steps for Pickup Directory:

1. Open Exchange Management Shell.

2. Run following commands
Set-TransportServer –PickupDirectoryPath "F:\Exchange\Pickup"

Permissions Required for Pickup Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Configuration steps for Replay Directory:

1. Open Exchange Management Shell.

2. Run following commands
Set-TransportServer -ReplayDirectoryPath "F:\Exchange\Replay”

Permissions Required for Replay Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Configuration steps for Transport Logs-Connectivity Logs Directory:

1. To change the location of connectivity logs run following commands
Set-TransportServer -ConnectivityLogPath "F:\Exchange\Logs\Connectivity"

Permissions Required for Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Configuration steps for Transport Logs-Protocol Logs Directory:

1. To change the location of Send Protocol Logs run following commands
Set-TransportServer -SendProtocolLogPath "F:\Exchange\Logs\ProtocolLog\SmtpSend"

2. To change the location of Receive Protocol Logs run following commands
Set-TransportServer -ReceiveProtocolLogPath "F:\Exchange\Logs\ProtocolLog\SmtpReceive"

Permissions Required for Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Configuration steps for Transport Logs-Routing Table Logs Directory:

1. To change the location of Routing Table Logs run following commands
Set-TransportServer -RoutingTableLogPath "F:\Exchange\Logs\Routing"

Permissions Required for Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Configuration steps for Transport Logs-Message Tracking Directory:

1. To change the location of Messaging Tracking Logs run following commands
Set-TransportServer -MessageTrackingLogPath " F:\Exchange \logs\MessageTracking"

Note: You might need to disable Message subject logging as part of compliance requirement, in order to achieve this you have to run following command

Set-TransportServer -MessageTrackingLogSubjectLoggingEnabled $false

Permissions Required for Directory

Administrator: Full Control

System: Full Control

Network Service: Read, Write, and Delete Subfolders and File

Exchange 2010 Beta for Download

Wants to try Exchange 2010 ? why wait, now Microsoft has published exchange 2010 beta on their website for downloading and using for 360 days, Following is the link;

http://technet.microsoft.com/en-us/evalcenter/dd185495.aspx

Note: Exchange 2010 Beta can only run on 64 bit machines.

Exchange 2007- Mapi session exceeded the maximum of 32 objects of type "session"

Two days back i faced a strange issue from one of the user that he is not able to open his outlook. Although he can open his OWA but whenever he open his outlook he is getting following error

“Unable to open your default e-mail folders. You must connect to your Microsoft Exchange Server computer with the current profile before you can synchronize your folders with the offline folder file”

After further investigation i found that there is an event logged in the application events of the mailbox server for the same user

Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Event ID: 9646
Date: XXXXX
Time: XXXXX
User: N/A
Computer: XXXXX
Description:
Mapi session "/o=firstorganisation=XXXX/cn=Recipients/cn=username" exceeded the maximum of 32 objects of type "session".

After researching i found following KB on the same issue

http://support.microsoft.com/kb/842022

As per this KB this issue may occur if the following conditions are true:

• You have installed Microsoft Exchange Server 2003 Service Pack 1 (SP1) on the Exchange Server computer.
• A program that is running on a client computer opens many MAPI sessions to the Exchange Server computer. The number of MAPI sessions is larger than the permitted limit.
• You are using Microsoft Office Outlook 2007, and you add a large additional mailbox to your profile. For example, this issue may occur if the additional mailbox contains more than one thousand folders.

I am unable to found any of the above condition in my case. Also fixes mentioned in this article is not relevant to my case except the last registry change which i don't want to apply for one user.

Then i decided to view the connections on the mailbox server. I downloaded TCP view utility from sysinternals (one of the best sites for troubleshooting tools). After running TCP view i have seen lots of connections coming to mailbox server but the user name which i was searching was not visible in the list of connections, then i ran following command on exchange management shell for finding out the source IP of the user having problem

Get-logonstatistics username | FT ClientIPAddress

After viewing the IP Address i searched the IP address in the TCP View and was able to found lots of connections coming from the same IP, i killed these sessions by using Kill option in TCP view and then again tried to open the outlook and the issue gone !!!! I have also informed the user to have a look on his PC for checking if there is any third party software or any other MAPI program which is causing this issue for having a permanent fix for this.

Outlook Web Access to Microsoft Exchange Server

Outlook Web Access (OWA) allows any client with a compatible browser to access Exchange Server folders.

The "premium" version of OWA in Exchange 2003 looks and acts much like the Outlook 2003 desktop client, although it does not offer support for the Journal folder. The "basic" version looks more like previous versions. You may want to try both and see which is faster for the conditions your users are likely to encounter. Microsoft Exchange Server Introduction to Exchange Server 2003 has a good chart comparing the feature sets.

For Exchange 2000, you create applications using web forms. No conversion tool is available to turn Outlook forms into Exchange 2000 OWA forms. For resources and tools, see Microsoft Exchange Server Development Technologies. With Outlook 2002 and Exchange 2000, you can connect using OWA and Outlook 2002's support for HTTP mail servers (at least those that use WebDAV). The correct URL would be http://servername.domain.com/exchange/mailboxname.

There are issues using OWA on Vista because the dhtml text editor control used by OWA is not supported in Vista. The Exchange administrator should have installed the security patch for this on the Exchange server months ago, but many have not. The patch will fix this and a similar issue that affects Windows XP users who've installed the security patch described in MS06-013. The hotfix installs a new editor for Internet Explorer which uses an iframe instead of an ActiveX control.

If you are an end-user, contact your administrator. If you are an administrator, see the following links for more information and the hotfix.

You receive an error message when you try to perform any editing tasks, or you must click to enable the compose frame in Outlook Web Access http://support.microsoft.com/default.aspx/kb/911829.

MS06-013: Cumulative security update for Internet Explorer (July 2006)
http://support.microsoft.com/kb/912812

OWA Problems on Vista (Exchange Messaging Outlook Jan 4 2007)

OWA 5.5 specific content is now at Outlook Web Access 5.5

OWA 2000 specific content is now at Outlook Web Access 2000

Managing Exchange Server Permissions

Viewing Permissions	If you don't see permissions on objects in the Exchange Administrator program (4.0 - 5.5), choose Tools | Options, switch to the Permissions tab, then check the box for Show Permissions pages for all objects.
Folder Permissions	In Exchange 5.5 and earlier, existing folders do not automatically propagate permission changes to child folders. However, new folders do inherit permissions from their parents. Also, using the Exchange Administrator program, you can propagate settings to child folders. If you are not the administrator and need to manage folder permissions, ask the administrator to set up some distribution lists that you can use for setting permissions on the folders. You will need to have permission to edit the DL. Then, when someone new needs to be added, you'll just change the DL -- adding and removing members through Outlook -- not the permissions on each folder. Also see: XCLN: How to Create Public Folders and Set Default Properties on All Subsequently Created Folders XADM: Propagating Permissions to All Public Folder Subfolders XADM: Using PFADMIN to Remove Public Folder Permissions XADM You Cannot Add a Distribution Group to Permissions of a Public Folder in Exchange 2000 -- If you want to use a distribution group, you need AD in native mode. XADM White Paper - Public Folder Permissions in a Mixed-Mode Microsoft Exchange Organization Using a Security Group to Create Public Folder Permissions Working with Store Permissions in Microsoft Exchange 2000 and 2003
User Reply Address	Several scenarios: You want a user to be able to reply to messages sent to a public folder with the folder's address. You want a user to be able to reply with another mailbox's address -- without the user's own address appearing anywhere on the reply. You want to be able to send using the return address of a distribution list in the Global Address List (GAL). The solution is the same in all cases: You must grant Send As permission on the folder or mailbox using the Exchange Administrator program or Active Directory. Send As is granted via accounts and groups, not mailboxes and Exchange distribution lists. If you want a user to send with a folder's address, the folder must not be hidden. Once the user has Send As permission, they can use View | From Field in Outlook to display the From box and either click From to choose from the Address Book or type in the name of the public folder or other mailbox. If the public folder is hidden from the GAL, the user should go to the folder's Properties page and add the folder's address to their own address book. See: HOW TO Grant Send As and Send on Behalf Permissions in Exchange 2000 Server XADM How to Grant a User Send As Rights in Exchange Server 5.5 and Exchange 2000
Tools	Active Folders Exchange Server folder and mailbox management tool suite including policy management, server-based compression, permissions management, audit trail and reporting. Can sweep all user mail folders -- including those in Personal Folders .pst files -- to locate particular attachments or messages meeting other criteria. ERSA "Exchange Security Risk Auditor" to audit and change mailbox permissions both for periodic security audits and for common events, such as an employee leaving the company. Exchange Permission Manager Assign permissions on multiple Exchange public folders and system folders and users can modify mailbox permissions in addition to public folder permissions, company-wide. It works with Exchange 5.5, 2000 and 2003; and, for reporting, you can print out current permissions on single or groups of folders. Version 2. Folder Permissions Manager Symprex Folder Permissions Manager allows administrators to centrally manage all permissions on mailbox folders and public folders on Exchange 5.5, 2000 and 2003. Folder permissions can be listed and changed manually, or using templates with permissions settings created using the built-in wizard. Permissions can be applied to any number of mailboxes and folders at the click of a button. OWA Permissions Control Web application for viewing and modifying folder permissions. Combines features of the company's former OWA Delegate Control and OWA Public Folder Control. (5 Mar) PFAdmin Tool from the Exchange 2000 Resource Kit to change permissions and replication settings for a folder and its subfolders. Does not work with any version of Exchange after Exchange 2000 SP1. Also see: XADM The Pfadmin Utility Does Not Work with Error Message OpenAddressBook Failed, Error 0x40380 XADM Error Message When You Set Permissions on Public Folders Invalid Windows Handle ID No 80040102 Exchange System Manager PFDavAdmin Free tool from Microsoft for managing permissions on public and mailbox folders, including all the way down to the item level. Requires .NET Framework. For use with Exchange 2000 Server, Exchage Server 2003 and Exchange Server 2007. PFInfo Tool from the Exchange Server Resource Kit for generating a file with information about public folder permissions and replicas. See XADM Error Message Opening Address Book When Running the Pfinfo.exe Utility. Public Folder Utility View folder permissions and other properties. Export folder properties and permissions to a text file or relational database for analysis. Send customized messages to folder owners. Manage orphaned public folder client permissions. SetPerm Allows you to set default permissions on individual folders within mailboxes throughout your organization or on groups of mailboxes. Free.
Set All Calendars to Reviewer	Many organizations want people to not only see each other's free/busy times but also get appointment details. Therefore, they want to enforce a policy of using Reviewer as the default permission on each user's Calendar folder. This is not a capability built into Outlook, but you can perform this task with some of the tools above. If you want to experiment, you could also create a custom application using CDO and the ACL Component from the Platform SDK to manage permissions; a version of Acl.dll compiled for Windows NT/2000 is available from Microsoft's FTP site (this site is not always responsive). If you need a Windows 95/98 version, you'll have to compile the C++ source yourself. More information: Sue Mosher's pre-conference Workshop from Microsoft Exchange Conference 99 -- The PowerPoint presentation for Segment 5 (324kb) includes details on the ACL model. The source code (473kb) includes a sample Outlook 2000 VBA project that runs on Windows NT only. Professional CDO Programming

Connecting Microsoft Exchange Server to a POP3 Account

Downloading mail from a POP account into Exchange Server in general is not a supported configuration (though you wouldn't know it from the proliferation of tools to accomplish this). However, Microsoft does provide an Exchange Connector for POP3 Mailboxes for Small Business Server, having licensed Internet Mailbridge from Acotec. It's still not a supported configuration for full Exchange Server.

I'd suggest that you read about both sides of this issue. If you decide to try this route, there are plenty of tools to choose from. Features common to all POP3 collectors include the ability to POP mail from multiple servers and from multiple mailboxes. You'll also be able to control how often it checks for new mail in the POP3 accounts and limit it to downloading new mail during specific hours, such as only during business hours. All of the commercial POP3 connectors should be able to deliver mail to any SMTP address, including mail-enabled Public Folders.

Many of these tools support these forms of POP3 collection:

One-to-one, which collects mail from a single POP3 mailbox and sends it to a specific SMTP address

Many-to-one, which collects mail from multiple POP3 mailboxes and sends it to single SMTP address

One-to-many, which collects mail from single 'catch-all' POP3 mailbox and sends it to multiple SMTP addresses

Note that the one-to-many collection method fails when messages are BCC'd to users.