Upgrade Email Signature Server to .NET 3.1
We upgraded the target framework for our email signature server from .NET Core 2.2, to .NET Core 3.1. The functionality of the email signature server was not changed. This was required because Microsoft dropped support for .Net Core 2.2.
Terraform scripts; Add kubernetes_version to input.auto.tfvars
We currently use a terraform script to install the Email Signature Server in Kubernetes in self hosted environments. Until now the version of the Azure Kubernetes Service was hard coded into the terraform script, this didn't allow to set it to the latest supported version. The version parameter is now exposed as an input variable, giving us and customer the opportunity to update it at installation time.
This change does not have a behavior impact on the Email Signature Server.
Invalid content id format
We have discovered a scenario which resulted in an email not having a signature attached. This affected a small subset of emails sent inside the organization.
The signature was not attached in case a user received an email with an attachment from inside the organization and later on he/she would forward the email also inside organization. If any of the recipients or senders were outside of the organization then this would not happen. This has now been fixed.
Handling Distribution lists in Email signature Server
Our current email signature server solution currently supports sending emails to distribution lists. However it had an issue when the size of the distribution list exceeded the size of the maximum recipient list, which has the default value of 500 in Exchange Online. An alternate solution has been implemented and now we support distribution lists of any size.
Upgrade VM size in terraform script
The Vm Size was set to D1_V2, which is only 1vCpu and the recomended one now is 2 vCpu, si it was upgraded to DS2_V2.
Refine powershell script that creates certificates for Email signature server installation Until now an admin needed to do several manual steps in order to generate a self signed certificate and export the certificate and related data into different formats. This powershell script automates this and all the admin needs to do is run it, enter a password and all the necessary output will be generated in the right format.
Failure inserting email signature for NullReferenceException
In a rare case when sending an email with a default client signature from a mobile device, a signature would not get applied. This was because of an unexpected HTML structure in the email when looking for the default client signature.
The structure of these emails can now be handled correctly.
Image gets removed from body when email has image just before default signature
When replacing a default client signature in an email there was a possibility that an image placed just before the signature would also be replaced. This has now been corrected so no visible element will be removed as part of replacing default client signatures.
Correctly detect emails sent from devices that do not have a mailbox
We are failing to update sent items for emails sent from devices, such as scanners. This is because the email address from which they are sent is not linked to a mailbox. We are now handling this case and skipping the update in Sent Folder process if no Mailbox is found for the sender email address.
Loganalytics workspace name must be unique
When creating a Email Signature Server environment with Terraform the log analytics workspace had a default name. This name is globally unique so this default name would not work in future setups. The log analytics workspace name is now part of the variables a tenant needs to configure.
Automate Email signature installation by using Terraform
We have introduced infrastructure as code (IAC) by using terraform with the aim of:
- simplify and reduce our setup process
- store the setup steps in code, with source control to provide history of the changes
- replicate a complete environment any number of times
Ensure default signatures are replaced without leaving up additional space
The problem we were facing is that if an email was sent from a mobile device and the email contained default device signature then upon replacing the default device signature with the Templafy generated one this one would be incorrectly placed one line bellow. Also if the user manually added multiple empty lines before the default device signature then this would be preserved, rendering the email signature only after these. With this improvement we make sure that in case an email has a default device signature then when we replace it with the Templafy signature this is placed correctly without having extra white spaces before.
Email Signature Server message visualizer
Note: This tool is only internal and is so far only available for developers of Email Signature Server. When investigating how an email will get a signature applied it can be cumbersome to process an email in the Email Signature Server to see the result. To make it easier to investigate where email signatures will be inserted and how the email was processed we can now use a visualizer tool. The tool can read files in EML, MSG, and HTML format and display the original and processed email side-by-side, among other information about the processing of the message.
Fail to retrieve email from exchange when trying to update sent item for read receipts emails
When accepting to send a read receipt for an email in outlook the Email Signature Server would not recognize it as an auto generated message and would apply a signature to the read receipt. This would also cause an error when trying to apply a signature to the sent item, which wouldn't exist in the senders mailbox. These read receipts are now correctly identified as auto generated messages and no signature will be applied.
Extend the logging to explicitly state Exchange Status Code returned
When an email would fail to send it was unclear which SMTP status code would be returned to Exchange. The returned SMTP status code will now be logged when an Email fails to send, which makes it easier to determine how Exchange will handle the email afterwards.
No reason logged when failing to fetch access token for the App Registration
In case the process of retrieving an access token to our app registration failed we would only get a generic error therefore not being able to see the real reason of the failure. This change logs the specific error code returned providing the details of the underlying exception.
Migrate to Email signature server to .Net Core 2.2.1
Currently our Email Signature Server component runs on .Net Core 2.0. Several performance issues that we experienced are linked back to this version of .Net Core so in order to solve them we decided to upgrade to the newest stable version, which is .Net Core 2.2.1. The change required us to update all our project files and also to use a new base image for our docker container. This change did not require any code updates or configuration changes in our orchestration framework. So the new version can be released to the rest of the customers without requesting them to do any changes to their hosting environment.
Auto-generated emails do not contain a valid sent date
In some cases an email will not contain a sent date. This is determined by the client that sends the email. We would previously not apply any signature to these emails, but with this change we will now use the current date as sent date instead of failing. This means that if an email is sent without sent date we will apply the signature that is valid at the time of processing the email, if a valid signature exists.
Obfuscate email addresses in table storage and slack integration for Email Signature Server
To be compliant with GDPR the Email Signature Server will now only save information about emails with obfuscated email addresses. This includes all log messages in table storage, slack, and all tracking information. Emails will be obfuscated with the same obfuscation format as website so they have the form "email@example.com".
Replacing default signatures can remove message when sent from Windows 10 Mail Client
When replacing the default email signature from Mail for Windows 10, part of the messages could accidentally be interpreted as part of the default signature. In such a case the default signature was not correctly identified and would result in the message being clipped. This has not been a production issue, as this particular email client is not in use, however, the issue has now been resolved.
Emails sent as RTF to external mail will not get signature applied
If a user without Outlook addins would send an email as RTF to a recipient outside the organisation there would be no signature applied by the Email Signature Server. This behavior is now changed so a signature will be applied when a user not using addins are sending emails as RTF to recipients outside organisation.
Email Signature Server monitoring doesn't log processing times
The Email Signature Server was no longer logging the processing time of emails since late November. This is now corrected so processing times for emails are now being logged again.
Ensure email signatures are only added to emails that do not contain a Templafy email signature already
This is extending our efforts on making sure that the email sent has a signature attached. Even if an email is sent from outlook with the VSTO add-ins enabled we check the content of the email to make sure that it contains a valid email signature. At the moment we are able to do this for email having message format of type HTML, which is the most common one. For plain text and RTF emails, we still rely on checking a header field value, which indicates if the VSTO add-ins were enabled when the email was sent. This story includes changes both for website and email signature server, the VSTO add-ins are not impacted. The website changes only contain the HTML template change on which the email signature generation is based. The customers are not going to get an email signature updated notification from VSTO.
Email with multiple HTML TextParts throws error
When sending emails from iPhone mail client with zip attachments the email will also include a HTML file. This file could be misinterpreted as a HTML body which caused an error when applying a signature. We will no longer now interpret HTML attachments as part of the email body.
Notify errors on Slack from Email Signature Server
This feature is the second step related to monitoring the Email Signature Server. In this feature, we focused on getting real time notifications if errors occur on our Email Signature server. We want to make these errors easier to address by making them more visible. We now stream these errors to a dedicated Slack channel: #log-emailserver-prod. Example errors are: to not being able to send an email, or not being able to update the sent item in the sender mailbox. For the moment, this feature is available only for Templafy and Templafy-hosted tenants (currently we have 2 partners).
Email Signature Server monitoring of events and run-time exceptions
The Email Signature Server for Office365 is supposed to process all emails sent from a tenant. In this story we take care of storing tracking and monitoring data for every email message processed. (Eg. UserIds, anonymized email address of the sender, type of email - new or reply/forward -, successful insertion, time used for processing the email and apply the signature, email signature server version ...). We collect useful data on the usage and performance of our service, but this story sets the grounds for future analytics work that we will eventually present to our admin users.
Email signature added in the wrong place for reply mail and not added following reply emails
When sending from a Sony Android device with default mail client the signature would either be placed incorrectly or would not be inserted at all. We are now able to handle this client and place the signature correctly.
Prevent invalid connections to Email Signature server
We had an issue where connections from non-Exchange Online servers could be made to the Email Signature Server. Although these connections wouldn't be able to send messages though Exchange, the Email Signature Server would still allow them to pass through. In order to avoid this, we have now implemented IP filtering based on the Exchange Online IP-addresses provided by Microsoft, meaning that any connections from a non-Exchange Online will now be rejected.
Make listening port configurable for email signature server
When testing Email Signature Server there are some difficulties always using port 25, like being unable to connect to that port from some locations. It is now possible to configure Email Signature Server to listen to another port than just 25.
Updating sent items fails in some cases
Updating an email in sent items would sometimes fail and the failure seems to correlate with the Email Signature Server having performance issues. The procedure of updating the sent items have been changed to avoid occurrence of this failure.
Ensure email sender validation in Email Signature Server
The connection between the Exchange Server and our Email Signature Server is done by IP. One malicious user, knowing our IP address, may directly connect to the Email Signature Server and send emails. This could lead to spoofing and impersonation attacks. For example, one could send an email as firstname.lastname@example.org, the email will be processed and will be sent back to Exchange Server as it was an email that came from it and should return. By this story, we ensure that the spoofing attacks and impersonation are not possible by using validation mechanisms. We now do not allow sending outside organization that does not contain a valid DomainKeys Identified Mail (DKIM) signature. DKIM validation is one method used for spoofing email prevention. The implementation requires the Office365 customer to have DKIM signatures enabled on all the domain registered (by following the steps provided https://docs.microsoft.com/en-us/office365/securitycompliance/use-dkim-to-validate-outbound-email ). DKIM verification is enabled by default but can be disabled by setting an environment variable in the Email Signature Server's Kubernetes setup. Exchange server is adding a DKIM Signature to all outside organization emails, that is validated by us. In case there is no DKIM signature or the signature is not valid, the email is considered spoofing and is blocked from sending. Emails inside the organization do not contain DKIM signature and are not put under validation. In the case of "Sent on behalf of" emails, the following rules are applied: Email Signature of the one the email is sent on behalf of will be applied Email is saved and updated in the sent folder of the sender (the one that sends on behalf of someone else)
Log insertion of signature when using plain text message handler
Until now it was not logged when a signature was inserted in a plain text email. This made it hard to see if a signature was inserted or not when troubleshooting. It will now be written to the log as information when a signature is added to a plain text email message.
Make it possible to disable EWS calls for load tests
In order to run performance tests we needed an option to disable web service calls to Exchange (Exchange Web Service). Disabling this will result in sent items not being updated and RTF emails will not get signatures added.
Set TLS 1.2 support in SmtpServer
In order to comply with the latest security requirements TLS was updated for Email Signature Server so TLS 1.2 is the default security setting.
Add StyleCop to Email Signature Server
To ensure the coding style is consistent across the entire solution, the Templafy.StyleCop package is added to make sure the rules are enforced.
Enable batch writing of log messages
We wrote log messages immediately, which resulted in some overhead since logs were written one by one. This has been changed to a more efficient approach, so we now write log messages in batches.
"Reply All" from sent items sets signature at the bottom
When using "Reply All" on an email in sent items in some edge cases the signature would be placed incorrectly. Signatures will now be placed correctly when using "Reply All" on emails in sent items.
Concurrency issue when two emails load default signatures at the same time
An error could occur in case two emails were being handled at the exact same time when default signatures needed to be loaded. This error would result in signatures not being added to the email. We now handle concurrent emails correctly in the case default signatures needs to be loaded.
Email Signature added 2 times, one before and one after the actual content
In some cases Email Signature Server would add a signature to the top of the email, even though a signature was already added by the Add-In. This situation could occur when creating a draft in Outlook Desktop with a signature inserted and sending it through Outlook Online. Email Signature Server will no longer insert a signature to the top when sending an email from Outlook Online which was created in Outlook Desktop with an Add In Signature already inserted.
Reply to certain email puts signature at the bottom
In some edge cases when replying to emails from outside the organisation, the signature would be placed incorrectly at the bottom of the email. Replying to these emails will no longer result in the signature being inserted incorrectly.
Default signature visible when sending with Outlook for Android Closed Default signatures was not being removed from emails sent with Outlook for Android. Outlook for Android removed the HTML tag that was used in inserted default signatures. We searched for that specific tag so, after this update, we were not detecting it anymore. This is now fixed by more dynamically identifying the default client signatures.
Add headers to emails processed by Templafy Email Signature server
Until now all emails sent by Outlook with the VSTO addins enabled were not forwarded to our Email Signature Server since they contained a special header. However in the future for analysis purposes we would require that all emails pass through our email signature server. To achieve that we changed the rule in exchange so that emails coming from Outlook with VSTO addins enabled will be forwarded to the email signature server and there they will get a new header called: Templafy-EmailSignatureServer-Processed: true.
Make sure we send the email
Until now if an error occurred in processing the message or when trying to update the sent items then it could theoretically happen that we don't send the email. We revised and changed so as long as we can retrieve an email we make sure we send it no matter what other components might fail.
Improve logging on configuration failure
In case there was a configuration error when starting up the email signature server we would only get a generic error message and we wouldn't be able to easily deduct which setting is wrong. Now we specify in the error message the exact details of the faulty setting.
Log message identification parameters in case message sending fails
When an error occured during a message processing we write an entry to our log repository. However in some cases we failed to log crucial identification data, like messageid and sender even though those were available. This is now added.
Add additional Email Signature Server tests
To make the system more reliable we added additional unit tests to verify both the flow and the insertion of email signatures.
Change configuration values to be explicit
Changed configuration variables to more accurately reflect what they are being used for, and removed hardcoded variables in favor of configuration settings.
Cleanup obsolete configuration files/update relevant ones
Since Email Signature Server was originally hosted on a different orchestration framework we still had some the configurations files related to that in the solution. These are now removed and only the actual ones are kept.
Sent items not updated
In some cases, the Email Signature Server was trying to update the email in Sent folder with the signature and not succeeding. This caused delays in sending the email or email not sent at all. This hotfix fixes it.
Sending email with add-ins disabled throws exception
When signature was not inserted (in case of no email signature available or signature already inserted in Outlook by a disabled Templafy VSTO Add-in), Email Signature Server was throwing an exception blocking the email. This is now fixed
Replace default device signatures
Default signatures added by email clients like "Sent from my iPhone" and "Send from Mail for Windows 10" are now replaced based on a master list maintained by Templafy, which also contain different translated versions. In HTML emails, default signatures are only replaced if a known signature container element with the exact text is found. In all other cases, the signature is inserted after the signature container element. In plain text emails, default signatures are only replaced if the last non-whitespace only line is identical to a default signature. In all other cases, the signature is appended.
Rename environment variables
We use environment variables to store setting related to the hosting environment and office365. Two of the environment variable names related to office365 didn't really make sense so we renamed them.
Email signature appended at the bottom of the conversation convert from html and danish
When you replied to a plain text email from Outlook (Windows or Mac) and convert the plaintext message to HTML, the Email Signature Server did not know where to introduce the signature so it was appended. This fixes the bug by adding support for PlainText to HTML conversion case.
Email signature inserted at the end of the email conversation
The problem was caused because of a case that was not covered in the Outlook Desktop identification rules. The problem is solved by adding this case of email bodies containing the email addresses between quotation marks in the rule base
Email signature not applied in sent items for email with attachment only
Email with large attachments would sometimes not be found in exchange because of the delay caused by the size. We increased the delay between search iterations in order for Exchange to finish processing the message.
Add option to log all email messages
We now support a global trace configuration for enabling tracing of all emails sent through the email signature server.
Stabilize Email Signatures for Office 365
Various improvements and fixes for signature insertion and email handling.
Support RTF mails
Email Signature Server now supports RTF emails by converting them to HTML through Exchange Web Services.
No signature added when replying from outlook for windows to an appointment
Replying to an appointment from Outlook for Windows was not working because the signature was not inserted but was appended at the end of the email and this was not supported. This is now fixed.
Send email with attachments fails if sender client is apple mail for ios
Sharing files from iOS devices through Apple mail was not working because of the Apple mail bug that converts the email body that is after the file preview. This is now fixed and the behavior is the one expected (with the Apple mail bug).
Signature not added to reply without the original message
When the original message of a reply/forward mail was deleted, the signature was not inserted at all. This is now fixed by appending the signature in case insertion matches.
Email signature not added when starting a reply email in outlook for windows and completing it in outlook online
Email signature was not added when starting a reply email in the outlook for windows and completing it in outlook online (IE browser). This is fixed now.
Ensure signed and encrypted emails (S/MIME) can be sent
Email signature server now avoids adding signature to signed and encrypted emails, this prevents corrupting the emails. It also removed DKIM signatures before adding email signatures, since Exchange will re-sign the mail with DKIM.
Don't throw exception when signature is not found
A small change in behavior, in case for some reason a user has configured his email signature, but it wasn't created and deployed correctly we no longer throw an exception, just create a log entry and continue.
Signature not added if email contains an .ics file and is sent outside the organization
Previously email signature were not added for mail with attached calendar events (.ics files), this is fixed now.
Investigate behavior when sending mail to groups and distribution lists
When sending to groups or distribution lists exchange generates automatic email for each of the user in the group, signature was not appended to these emails previously. It is fixed now.
Ensure Email Signatures are not added twice with add-ins disabled
Website: Email signatures generated by the Office add-ins did not have any identification, making it impossible for the email signature server to identify, whether they were already inserted, even with the add-ins disabled. This affected some users, where add-ins were disabled, thus getting the same signature applied twice. The email signatures generated by Templafy now contain an identifier, which can be found by the email signature server, instructing the process to skip the email from further processing. Email Signatures for Office 365: Emails sent with a signature generated by Templafy are now ignored from further processing.
Wrong conversion for special characters when replying to internal emails
Emails sent with a different encoding than UTF-8 would get destroyed when inserting the email signature. This has now been fixed and all email messages are now converted to UTF-8 to ensure compatibility with all character sets.
Sending meeting invites as TNEF gets converted to emails
Certain meeting invites didn't get correctly identified, when sent through the email signature server. This was caused by the fact that some mail clients sends meeting invite in TNEF, which was not previously encountered.
Email Signature for Office 365 is not added for internal mails
When sending emails within the same organization, messages would not get an email signature applied. We have implemented support for this scenario and are now converting the message to a format where we can insert the email signature correctly.