2014-08-01 – Mail server outage postponed

We have received word from our provider that the mail server move is not going to proceed today, and will be rescheduled for an unknown date in the future.

More information will be posted as it becomes available.

2014-08-01 – Upcoming mail server outage

The datacenter where our primary mail server is hosted has notified us that our server is being migrated to a new facility. Beginning August 1st at 6PM Pacific Daylight Time, the mail server will be shut down to be physically moved to a location. There will be no change to assigned IP addresses, hostnames or similar.

We have taken precautions with additional off-server and off-site backups, and deployed a backup mail server which will accept and hold mail for the duration of the outage, so we do not expect any mail to be lost or misdirected during the outage.

The total amount of downtime experienced by each server will vary based on its location in our current facility, but we expect that most systems will be down for no more than 4 hours, and that the greatest impact should not exceed a maximum of 8 hours.

If you have any questions or concerns, please open a support ticket. Note that our support ticket system will be functional during the outage, however, you may not receive email notifications.

 

Mail server not responding

At approximately 15:55 PDT (-0700) today the mail server went offline. I have a ticket open with the datacenter regarding the issue, they have a networking issue and are working to resolve the problem.

If you need immediate support, please open a support ticket or use the information on our contact us page.

Please accept our apologies and know that we’re working on addressing this issue as quickly as possible.

 

ActiveSync Server Offline – RESOLVED

Due to a licensing issue, our ActiveSync server is currently offline and is expected to remain offline until Monday morning. IMAP, POP3 and webmail services still function, however, mobile devices will be unable to send or receive email until this is resolved.

I am working on finding a temporary workaround to get this resolved faster.

UPDATE: I was able to find a temporary workaround, all systems are now running normally.

 

Mail server not responding

Our monitoring script reports that the mail server is not responding, we are investigating but it appears to be an issue with the datacenter.

Inbound/Outbound email delayed

For the last 1.5-2 hours, inbound and outbound email has been queued and held as part of the SMTP delivery service was offline. The problem has now been resolved and mail will be delivered shortly.

Email sent/received between users on our system were not affected.

Mail server slow to respond

We’ve had a few reports of intermittent mail server slowness in the last couple of hours. We believe that this is network related with our primary carrier and are working on confirming the problem. The server itself appears to be responsive and functioning normally.

Please feel free to open a ticket if you have specific concerns.

Sorry, <...> is not allowed access from your location — RESOLVED

If you have been receiving “<-- 550 Sorry, is not allowed access from your location” errors for the last handful of hours, please accept my apologies. The underlying issue has been resolved and we’re taking steps to avoid allowing this situation to occur again.

Mail Server Outage – Resolved

At some point over the evening last night, the mail server started returning “421 Too Busy” or similar errors to most/all requests. This issue has now been resolved, please accept our apologies.

We do have any automated tool that detects and correct this particular situation, but this time the process refused to terminate upon request, and it took human involvement to resolve, explaining the several-hour length of this outage.

Since most SMTP servers retry for 24-72 hours (and since our server was returning a correct “retry later” error), we do not expect any mail to be lost, and based on the current traffic graphs, mail is currently flowing at a very high rate as the mail flow catches up.

SSL certificate changes, SSL/TLS configuration changes

Just a quick note that we recently renewed our primary SSL certificates and are in the process of deploying the new certificate to all servers and services. This process is routine and painless and should not cause any issues for any users.

We also took the time to re-evaluate our SSL/TLS configuration, and have disabled SSL 2.0 due to the fact that it’s insecure and considered broken. As part of this process, we are implementing newer versions of TLS and adjusting the cipher priorities. In theory, this shouldn’t cause any problems at all, but in practice, if you run an extremely out-of-date browser, you may have difficulties connecting to our SSL servers.

Note that not all servers/services are updated at this time — The configuration will be applied automatically, but the changes may not take effect until each server/service is restarted which will happen over the course of the next week.

Please contact support if you have any concerns.