Resolved -
The affected ISP has issued their RFO and advised a hardware card failure on their network backbone. An ISP field engineer dispatch was required to move traffic off. A new card was shipped to site and was replaced by the field engineer. Note, ETHERFAX kept the incident open although in a monitoring state until the repair was completed by the ISP.
Mar 19, 14:24 EDT
Update -
Engineers are beginning to move some services back to primary ISP. An issue was noted with DirectFax Email Services rejecting emails due to a configuration error with our domain validation check. This was just resolved.
Mar 6, 11:07 EST
Monitoring -
All services are operating as expected.
For any applications that remain unable to access our API or endpoints, ensure that outbound firewall rules permit this access. DNS changes made during failover events mean that hard-coded IP address rules or DNS entries may prevent normal operation.
Engineers will continue to monitor performance of the network into tomorrow.
Mar 6, 00:45 EST
Update -
Most services have restored fully through alternate paths. Engineers continue to work with our ISP and monitor performance.
Mar 5, 22:26 EST
Identified -
Our primary ISP has reported a widespread issue. We have completed DNS updates to route API traffic around the affected provider. Engineers are continuing to monitor services and failover processes.
Mar 5, 20:42 EST
Investigating -
ETHERFAX engineers are investigating an ISP level routing issue impacting connectivit to the API. Changes have been initiated to route around the impacted ISP.
Mar 5, 20:16 EST