There are no Open Network Issues Currently
Fortigate Firewall Issue with Servers Australia (Resolved) Critical

Affecting System - Data Centre HA Fortigate Firewall

  • 05/02/2026 08:46 - 05/02/2026 09:10
  • Last Updated 05/02/2026 09:37

Fortigate Firewall Issue with Servers Australia

Scheduled 05/02/2026 @ 02:00 AEST - During the times listed in this maintenance, engineers will be performing firmware upgrades to our FortiGate firewalls in the Sydney region. These firewalls are in HA and customers should expect no service interruption under ideal circumstances, but please be aware that some firewall sessions and/or IPSec tunnels may reset during the switchover.

Upgrade went ahead as per notification

After the Upgrade they found that IPSec tunnel connectivity issues.

7:21 AEST Servers Australia with the vendor reverted back to previous Firmware and this resulted in 5 mins downtime

More info on the issue can be found at https://status.mysau.com.au/

UPDATE 9:10 AEST - Issue resolved

Issue with DR Replication Over night (Resolved) Critical

Affecting System - Cloud Infrastructure

  • 01/11/2025 06:44 - 01/11/2025 21:09
  • Last Updated 01/11/2025 16:06

Issue with DR Replication Over night

We are having some issues with DR Replication on our Cloud infrastructure were VM 3 has locked at 6:44 AEST (5:44 Qld time).

Servers Australia are already looking into it as to why repication has locked one of the VM

Connection to the Cloud PBX is at the moment offline

We appologise for this inconvenience.

More update as they been sent to us will be posted

 

UPDATE: 8:29 AEST - No updates at this time, engineers are still trying to unlock the VM

UPDATE: 8:33 AEST - Issue has been escalted to VMware to their 24/7 support as it seems the issue is within VMware

UPDATE: 9:25 AEST - Issue has now been resolved, more updates to be posted on this issue as this is not acceptable as to what have happened.

UPDATE: 10:18 AEST - Data Centre will be running manually DR replication after they have confirmed that everything is stable. Replication is the last step to make sure everything is 100% working fine.

Replication is expected to start at around 10:30 AEST.

UPDATE: 16:05 AEST - DR replication has been successful

CLOUD PBX Upgrade to V 5.3.4.3.0 (Resolved) Critical

Affecting System - CLOUD PBX

  • 18/09/2025 20:00 - 21/09/2025 08:33
  • Last Updated 19/09/2025 04:43

Cloud PBX upgrade to V 5.3.4.3.0

Our Software manufacturer for the Cloud PBX will be upgrading the software to resolve some issues with Ring Groups, plus some more enhancements in Queue Call Monitoring Reporting / Added Security and adding new features.

Our software manufacturer issued us with a window of 3 hrs but expected to start at around 20:30 on the 18/09/25 and can take approx. 30-50 minutes to fully complete.

During this time no incoming and outgoing calls can be made from the Cloud PBX

If for any reason your phones will not auto connect, please restart them after the upgrade
.

We will test all features after the upgrade and post more notifications.

--------------------------------------------

IMPORTANT NOTES from the software manufacture issued on the 17/09/2025:

Post upgrade completion required actions:

  • Cache Clearing: Please make sure all users will clear cookies and cache before starting using the new version and after login do a hard refresh once with any of the steps below...

1.   Hold down Ctrl and click the Reload button.

2.   Hold down Ctrl and click F5.

3.   Hold down Ctrl + Shift and click R.

 

UPDATE 20:12 AEST - Database backup started

UPDATE 21:22 AEST - Database backup finished, now starting the Upgrade

UPDATE 23:16AEST - Upgrade finished and testing started

UPDATE 19/09/2025 04:25 AEST - System check finished including APPS / IVR / Queues / Inbound and Outbound calls and everythng seems funtional. PS. Logging into the CLOUD PBX, please clear history / cache and cookies, otherwise you will get errors from the old links stored into your browser.

CLOUD PBX Issues (Resolved) Critical

Affecting System - CLOUD PBX

  • 29/05/2025 09:06 - 29/05/2025 14:05
  • Last Updated 29/05/2025 17:16

CLOUD PBX Issues

We just got informed from the dtat center, that there seems an issue

Data center are trying to move the severs onto a different enviorment and it failing for the moment.

They are trying to find the issue

 

More Updates to come.

 

UPDATE 10:32 am

We just got informed from the DATA CENTER, that a reboot of some hosts needs to be done in the cloud. Expected downtime of approx 10-15. This will happen in the next 15 mins.

This will only effect customers that are on the CLOUD PBX

 

UPDATE 11:12 am

We just been informed that reboot was done but unfortunetly VM cannot conect to network and they are trying to look what happened and fix.

 

UPDATE 11:29 am

Seems all back to normal as of 11:22am and we can see calls going through and conection fine. We still yet did not get the CLEARANCE from the data center as yet.

 

UPDATE  11:38 am

Issue reserfaced again

 

UPDATE 11:52 am

A full cloud System needs to be rebooted and replication needs to be moved in a different enviourment.

This will take approx 10 -20 mins

This will effect all services and SIP Trunks

All calls will fail till everthing will be online again

 

UPDATE 1:24 pm

SIP Trunks are all back online as of 1:06 pm

Cloud PBX still offline

 

UPDATE 2:05 pm

Data center have now contacted us and were told that the issue with networking on the CLOUD PBX should be resolved

We appologies for this

 

UPDATE 2:31 pm

Issue was a networking problem between vSAN host and VCD (Cloud Director) not comunicating properly on one of the vms.

CLOUD PBX - APPS Not receiving calls (Resolved) Critical

Affecting Other - CLOUD PBX

  • 15/04/2025 07:15 - 15/04/2025 19:00
  • Last Updated 15/04/2025 19:09

CLOUD PBX - APPS Not receiving calls

WDP Chrome APP / Android APP

Iphone APP should not be effected by this issue.

Overnight Google Push Servers has changed the behaviour on how to send calls via their Push servers. Due to this change incoming calls on WDP Chrome APP and Android APP stopped working.

Temporary Work Around for APP to work:

  1. Android APP - Leave APP in Forground
  2. WDP Chrome APP - No work around available as it expect the call to be processed by Google Chrome Browser.

Engineers are looking into it, as we speak, but it seems that some tweaking needs to be made to the apps and then upload the Apps for approval to Google. This can take some time to get approved after the changes and we expect that within 10 to 15 hrs, all will be resolved.

We appologize for this inconvenience.

Error from Google Push server as of this morning:

error

 

UPDATE 19:00: Issue is now resolved

Important Announcement regarding Cyclone Alfred in QLD (Resolved) Low

Affecting Other - WorldDialPoint Office Phones and email ONLY

  • 05/03/2025 10:07 - 14/03/2025 09:05
  • Last Updated 20/03/2025 09:06

Important Announcement regarding Cyclone Alfred in QLD

 

We would like to notify that our Office in Queensland will be under a direct hit from Cyclone Alfred and there is a chance that from Thursday the 06/03/2025 will will NOT have acess to any Electricity / Internet / Phones and Mobile service.

At this moment the QLD goverment have no idea what the damage will be but it is informing everyone, that distructive winds of upto 150klm/hr is expected with huge rainfalls and a lot of flooding. They are informing everyone that power can be OFF from a few days upto a week.

We do appologise for this inconvenience and will do whatever we can to reply to any emails and phone calls if possible.

We also would like to point out, that our services will not be impacted by what ever is happening in QLD as our services are hosted in Equinix SYD and set as a Private Cloud within 2 data centers.

Office Phone Sales / Accounts and Support Closed due to Maintenance (Resolved) Low

Affecting Other - WorldDialPoint Office Phones ONLY

  • 20/01/2025 09:00 - 22/01/2025 21:08
  • Last Updated 15/01/2025 17:16

Monday 20/01/2025 from 9:00 till 17:00 due to renovations in our Office on power supply.

All Phones calls for Sales / Accounts and Support Lines, will not be answered due to maintenance on the power supply.

Power supply will be off all day, until 3 phase electricity is installed and tested by electricians.

Please use our Ticketing system for any Account and Support Queries and we will get back to you as soon as possible. Office Phones will be back Online by Tuesday 9:00am.

We apologize for this inconvenience.

If you need URGENT support please open a Support Ticket and we will get back to you as soon as possible.

Opticomm Outage Service Lifts and Voice Services (Resolved) Critical
  • 31/10/2024 06:30 - 31/10/2024 10:35
  • Last Updated 31/10/2024 11:39

Opticomm Outage Service Lifts and Voice Services

OTT000000244518

Chrome APP failing to receive calls after x amount of hours (Resolved) Low

Affecting System - CLOUD PBX

  • 28/09/2024 06:53 - 12/10/2024 16:54
  • Last Updated 12/10/2024 16:54

Chrome APP failing to receive calls after x amount of hours

This issue is being investigated by the software manufacture and Google Chrome support.

After diagnostics from the software manufacturer, thay can see that the Chrome APP goes into deep sleep in the Chrome browser and it will NOT pop up when an incoming call from the push server is sent to the chrome browser.

Engineers are looking into it.

For the moment to overcome the issue is to logout and login in the chrome APP every few hours.

PS - Closing the browser and opening again, the app will stop working properly, you would need to logout and login again in the app, even if it shows as green.

 

UPDATE - 12/10/2024 - Issue is now resolved.

Customers from Brisbane with Aussie Broadband and their resellers cannot connect (Resolved) Critical

Affecting Other - SIP Trunks / Cloud PBX

  • 27/09/2024 08:30 - 28/09/2024 06:53
  • Last Updated 29/09/2024 07:47

We are getting reports that customers in Brisbane and Gold coast area that are with Aussie Broadband and their Resellers, eg: AGL ect... cannot connect

This was investigated and sent to our data center for further investigation due to the fact that tracerouting from customers premeses was showing broken route up to hand over to IX.

At 9:30 am the Data Center acknowledged the issue and contacted IX for more information

IX informed them that they know about the issue already but no ETA for a fix

At that time the Data Center, disabled all routes for IX and they started seeing traffic from Aussie Broadband IP addresses flowing fine

This is what our data center issued as notice at 9:39am this morning

 

----------

Identified - An issue has been identified with IX Australia's QLD-IX. They have provided a status incident available at https://status.ix.asn.au/incidents/9x5g2nr33nyw

This incident would have affected services in Queensland primarily, but could affect connectivity to services in other locations for traffic collected from QLD-IX and carried to other locations.

We have disabled all BGP sessions on that internet exchange and are awaiting advice from the provider. Traffic should now be flowing normally.

Sep 272024 - 09:39 AEST

APPS failing to receive calls (Resolved) Critical

Affecting Other - CLOUD PBX

  • 04/09/2024 13:32
  • Last Updated 07/09/2024 06:51

At around 13:30 pm today, we started receiving reports that APPS (Android / Chrome APPS) stopped receiving calls.

IOS APP not effected

After diagnostics where done, investigations all point to our PUSH SERVER stopped somehow processing calls to be sent to the APPS.

We have now engaged engineers from the software manufacturer to investigate and fix the issue.

 

More updates to come.

 

Update 15:17 - Issue still not resolved at this time

Update 17:08 - Issue still not resolved at this time

Update 17:57 - Issue found - Android and Chrome app, stopped working because Google this morning has changed their policy. Team is working on this on priority, as soon as we get more information we will post it here.

UPDATE 05/09/24 7:27 - Andriod APP is now fixed and receiving calls fine. If you are using the BLUE APP please uninstall and install the new APP via this link https://play.google.com/store/apps/details?id=com.hodupbx.hoduphone 

WDP Chrome APP is still not fixed at this time. More updates as they come will be posted.

UPDATE 07/09/2024 6:50am - WDP Chrome can now receive calls

Inbound issue with the upstream carrier (Resolved) Critical

Affecting Server - Business Service

  • 08/08/2024 12:30 - 16/08/2024 11:58
  • Last Updated 08/08/2024 13:01

Inbound issue with the upstream carrier not conecting

We have contact our upstream carrier regarding incoming calls and were told thet they are looking into it

We also received a notification from them:

Investigating - We are presently experiencing an outage on the Voice network.
Senior engineers are presently looking into the issue.

A further update will be provided in 30 minutes.

UPDATE - Issue reselved at 12:51 and inbound calls started working

CLOUD PBX Upgrade (Resolved) Critical

Affecting System - CLOUD PBX

  • 06/04/2023 20:00 - 06/04/2023 22:30
  • Last Updated 18/04/2024 08:53

We would like to inform our Cloud PBX customers that on 06/04/2023 at 20:00 there will be an upgrade of the CLOUD PBX.

The full upgrade is expected to take 3 hrs with a maximum downtime of 30 mins

This upgrade is a major upgrade specially the online GUI and enhancement of the Fax Server plus some security upgrades.

 

We apologise for any inconvenience caused by this short notice as we just got informed from the software manufacturer that this upgrade needs to happen as a priority and we will update once upgrade starts and finishes

 

-- 20:03 - Upgrade starting

-- 22:23 - Upgrade is now finished

Issues with Inbound and Outbound Calls (Resolved) Critical

Affecting Server - Business Service

  • 03/04/2024 12:07 - 08/04/2024 19:53
  • Last Updated 03/04/2024 15:34

We are having some issues with inbound and outbound calls .

Engineers are looking into it right now.

More updates will be posted as soon as it is available

 

Update 13:26pm - It has been found that some routing from third party providers are having huge issues and effecting our routing to both carriers.

We are awaing more info about the routing issue as it just been excalted from the Data Center

 

Update 15:03 - 

It seems the issue was resolved; it was something to do with the routing on the internet that was affecting us badly

It seems stable as off 14:20 AEST

 

Update - 15:31

Issue 100% resolved. We also got confirmation from Data Center about the cause.

Cause was due to someone injected a wrong outgoing route and was effecting some IP Subnets. Once it was identified and route deleted everthing stabilized.

Migration of our Cloud and Services (Resolved) Critical

Affecting Server - Business Service

  • 24/10/2023 23:00 - 02/11/2023 16:12
  • Last Updated 25/10/2023 01:21

Migration of our Cloud and Services

Private Data Center

Migration Cancelled for the 19/10/2023 and moved to the 24/10/2023

Stage 1

Start - 24/10/2023 @ 23:00 AEST

Finish - 25/10/2023 @ 1:00 am

Services effected - Cloud PBX / SIP Trunks / Business Services / Conference services / Fax2Email

Due to growth in the last 12 months, we would like to notify everyone that during the time above we will be moving all our services from a Private Cloud (HADRaaS) to a Private Data Center Environment (VM Cloud Director and Fortigate Firewall Hardware) in Equinix SY3.

For the time being all our services will be running without Failover, till Stage 2 is implemented. We were given the assurance from SAU that this would be safe for a few weeks, as they do not expect any issues with Equinix SY3 going offline.

During this time all our services will not be accessible, that means, it will be not possible to make calls / receive calls or access the online Portrait. The transfer should not take long, but engineers requested a 2 hr window, just in case, as routing / IP addresses and Licencing will also needs to moved.

We will be updating this post once the Transfer starts and finishes and all tests done.

What is expected:

21/10/2023 @ 1:00 AEST - First Replication of all existing VM's (DONE)

24/10/2023 @ 23:00 AEST - Start of migration.

  • Services will go offline and switched off.
  • Last replication of all VM's (approx. 30 minutes to finish).
  • Move replications to new Private Data Center
  • Move all IP addresses from HADRaaS to Private Data Center
  • Build all routing including the Fortigate Firewall hardware
  • Switch on to LIVE
  • Move licensing from HADRaaS to Private Data Center for Cloud PBX and Business Services
  • Testing to make sure everything is working fine

If for any reason any IP phone / APPS like softphones or if you are connecting with your own PBX (freepbx / 3CX / Ericson etc.. and it does not come online automatically, please restart manually, so registration will again be transmitted/initiated from your equipment as sometimes PBX's stop trying after some time. 

 

Stage 2 will be implemented within 4-5 weeks after this upgrade, were the physical Fortigate Hardware will be replaced with a new VM of Fortigate Firewall within VM Cloud Director and service replication in SY4 of the Private Data Center will get activated as a Failover / disaster recovery and backup. No downtime is expected in Stage 2.

 

UPDATE 24/10/2023 23:00 - Migration Started

UPDATE 25/10/2023 00:20 - SIP Trunks / Business Services back operational. CloudPBX is work in progress

UPDATE 25/10/2023 01:19 - CloudPBX is now fully operational. All testing has been done. If for any reason any IP Phones / APPS are not connecting, please restart them and it should connect.

Firewall issue (Resolved) Critical

Affecting Other - Wholesale / SIP Trunks / Carriers

  • 09/06/2023 12:30 - 09/06/2023 12:54
  • Last Updated 10/06/2023 19:19

We had an issue at 12:30pm with our firewall were some services were effected and thus either could not register or make / receive calls

This issue has now been resolved

 

We appologize for this issue

Issue with Data Flowing into our network (Resolved) Critical

Affecting Server - Business Service

  • 13/05/2023 17:29 - 15/05/2023 14:23
  • Last Updated 13/05/2023 19:02

There is an ongoing issue with data flowing into our network via Servers Australia Network

Issue is being investigated by Servers Australia Engineers

Issue started at 16:50 and still ongoing.

 

More Information on the issue can be found via this link - https://status.mysau.com.au/ 

 

UPDATE - ISSUE has been resolved at 18:42 AEST

Cloud PBX Upgrade to V 5.2.1.0.3 (Resolved) Critical

Affecting Server - Business Service

  • 13/09/2022 20:30 - 28/10/2022 10:52
  • Last Updated 13/09/2022 21:55

Cloud PBX upgrade to V 5.2.1.0.3

Softare manufacturer for the Cloud PBX (Cloud3) will be upgrading the software to resolve some issue with the Web Chrome Phone APP and IPhone / Android APP, plus some more enhasments in Queue Call Monitoring Reporting

Software manufacturer issued us with a window of 3 hrs but expected to start at around 20:30 on the 13/09/22 and take approx 30-50 minutes to fully complete.

During this time no incoming and outgoing calls can be made from the Cloud PBX

All other services  will not be impacted

If for any reason your phones will not auto conect please restart them after the upgrade.


UPDATE: 21:54 - Upgrade finished. Admin/Agents before login please CLEAR Browser Cache

.

Cloud Infrastructure Upgrade (Resolved) Critical

Affecting System - Infrastructure Upgrade and Migration to new Environment

  • 23/08/2022 21:00 - 24/08/2022 14:56
  • Last Updated 24/08/2022 07:06

Cloud Infrastructure Upgrade

Infrastructure Upgrade and Migration to new Environment

We would like to notify all our customers that on the 23/08/2022 between 21:00 and 23:59 AEST our Cloud Infrastructure will undergo upgrades and Migration to new Environment.
During this time no incoming and outgoing calls will be possible due to the upgrades

Software manufacturers gave a window of 3 hrs but this is expected to take less

We appologize for this inconvenience and we will keep this notice updated
One of the many benefits of this upgrade, is that within the next 12 months all SIP Trunks  will be True Geo Redundant and not based only within SAU Data centers infrastructure and IPv6

Any customers CPE that does not reconnect after the maintinance is finished, a reboot would be required.

UPDATE: 18:20 - Primary checks have started a little bit earlier than the time stipulated from the software manufacturer and they will start shorthly upgrading firmwares once all clear.

UPDATE 22:30 - Cloud in Maintinance mode as they are moving equipment and IP's.

UPDATE 22:49 - Cloud out of Maintinace mode and calls should be working now. Any IP phone or PBX that did not come back a reboot is required. We will be testing every feature

UPDATE 24/08/2022 02:30 - Testing completed at 2:30am and all test are good 
.

Conectivity Issue within Serversers Australia (Resolved) Critical
  • 08/08/2022 09:07 - 19/08/2022 06:46
  • Last Updated 08/08/2022 09:58

We are seeing conectivity issue within Servers Australia Data Centers On both Equinix Data centers

This has been reported and Engineers at Servers Australia are on it at the moment

https://status.mysau.com.au/

Aug 8, 09:17 AEST - Investigating - Network engineers are investigating connectivity issues to an edge switch in the Equinix SY4 facility.


Aug 809:30 AEST - Update - Network engineers have confirmed that this event is also impacting the Equinix SY3 facility. Further updates to be provided.


Aug 809:45 AEST - Update  Links Up and Cloud is rechable in both data centers Equinix 3 and Equinix 4

.

CLOUD PBX - Maintenance and Upgrade (Resolved) Critical

Affecting Server - Business Service

  • 08/07/2022 21:00 - 08/08/2022 19:29
  • Last Updated 09/07/2022 13:09

CLOUD PBX - Maintenance and Upgrade

Software manfaucter will be doing some work on the CLOUD PBX due to Goolgle Play Store changing requirments for the APPS (Chrome / Android). They will also be preparing the first steps for a future upgrade for IPv6 SIP support.

Calls during this time will not be possible

After upgrade is finished and your equipment does not conect automatically, please reboot device.
Any apps (Google Chrome / Android APP and IPhone APP will most likely needs to be logged out / resarted and logged back again to receive calls, due to new push is required.

We will be updating on here once everthing is done and tested.


21:55 - Upgrade Started

23:40 - UPDATE: All testing was succesfull. All APPS needs to be Logged OFF and Logged IN again
.

NSX Firewall Reboot Required (Resolved) Critical

Affecting System - Firewall

  • 08/06/2022 20:00 - 08/07/2022 08:48
  • Last Updated 08/06/2022 21:39

We would like to advise that the Data Center at SAU were the NSX Firewall is located, need to be rebooted
Any customers trying to make or receive calls will not be possilble till the reboot is fully complete.

Reboot should happen between 20:00 and 21:00 today. Once rebooted is started this should take approx 2-3 mins

Applogies for this issue but we could not get an exact time when it is happening but instead we were given a 1hr window.

UPDATE21:39 - Reboot Done succesfully at 21:19
.

Issue with Phone Numbers hosted with Vocus (Resolved) Critical
  • 21/04/2022 11:34 - 08/06/2022 12:11
  • Last Updated 21/04/2022 15:32

We have been getting reports for specific phone numbers that are hosted with the VOCUS network that are not receiving calls

Vocus has already been notified about the issue and they are looking at it

We will update once we receive more information from them.

12:28 pm- Latest Update from VOCUS

As per their latest update, we looking at a possible routing issue and now being escalated further higher to operations

12:35 pm - We can now see incoming calls from the VOCUS network hitting our network. It seems to be fixed but not yet a fully confirmation that is fixed

15:31 - Latest update regarding this outage is that the carrier had an unplanned call routing issue which they sorted out after re-routing the calls onto the correct network on their side.

​The said outage is now tagged as resolved
.

CLOUD PBX Upgrade (Resolved) Critical
  • 10/02/2022 22:00 - 08/06/2022 12:17
  • Last Updated 11/02/2022 08:16

CLOUD PBX Upgrade During the mention time on Thursday 10th February 2022 at 22:00 our CLOUD PBX will go in Maintenance mode due to Upgrade to the latest version 5.1.0.0

This will add the long awaited SMS integration addon and some other extra minor things.
Send and receive SMS
SMS notifications on missed calls

If for any reason after the upgrade your phones / APPS does not connect correctly, please re-start them manually
Regarding the APPS (Chrome APP / Android APP and iPhone APP) it is recommended to LOGOFF and LOGIN again using the QR Code for Android and iPhone.
For Chrome APP just log off and Login again so that a new push notification get created.

In the next version a new Chrome APP will be available also which should have BLF incorporated into it.

UPDATE - Upgrade will start in 2 hrs

UPDATE 22:11 - Upgrade started

UPDATE - 00:45 - Upgrade Finished and at 01:15 all testing has been done
.

Cloud PBX Maintenance (Resolved) Critical

Affecting System - Cloud PBX Night Switch upgrade

  • 15/09/2021 20:30 - 15/09/2021 22:12
  • Last Updated 15/09/2021 22:14

Night Switch Upgrade

On the 15/09/2021 between 20:30 AEST and 21:30 AEST Cloud PBX users will experience a 2-5 minute downtime where no calls can be made or received.

Engineers will be performing a maintenance on the Cloud PBX to resolve an issue for those customers that have a Night Switch programmed on their phones.
This will resolve the issue where the light turns green at random times or where the timeout of the phone exipries.


UPDATE 15/09/2021 22:12 Maintenance has now finished and Night Switch issue resolved

CLOUD PBX Chrome Extension (Resolved) Low

Affecting Other - CLOUD PBX Chrome Extension OFFLINE

  • 06/09/2021 19:55 - 22/10/2021 11:11
  • Last Updated 06/09/2021 20:00

Sotware Manufacturer had for the moment Disable the Chrome Extension due to incompatibility with the latest Chrome version and is freezing randomly.

They will be working out what the issue is and once it becomes available again we will post here.
.

NSX Firewall (Resolved) Critical

Affecting Other - Firewall issue

  • 04/08/2021 13:51 - 06/08/2021 07:53
  • Last Updated 04/08/2021 20:00

We are aware of an issue were all conections have dropped. This was related to our NSX Firewall and has been disabled and paypassed for now.

The root cuase is within Servers Australia network effecting all their customers. Link to issue https://status.mysau.com.au/ 

Services on our Cloud has been stabized after disabling the NSX Firewall.

We will be following it with Servers Australia as to why it happened.

UPDATE 14:25 - Issue reserfaced we are contacting Servers Australia

UPDATE 14:26 - Servers Australia directed us to their Network Outage page and was advised the Engineers are doing the most to find the issue.

SERVERS AUSTRALIA - 

The engineering team has confirmed the core networking device has experienced a further reload resulting in further disruptions at this point in time. On-site engineers are investigating further at this point in time.

UPDATE - 14:53
Engineers have confirmed the impacted core networking device is operational again and are continuing to review the affected device and cause of the disruption

UPDATE - 15:19
Engineers have confirmed the stability of the core networking device and will be monitoring the situation further. A report will be attached to this incident once investigative reports have been completed.

UPDATE 18:08 - Issue resurfaced again via Servers Australia

UPDATE 18:51 - Services are operational, however SAU engineers will need to conduct further emergency works this evening to mitigate the chance of a reoccurrence. Further details will be shared when the maintenance window has been locked in

UPDATE - 19:45 - Engineers will push emergency firmware upgrades to our core switching environment at 8 PM AEST tonight in the Equinix SY1 facility.

During this time any services in the Equinix SY1 facility or running through this facility will experience a disruption of up to five minutes.
.

CLOUD PBX Emergency Maintenance (Resolved) Critical

Affecting System - CLOUD PBX

  • 03/08/2021 21:15 - 04/08/2021 14:05
  • Last Updated 03/08/2021 21:55

CLOUD PBX Emergency Maintenance

Software manufacture will be implimenting an update in our Cloud PBX.
A 15 min downtime is required. All outgoing and incoming calls from all customers will not be conecting till maintenance is finished.

If for any reason your phones does not come back online automatically after the maintenance, please reboot phones to start connection.

Updates will be posted once maintenance starts.
.

UPDATE 21:24 - Maintenance started
.

Opennetworks / LBNCO outage Australia Wide (Resolved) Critical

Affecting Other - Opennetworks / LBNCO outage Australia for Fibre Phones and Lifts

  • 07/07/2021 06:03 - 03/08/2021 18:21
  • Last Updated 07/07/2021 11:35

We are aware of a MAJOR outage on the Openetworks / LBNCO Network effect all Fiber phone services and Lifts across all states in Australia.
The outage started at 6:03 am and is still ongoing.

The issue has already been reported to Failts@lbnco and even via SPACE ordering system at 6:20am.
Several calls were also made to their support line for a followup and the responce from them is at this time "We are looking into it"

We will keep this notice open and post any details as soon as it comes to us.

UPDATE: 07/07/2021 at 11:24 am
No Updates sent from Openetwork / LBNCO
.

UPDATE: 07/07/2021 at 11:35 am
Our monitoring showing that services are back ONLINE, there is still not Update yet from Openetwork / LBNCO
.

Issue with Inbound calls (Resolved) Critical

Affecting Server - Business Service

  • 30/06/2021 12:26 - 01/07/2021 07:55
  • Last Updated 30/06/2021 13:45

We are aware of an issue with inbound calls. Issue is within the carrier and we are trying to get in more information at this time.
.

UPDATE 13:18 - Issue seems to be resolved at 12:50pm, but no full confirmation that is 100% fixed as yet. We will update once we have more information on this matter.
.

UPDATE 13:44 - We have received the reason from our carrier of to what had accually happened:

Period From: 2021-06-30 12:15:00 AEST
Period To: 2021-06-30 12:50:00 AEST
Location: Victoria/NSW Internet services and National Voice.
Services Affected: Data and Voice Services
Detail:
Dear Customer,

We experienced a major network outage earlier that was caused by an upstream fault by one of our major Transit/Internet carriers.

Our engineers have manually fixed the routes which has brought up most of our client services

CLOUD PBX not passing any calls (Resolved) Critical

Affecting Server - Business Service

  • 26/05/2021 13:50 - 26/05/2021 14:53
  • Last Updated 26/05/2021 15:18

We are aware of the issue Cloud PBX is not passing any incoming and outgoing calls
Engineers are looking into it.

UPDATE 14:53. Call are now working again. Engineers are looking into the actual root issue

UPDATE 15:16 Root cause found and fixed permanetly. Cause was due to one of the modules for Freeswitch Licencing 
.

CLOUD PBX random drop outs of registration (Resolved) Critical

Affecting Server - Business Service

  • 19/05/2021 11:00 - 19/05/2021 20:06
  • Last Updated 20/05/2021 07:46

This morning at around 11am we began receiving reports of phone and calls dropping out on out CLOUD PBX.

At around 11:45 we could see errors happening but could not pinpoint to the cause, so the issue was escaled as preority to our software manufacturer to analyse the error and rectify the issue.

Software engineers acknowledged the issue and they are looking into as we speak.

First reports that we received for them is FreeSwitch core is crashing randomly within every hours give or take and thus dropping all registartions and calls till it restarts agian.

We will update as soon as more info come at hand
.

UPDATE- 16:03

Software manufacturer will upload a patch to the CLOUD PBX at approx 19:00 or little bit later. Once the patch is applied they will need to re-start the whole system so any calls and registration will drop out temporaly for a few minutes
.
UPDATE- 20:06

Patch applied and CLOUD PBX rebooted

Services not conecting (Resolved) Critical

Affecting System - Voice services

  • 17/03/2021 11:05 - 29/03/2021 16:47
  • Last Updated 17/03/2021 17:28

Servers Australia network are having huge problems and Engineers are trying to resolve all there issue

Link of servers Australia issues https://status.mysau.com.au/

We are waiting for them to fix the issues

UPDATE: 12:53 pm

Services seems to be stabilized now and we are seeing al lot of calls being succesfull. If you still did not connect please re-start your network so it can connect fine.

Even Servers Australia have now posted that all their issues are fixed

UPDATE:17:25 pm

Issues with Servers Australia Network were ractified and not more issues resurfaced from 12:27 pm
.

HADRaaS Nodes, and vSAN datastore Upgrade (Resolved) Critical

Affecting System - Cloud Environment

  • 31/10/2020 10:30 - 31/10/2020 14:53
  • Last Updated 31/10/2020 14:53

HADRaaS Nodes, and vSAN datastore Upgrade

Start - Saturay 31/10/2020 at 10:30am AEST (09:30 am QLD time)
Finish - Saturday 31/10/2020 at 11:00am AEST (10:00 am QLD time)
Total time expected 30 mins

Due to the distributed nature of HADRaaS, normally we can apply upgrades to the Nodes and vSAN with no downtime as the cloud is distributed in different data centers .

This is accomplished by moving the VMs on each node over to the other node temporarily, then upgrading each node while it is not running the VMs, the upgrade process normally has low or no impact while services continue to run on each alternating node.

 

This is the overall process in more detail:

vMotion VMs to a single node

Upgrade the empty node

vMotion VMs to the upgraded node

Upgrade the remaining node

Re-balance the VMs across nodes with DRS

Upgrade the vSAN witness

Upgrade the vSAN version

What is expected during this time?
Everthing should work with no interruption to any services, but there maybe moments when moving the nodes a 5-10 second drop out on calls with no audio.
We will be monitoring all services while all the above is happening behind the scenes.

UPDATE 13:50 - Upgrade finished at 11:45 AEST with any issues and everthing to plan

Inbound Call failure (Resolved) Critical

Affecting Other - Incoming calls failure due to routing issue

  • 17/09/2020 06:30 - 03/10/2020 12:40
  • Last Updated 29/09/2020 08:29

Inbound Call failure

We are aware of incoming calls failing to conect

Engineers are investigating this issue as High Priority.

Issue is within the SAU Network routing Policy

More updates to come

UPDATE 8:20 am
SAU Identified Issue - Engineers have identified an issue with edge devices which may be causing intermittent connectivity to some providers. We are working on a resolution to this issue and will provide further updates as available.

UPDATE 8:53
 SAU Engineers - We're working on resolving some issues with one of the devices in our SY1 facility which seems to have been causing intermittent reachability to this and a handful of other prefixes learned over peering in Sydney. Will keep you posted as we have more information.

UPDATE 11:16 am
Engineers are continuing to work on a fix for customers affected by this issue. Further updates will be provided as available.

UPDATE 12:40 pm
Engineers are continuing investigations, including with upstream peering and transit providers. As soon as a fix is available it will be implemented to alleviate the issues being experienced

UPDATE 02:00 pm
Issue seems to be resolved, 

UPDATE 02:29 pm
SAU Engineers Update - Engineers have identified the issue and are rolling out fixes to impacted customers as quickly as possible. Further updates will be posted here as these fixes are applied.

UPDATE 02:43 pm
SAU Engineers Update - Engineers have implemented a temporary fix for some impacted services while a permanent solution has been tested. The current ETA for this permanent solution to be deployed is 3:15PM AEST. Further updates will be posted when this has been completed. 

UPDATE 03:20 pm
Engineers have begun deploying a permanent fix to resolve the intermittent network reachability to some providers. Once this fix has been completed engineers will be monitoring the environment to ensure no further issues are experienced. Further updates will be posted when this status transitions to monitoring

UPDATE 04:05 pm
Engineers have deployed the fix and are now closely monitoring the network reachability to ensure there are no reoccurrences.

LAST UPDATE 29/09/2020 08:26 am

A postmortem for this incident has been published on Servers Australia Website

Read the full postmortem.


Inbound Issues via Vocus Numbers cutting out (Resolved) Critical

Affecting System - Inbound Numbers via Vocus effected

  • 16/09/2020 10:32 - 17/09/2020 06:55
  • Last Updated 16/09/2020 17:00

Inbound Issue via Vocus Numbers cutting out.

We are aware of some inbound calls that numbers are with Vocus are having issues with audio cutting out.

This issue already reported to Vocus Networks and they are trying to resolve it.

Inbound calls via Symbio are not effected
Outbound calls are not effected as we are not using Vocus for outbound calls

More updates to come

UPDATE 11:56 am
Vocus still investigating the issue

UPDATE 12:41 pm
Vocus still investigating the issue

UPDATE 14:10 pm
Vocus still investigating the issue

UPDATE 15:47 pm
We are getting reports from customers, that issue seems to be resolved even though no official notice from Vocus yet received.
We will post excatly their reply when Vocus send it

LAST UPDATE 16:58 pm

We just received an update from our carrier and have advised that call quality should be clearer now after making some changes to their network.

They are currently monitoring the service.

VSC took two corrective action

1) Around 2:30 pm, policer issue under POR176738 for vid 153 is rectified for the service UCA128242
2) Around 3:40 pm, policer issue under L3 interface for the same service UCA128242 has been rectified.

After the 2nd corrective action, Mr Melih confirmed service is looking much better. 

-

Firewall Upgrade (Resolved) High

Affecting System - maintenance

  • 04/09/2020 21:00 - 16/09/2020 11:57
  • Last Updated 04/09/2020 22:34

We would like to advise that NOC will be upgrading the firewall that sits infront of our Cloud infrastructure as a first defence, to protect our voice services.
After the upgrade the firewall will not be operating till more configuration is done next week. All voice service will still be protected by their own firewalls.

Due to this upgrade services will not be available or approx 5 mins between 21:00 and 22:00 AEST on the 04/09/2020.

After the upgrade if any of your services does not come back online, please restart your equipment

UPDATE: 22:32 - New Firewall installed. Maximum downtime was 2 mins

Symbio Issue on inbound Calls (Resolved) Critical

Affecting Server - Business Service

  • 17/08/2020 09:00 - 04/09/2020 06:30
  • Last Updated 17/08/2020 14:11

We have received a notification that Symbio Network is having some issues and working on it

Effected calls - Only inbound that are via Symbio, we are not using Symbio for outbound call.

Notice from Symbio:
Intermittent Inbound and Outbound call quality issues observed

9:00 AEST We are observing a surge of inbound traffic which in turn is causing intermittent audio quality issues.

Our engineers are currently investigating the issue and further updates will be provided shortly.

11:15 AEST
The issue still persists and investigations are continuing. 

11:20 AEST Status Update

This ongoing issue is impacting both inbound and outbound calls

11:40 AEST Status Update

A workaround is in place to alleviate the impact, our engineers will monitor and continue investigating the root cause

 14:08 AEST Status Update

Mitigation efforts have thus far been successful , last occurrence circa 10:30AM AEST , monitoring continuing at this stage to confirm resolution

SAU Emergency Maintenance on Routing (Resolved) Critical

Affecting Other - Maintenance Update from Servers Australia

  • 23/07/2020 01:00 - 17/08/2020 14:10
  • Last Updated 22/07/2020 15:34

Title: Emergency Network Maintenance - NSW

 

Details: During the times listed in this emergency maintenance window, engineers will be performing works to cut-over production services to a new routing environment in the Equinix SY1 & SY3 facilities.

 

While downtime to services is not expected or planned during this time, there is a potential for packet loss and latency issues to occur throughout the maintenance window.

These works are isolated to the Equinix SY1 & SY3 facilities however services in other NSW facilities may also be impacted during the maintenance window.

.

Openetworks Outage Nation wide (Resolved) Critical

Affecting Other - Openetworks Outage Nation wide

  • 21/07/2020 13:26 - 22/07/2020 15:30
  • Last Updated 21/07/2020 21:45

Please be advised that there is a nation wide outage on the Openetworks network
We already have notified LBNco but still no responce from them.

Update 15:31 - We received that there is an outage on AAPT and are waiting for them to fix the issue
No ETA was given

Update 21:45 - All services are back operational

.

Email2Fax Issues (Resolved) Critical

Affecting System - Sending Email2Fax not working

  • 26/06/2020 09:58 - 21/07/2020 14:26
  • Last Updated 26/06/2020 15:56

Please note that we have some issues when customers are sending Faxs it not working.

Engineers are looking into it

Fax2Email is not effected at this time

Please use Web2Fax for the moment

UPDATE: 15:56 Issue is now resolved

.

Cloud PBX Upgrade (Resolved) Critical

Affecting System - Upgrade to latest software

  • 24/06/2020 22:30 - 25/06/2020 01:39
  • Last Updated 24/06/2020 22:51

Software manufacturer for the Cloud PBX will be doing upgrades to the software. This will take approx 1 hour to finish everthing and testing.

If you are using the CLOUD PBX you will not be ble to make or receive calls.
This will also apply to FAX2EMAIL and Conferencing Services

Upgrade Started at 22:33
.

Randon Inbound Calls Failures via Vocus Only (Resolved) Critical

Affecting Other - Random calls failing or delay

  • 15/06/2020 11:02 - 24/06/2020 22:48
  • Last Updated 15/06/2020 14:40

Invoming Calls via VOCUS only or calls to Vocus Phone numbers Issues

Vocus has reported an issue with their voices service for oubound calls. They have found the issue and trying to reslove it

Also vocus has issued an outage links on their website at http://status.vocus.com.au/view-incident.aspx?IncidentID=565

Incoming Calls via Symbio are NOT effected


UPDATE 15/06/2020 12:36: 
The Vocus' engineering team are continuing the investigation into the Voice issues and are working to restore services as soon as possible. No ETR is currently available. Further updates will be provided within the hour

UPDATE 15/06/2020 13:17:
Vocus engineering are continuing their investigation into intermittent inbound\outbound call failures across IPTel, Accella, Engin, Gendband, and Mera services. Please be advised this is also impacting call diversions. No ETR at this time, further updates will be provided within the hour.

UPDATE 15/06/2020 14:00
Please be advised that issues with call failures have now been resolved.

Openetworks Outage (Resolved) Critical

Affecting Other - Outage

  • 15/06/2020 09:02 - 15/06/2020 12:31
  • Last Updated 15/06/2020 11:05

There is an outage in on OpenNetworks infrastructure

This has been reported straight away and Opennetworks have acknowledge the outage and looking into it.

No further updated received from them at this time, , once we receive anything it will be posted here.

UPDATE 11:04 am: Openetworks seems to be their network up again at 10:46 am

Openetetwork Outage (Resolved) Critical

Affecting System - Network Outage

  • 02/06/2020 12:27 - 15/06/2020 09:10
  • Last Updated 02/06/2020 17:26

There is an outage in SYD / Canberra and VIC.

This has been reported straight away and Opennetworks have acknowledge the outage and looking into it.

No further updated received from them at this time, , once we receive anything it will be posted here.

Queensland is not affected by this outage

UPDATE: 17:19 It seems that VIC / SYD and Canberra are now back online. Still no word from Openetworks (LBNCO)

.

Openetworks VIC (Resolved) Critical

Affecting System - maintenance

  • 05/06/2020 00:00 - 24/06/2020 22:49
  • Last Updated 02/06/2020 16:50

Planned Outage: OPENetworks VIC Sites

An upstream provider will be performing critical maintenance which may cause an interruption for up to 3 hours for OPENetworks' Broadband and Voice services in Victoria, between 12AM - 6AM on June 5th 2020.
Apologies for the interruption and thank you for your understanding.

Cloud Issue (Resolved) Critical

Affecting Server - Business Service

  • 19/05/2020 07:46 - 02/06/2020 16:48
  • Last Updated 19/05/2020 09:40

Incident Status

Degraded Performance

Components

Cloud Services

Locations

Sydney - Equinix SY1, Sydney - Equinix SY3, Sydney - Equinix SY4, Sydney - Syncom SYD1, Sydney - Syncom SYD2, Sydney - Vocus, Brisbane - Syncom BNE, Brisbane - NextDC B1, Brisbane - NextDC B2, Melbourne - Equinix ME1, Melbourne - NextDC M1, Perth - Equinix PE2



May 19, 2020 6:34AM AEST
May 18, 2020 8:34PM UTC
[Investigating] Degraded performance has been noticed with the Vmware VPS / Hadraas and Standard VPS Platforms. Engineers are looking into this now. Symptoms include high load / loss of connectivity.

May 19, 2020 6:59AM AEST
May 18, 2020 8:59PM UTC
[Investigating] Engineers are continuing to identify the source of the disruption and further updates will be provided ASAP. Vmware VPS / VPS / Hadraas Customers Possibly Affected

May 19, 2020 7:42AM AEST
May 18, 2020 9:42PM UTC
[Identified] Engineers have identified the source of the disruption and are working to restore services.

May 19, 2020 8:08AM AEST
May 18, 2020 10:08PM UTC
[Identified] Engineers are now finalising the restoration of affected services impacted this morning. A further update will be provided as post-restoration works are completed.


May 19, 2020 8:49AM AEST
May 18, 2020 10:49PM UTC
[Monitoring] Engineers have been able to restore services after further troubleshooting. Further monitoring is being performed on impacted services at this time while Engineers review further. A post-incident report will be provided once the root cause has been identified.

Symbio issue (Resolved) Critical

Affecting Server - Business Service

  • 16/03/2020 12:59
  • Last Updated 19/05/2020 07:47

Symbio issued a notice about incoming and outgoing calls
This outage is not with us but becasue Symbio carries a lot of traffic in Australia for incoming and outgoing, it is effecting some customers calling certain numbers.

"Due to unprecedented call volumes, we are experiencing congestion on our carrier interconnect trunks. Our engineers are investigating. We'll provide an update once more information is available."

Conectivity Issue Within the Data centers (Resolved) Critical

Affecting Server - MOR

  • 14/01/2020 18:57 - 16/03/2020 12:58
  • Last Updated 14/01/2020 22:06

We are expierincing an issue regarding conectivity within the Servers Australia network

We have received a notice about 30 minutes ago advising that engineers are looking at the issue within Servers Australia Network

Incident notification from Servers Australia

Current Status: Partial Service Disruption

Started: January 14, 2020 6:35PM AEDT

Resolved:

Affected Infrastructure:

Components:  Cloud Services, Colocation, Dedicated Servers, Infrastructure, Mail Filter, Managed Wordpress Hosting, Network, Service Desk, xDSL/EFM

Locations:  Brisbane, Brisbane - Equinix BR1, Brisbane - NextDC B1, Brisbane - NextDC B2, Brisbane - Syncom BNE, Melbourne - Equinix ME1, Melbourne - NextDC M1, Melbourne - Vocus, New Zealand, Perth - Equinix PE2, Sydney, Sydney - Equinix SY1, Sydney - Equinix SY3, Sydney - Equinix SY4, Sydney - Global Switch, Sydney - NextDC S1, Sydney - Syncom SYD1, Sydney - Syncom SYD2, Sydney - Vocus

 

Update: Engineers are investigating network issues at some sites.

Further updates will be provided ASAP.

January 14, 2020 7:15PM AEDT
January 14, 2020 8:15AM UTC
[Investigating] Network engineers are continuing to investigate this issue, and will provide further information as available.

January 14, 2020 7:51PM AEDT
January 14, 2020 8:51AM UTC
[Identified] The issue has been identified to be a routing issue within the network. This is being worked on by a team of engineers and services are coming online and then going offline again, we are working to resolve this as soon as possible.

January 14, 2020 8:27PM AEDT
January 14, 2020 9:27AM UTC
[Identified] Engineers are continuing to restore services, we are working as fast as possible. Services should start to restore shortly. We thank you for your patience.

January 14, 2020 9:17PM AEDT
January 14, 2020 10:17AM UTC
[Monitoring] Engineers are currently monitoring the recent issues. All services should now be stable, however we will continue to monitor the situation for any further issues.

Issue with DAT to Evoice servers (Resolved) Critical
  • 14/12/2018 14:28 - 08/03/2019 16:31
  • Last Updated 14/12/2018 14:47

We are having some issues with the hosted PBX on servers 27.50.78.2.
Engineers are investistigating

UPDATE - 14:36 - Engineers found the casue and it seems some data corruption happened on the HAAS envierrment effecting 27.50.78.2. Engineers are trying to fix the issue
More update to come

Update - 14:45 - Hosted PBX should be up now and fully functioning.

Openetworks planned network (Resolved) Critical

Affecting Other - Upgrade

  • 28/11/2018 23:00 - 07/12/2018 19:22
  • Last Updated 23/11/2018 18:04

We would like to advise that Openetworks will be doing an upgrade in their Brisbane Data Center.
This Upgrade will effect all customers that have services on the ONU

During this time customer on the Openetworks infrastructure will no be able to make or receive any calls


NOTCED issued by Openetworks



Please be advised there will be a planned network outage for approximately 7 hours from 11:00pm Wednesday November 28th until 6:00am Thursday November 29th.

This outage will affect all RSP Broadband Data and Voice services at all Queensland sites.

Furthermore, any OPENetworks provided services, e.g. Wi-Fi services, will also be affected at all sites Australia wide.

This outage is necessary to perform an upgrade to carrier equipment within the Brisbane data centre.

 

Please visit the Network Status page on our website to obtain the latest update on this outage at: https://www.openetworks.com.au/support/network-status

Apologies for any inconvenience this outage causes and thank you for your understanding.

 

Regards,

NOC OPENetworks Pty Ltd

Migration to new Hardware (Stage 1 of 2) (Resolved) Critical
  • 15/08/2018 00:01 - 27/08/2018 18:10
  • Last Updated 23/08/2018 00:57

Stage 1 of 2 Migrating to High Availability and DRaaS Solution between 2 data centres (Equinix 1 and Equinix 2)

 

At midnight the 15/08/2018 Evoice SIP Softswitch will be migrating to a new hardware with extra redundancy between EQUINIX 1 and EQUINIX 2 in case of hardware failure in one data centre.

There will be some downtime involved of approx. 30 minutes.

 

In the last 2 weeks we have managed and prepared the new platform with continuous synchronize of the old platform with the major bulk of the data already moved.

The last step is to move IP addresses and sync.

Customers does not need to do anything as everything should come back normal after the downtime.

Stage 2 will involve the same for our wholesale system but this is planned for one week after.

UPDATE - 00:05 Migaration started

Update - 01:12 am Migarting Finished and all services online as normal



STAGE 2 

At midnight the 22/08/2018  at 00:01 am our Wholesale / Main SIP Softswitch will be migrating to a new hardware with extra redundancy between EQUINIX 1 and EQUINIX 2 in case of hardware failure in one data centre.

There will be some downtime involved of approx. 30 - 60 minutes.


UPDATE 21/08/2018 21:40 Stage 2 Migration moved to 23/08/2018 at 00:001am

UPDATE 23/08/2018 00:05 Migration Started

UPDATE 23/08/2018 00:35 am Migarting Finished and all services online as normal
.

Outage in data center (Resolved) Critical

Affecting Server - Business Service

  • 13/07/2018 09:43 - 17/07/2018 12:58
  • Last Updated 13/07/2018 18:09

Current Status: Service Disruption

Started: 13/7/2018 9:39am (+1000)

Resolved:

 

Affected Infrastructure:

Components:  Cloud Services, Colocation, Dedicated Servers, Infrastructure, Managed Wordpress Hosting, Network, Service Desk, xDSL/EFM

Locations:  Brisbane, Brisbane - NextDC B1, Brisbane - NextDC B2, Brisbane - Syncom BNE, Melbourne - Equinix ME1, Melbourne - NextDC M1, Melbourne - Vocus, New Zealand, Perth - Vocus, Sydney, Sydney - Equinix SY1, Sydney - Equinix SY3, Sydney - Equinix SY4, Sydney - Global Switch, Sydney - NextDC S1, Sydney - Syncom SYD1, Sydney - Syncom SYD2, Sydney - Vocus

 

Update: Network engineers are currently investigating network interruptions.

Further updates will be provided when possible.

Status Page: https://status.mysau.com.au


UPDATE 10:08

Update: Engineers have confirmed connectivity has been re-established network-wide.

A PIR for this incident will be provided ASAP.

Please contact our helpdesk if you experience any further issues with your services.

Status Page: https://status.mysau.com.au


UPDATE 10:12

Update: Some services are still affected by this and currently being worked on.

Additional updates to come.


UPDATE 10:42

Engineers are still working on restoring services.

Another update will be provided shortly.

UPDATE 13:57

Issue has been resloved from SAU at 11:39am.
We have been monitoring since then and no issues reported

UPDATE 15:57

We just received a NOTIFICTION from SAU that they need to Reboot certain services at 16:30
This will take approx 3 mins to complete.
After this reboot  we are told that any other issues will be resolved.

UPDATE 16:10

Update: Engineers will be implementing a fix to resolve packet loss and latency issues at 4:30 PM AEST.

Between 4:30 PM and 4:40 PM clients may experience up to 3 minutes of downtime to services while engineers apply a fix to some devices in the Sydney, Melbourne and Perth network.

Outbound Call Issues on one of our carriers (Resolved) Critical
  • 17/05/2018 12:01 - 13/07/2018 09:44
  • Last Updated 17/05/2018 13:00

Outbound Call Issues on one of our carriers started at 12:01pm

We have found that one of our outbound carriers was not terminating any calls or they had huge delay to conect and they were not sending any error back to us but instaed they were sending us that the call is being progressed and thus our second and third carrier could not kick in.

We have now eliminated this one carrier out of the mix at 12:40pm and now all outbound calls are conecting fine and no delay.

We also have requested this specific carrier for a please explain and what will be done moving forward so this issue will not happen again.
If we are not satified with the outcome then this carrier will be dropped permanetly.

We appologise for this mishap
.

Emergency Vlan setting change required (Resolved) Critical

Affecting System - 27.50.78.1 / 2 / 9 / 10

  • 23/04/2018 22:00 - 17/05/2018 12:51
  • Last Updated 24/04/2018 10:32

Start time - 23/04/2018 22:00
Finish - 23/04/2018 22:30

We would like to advice everyone that the data center where all our cloud services are, needs to do some EMERGENCY work on our VLANS.

Due to this change there will be an outage of approx 30 to 60 seconds.
While this is happening no calls will be able to be made or received,

This is required, as one of the seniour enginners at SAU have found on all IP's on a particular ingress point to the network are not being correctly leaked out of the peering VRF into the switching fabric. This basically means that the IP's are taking an older, slower path that's showing packet-loss under a certain condition.
.

UPDATE 22:09 - Starting the changes now
.

Update - Work finished at 22:12

Maintenance Required by software vendor (Resolved) Critical

Affecting Other - All services

  • 15/12/2017 22:30 - 15/12/2017 22:45
  • Last Updated 16/04/2018 19:16

Maintenance Required by software vendor

Last month there was a software patch to our cloud instances which we thought was fixed, it's now been brought to our attention there needs to be another patch to ensure stability is maintained.

 

So now we are booked in for this Friday night 9PM to 11PM the same time, we finally have a solution patch from the software vendor which has been tested thoroughly in a replicated Dev cloud environment. 

Again it will be around 15 minutes from start to finish

We do apologize for any inconvenience caused

UPDATE 22:45 - Maintenance Finished. Total down time 10 mins, everything back online


Emergency Maintenance Evoice Stsrem (Resolved) Critical
  • 16/04/2018 21:00 - 23/04/2018 14:52
  • Last Updated 16/04/2018 19:15

There will be some interuption on the evoice system to reselove an issue that should clear the issue we had last Friday 13/04 at 16:00

Engineer will be working on the cloud 27.50.78.2 for approx 30 mins between 21:00 and 22:00 and there could be some interuption with incoming and outgoing calls and should not last more than 10 minutes.

We appologize for this inconvenience caused.

Packet Loss on our links (Resolved) Critical

Affecting Server - Business Service

  • 08/03/2018 15:00 - 12/03/2018 17:00
  • Last Updated 12/03/2018 17:07

We are seeing random packet loss on our links and have reported it to the data center. We were advised that they are trying to mitigate the issue and clear all the random packet loss

More updates to come

UPDATE 16:29 - The data center is trying to advertise our IP range out of Vocus

UPDATE 12/03/2018 17:00 - Issue now fully resolved. The issue was randmly packet loss of an interval of approx 2 hours  for a duration of 2-3mins on AS 45671 via Vocus. Now AS 45671 has been routed via Equinix, you will only see vocus if your provider is peering with in brisbane but in saying that the packet loss is not happening anymore via vocus peers in the last 24hrs.

Migrating of Prifixes to new switch (Resolved) Critical

Affecting Server - MOR

  • 05/12/2017 21:00 - 14/12/2017 15:06
  • Last Updated 05/12/2017 16:15

Tonight we will proceed in moving our prefixes over to a new switch to where the voice cloud servers are, and from a core device we are looking at decommissioning in the future.

This will take the below prefixes offline for up to two minutes while they are dropped and picked back up in OSPF on the new switch:

27.50.78.8/29
202.130.35.64/29
27.50.78.0/29

Data Center Cloud Maintenance (Resolved) Critical

Affecting Server - MOR

  • 10/11/2017 22:00 - 05/12/2017 16:14
  • Last Updated 10/11/2017 21:21

We just have been informed from the data center that a reboot is required due to some urgent maintenance on one of the cloud services.
This will be impacting only services that are conecting to IP 27.50.78.10 no other services will be interupted.
Services will be unavailable for approx 10- 15 mins between 22:00 - 23:59 AEST tonight

We could not get an exact time frame due to the complexity of the data center cloud services but expected to be at around 22:00 AEST

We appologise for the short notice but this has also got us by surprize too.

UPDATE - 20:17 Reboot of cloud services finished
.

Phone numbers certain batches (Optus Issue) (Resolved) Critical

Affecting Server - Business Service

  • 02/11/2017 14:04 - 03/11/2017 16:00
  • Last Updated 10/11/2017 19:40

On the 02/11/2017 we noticed that out main number 0731077420 was not receiving calls fromthe mobile and pstn network accross Australia, we then have put a fault with our carrier for Urgent fix.
After a while we started receiving email that some other numbers are also not working.

After extensive diagnostics with checking the numbers, we have found that there is an issue with some ranges of phone numbers that they cannot receive any calls from any network in Australia or overseas.
We have esaclated the issue to the appropiate carrier straight away

Phone numbers that are ofline are in the ranges as follows:
073107xxxx (back online at 7pm on the 02/11/2017)
029007xxxx (back online at 3pm on the 03/11/2017)
039912xxxx (back online at 1:30pm on the 03/11/2017)
088122xxxx (back online at 4pm on the 03/11/2017)

UPDATE 16:00 Optus has advised that all there numbers efected are back online

On a different note we will be moving any numbers away from Optus that we have left to our main carrier.

This is the whole incident report from Optus

Updates from upstream:

03-11-2017 12:06

We currently understand this issue to be affecting inbound calls for some ported numbers that are being delivered by Optus.
Optus have scripted a solution for this problem which is scheduled to finish running as of 1pm AEDT and will see numbers restoring as they are reconfigured on the network.

We have escalated this issue to our Account Manager but are unable to have our numbers prioritised due to the nature of the fix for the fault.

We will provide further updates as they become available.

 

03/11/2017 07:00
Internal team has successfully tested 25 number ranges after testing a re-configuration deployment. They have identified a total of 34,000 number ranges affected by the outage. Internal team has advised rollback action plan and engaged Tier 2/3 to support in deployment. They have begun rollback of select service changes with a tentative ETR of around 5 hours

03/11/2017 03:00
Switching resources have advised that too many services remain to be restored (around 28,000 number ranges) to be completed manually. We are now awaiting an update from our internal team regarding their plan to bulk change configuration to restore all services. No ETR is yet available but a further update is expected during early morning hours.

03/11/2017 01:00
Switching team working alongside Tier 2 to manually restore services is still ongoing. Tier 2 engaged internal team to see if changes can be scripted and deployed for faster resolution. No ETR yet available for full restoration

02/11/2017 23:00
Optus switching engineers continuing to restore services 100 number range at a time. Services will slowly be coming back over the next few hours but no ETR is yet available

02/11/2017 21:00
Optus Engineers have recognised a fix and are now manually correcting the ranges to have been affected. The root cause is still being investigated at this time. There is no ETR yet but services are slowly being restored.

02/11/2017 19:58
Optus has received several reports of calls failing to individual numbers and ranges that appear to have been ported to another network.

At this stage the root cause has not yet been identified however our Incident Management team have engaged the relevant fixer groups to investigate and restore services.

This case is of the highest priority within Optus Wholesale and Networks.

There is currently no ETR but we will provide further information as soon as it becomes available.

BBvoice Network Outage (Resolved) Critical

Affecting Server - Business Service

  • 13/09/2017 07:40 - 08/10/2017 06:43
  • Last Updated 13/09/2017 15:17

BBVOICE network  (27.50.78.10) has lost a core router.
Enginners has been already despached to the data center at 7:am when monitoring started giving errors

This is effecting incoming calls customers who are conecting via 203.176.185.10 and all incoming calls

ETA at this moment is 08:45 am

Core router changed and service moved on to it.
Service fully restored at 8:00am

Issue restarted (it is not the core router) currently have engineers at our data centre rectifying the issue.

UPDATE - approx. 10:00am - 10:15am EST

UPDATE 10:21 - ETA changed we should receive the next update bu 10:35 am

UPDATE - 10:36 - ETA Moved 11:15 am

UPDATE 11:25 - Engineers are still working on the issue, we will update again within the next 30 mins

UPDATE 11:48 - We just received from the Data center in MLB that core Components of the network will be UP and running within the next hour and then external connectivity will follow

UPDATE 12:20 - SERVICES ALL ONLINE

Voice Outage (Resolved) Critical

Affecting Server - Business Service

  • 21/08/2017 13:51 - 04/09/2017 07:29
  • Last Updated 21/08/2017 14:24

There is an outage in Servers Australia network that is not allowing traffic to flow properly and thus no audio or no connection.
At the moment there are engineers on site looking into it

UPDATE: 13:56 Network engineers have been alerted to a down transit port in our Vocus Doody St data centre.

UPDATE: 14:05 Transit links back online. Services also back normal

UPDATE 14:23
We have received confirmation
Engineers have restored connectivity to all offline services and our provider port is now back online.

One way Audio on random numbers (Resolved) Critical
  • 11/08/2017 08:53 - 21/08/2017 13:53
  • Last Updated 11/08/2017 10:47

Details are as follows:

 

-------------------------------------------------------------------

Type: Un-Planned

Severity: Critical

Period From: 2017-08-11 08:30:00 AEST

Period To: 2017-08-11 00:00:00 AEST

Location: Melbourne Only

Services Affected: One Way Audio and no DTMF

Detail:

The issue with our upstream carrier has started again this morning.

Optus have been advised and we hope they will rectify the issue soon.

UPDATE - 11/08/2017 at 09:51 am

Optus have done a temporary fix by resetting their equipment to resolve the present issue.

Optus will be relocating all our numbers that is effected to a new switch during the course of the day for the permanent.

Our Carrier was already in the planning stage of moving our numbers to a new interconnect away from Optus that will provide far greater control and will allow us to be more nimble and to act much faster if any issues arise. Unfortunately when problems have occured on the Optus network they move slowly to resolve.

UPDATE - 11/08/2017 at 10:15 am

Issue have restarted again, Optus has already been notified, hopefully they will restar the equiment again.
Will keep you updated

UPDATE - 11/08/2017 at 10:45 am
It seems to be working again but at this time no update from the carrier as yet

Issue with Routing and IPV4 in data center (Resolved) Critical

Affecting Server - Business Service

  • 19/06/2017 18:29 - 11/08/2017 08:55
  • Last Updated 19/06/2017 20:06

Engineers at the SAU data center are investigating an IPV4 issue affecting services in Sydney. Updates to follow.

Update 1 19:20 - Issue seems to be now resolved but we do not have any confirmation yet from the SAU Data center. We will update once we receive full confirmation

Uptae 2 20:04 - Data Center just confirmed that the issue is fully resolved
.
.

Upgrade - WorldDialPoint Network Upgrade (Resolved) Critical
  • 13/04/2017 21:00 - 13/04/2017 21:35
  • Last Updated 15/05/2017 19:04

WorldDialPoint Network Upgrade

UPGRADE FINISHED


THIS IS AN UPGRADE

Affecting Server – Evoice Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2017-04-13 21:00:00 AEST

Period To: 2017-04-13 21:35:00 AEST

Location: SYD (Evoice Network)

Services Affected: Hosted PBX / Conferencing / Sip Trunking and other services conecting to 27.50.78.2 – Upgrade

Detail:
Upgrading the SIP Software to Version 4.1

 

Engineers will be performing an upgrade to our SIP servers. This needed to be done to eliminate some software bugs that were found lately effecting Attendant Transfers via a Queue using the star code  / Conferencing system and SIP Trunking.

Important - If you are connecting using an on-site PBX (eg. FreePBX/Asterisk or any other flavour and after the upgrade you cannot make calls out, you will need to check your configuration trunk setting and make sure you have the entry FROM=USERID (eg FROM=xxxxx*xxxxx where the x represents your userID) This is required for SIP trunking to work properly. This setting will not effect customers connecting with an IP PHONE or an ATA.

On a positive note, this upgrade will give the ability receive calls using 1 extension (register the extension on different phones and when someone calls your number it will ring on every phone, previously it will ring on the last phone that registered last with our system). This is called SIP FORKING

During this VoIP upgrade calls will be interrupted for at least 30 mins.

A completion notice will be published on our website once we have completed our internal testing to confirm the completion of the upgrade.

 

Please contact our helpdesk via support@wdpvoip.net.au if you experience any issues after the upgrade.

We apologise for any inconvenience this may cause and we appreciate your ongoing support.

Update 14/04/2017 00:30
Upgrade only took 35 mins and was finished at 21:35. After that we started extensive testing including  cpe registration, up until 00:30 am and all testing has been successful.

Inbound Calls (Resolved) Critical

Affecting Server - Business Service

  • 03/03/2017 08:57 - 31/03/2017 18:26
  • Last Updated 03/03/2017 18:56

Details are as follows:

 

-------------------------------------------------------------------

Type: Un-Planned

Severity: Critical

Period From: 2017-03-03 08:40:00 AEST

Period To: 2017-03-03 12:34:00 AEST

Location: National

Services Affected: VOIP

Detail:

 

We are currently experiencing an outage affecting all incoming calls. The issue is higher up in the network with our upstream carrier.

Engineers are working on this now and we will advise when fixed.

We apologises for the inconvenience caused.

UPDATE 10:00 - Vocus found the fault and they are re-uploading the numbers in the database.
We should start seeing some inbound calls working again

UPDATE 10:15 -Vocus Notice Issuedhttp://status.vocus.com.au/view-incident.aspx?IncidentID=234 

UPDATE 12:05 - Please be advised that according to the upstream carrier the restoration process is underway although is taking longer than anticipated.

At this point we expect full restoration before 12:15 AEST.

We will advise once services have been restored we will provide full details from our upstream carrier.

UPDATE 12:34 - We are adviced that All phone numbers has been restored 

Data Center Emergency Maintance Required (Resolved) Critical

Affecting Server - Business Service

  • 23/02/2017 02:00 - 03/03/2017 10:03
  • Last Updated 22/02/2017 16:16

Type: Scheduled
Status: Open
Start Date: 2017-02-23 02:00:00
End Date: 2017-02-23 04:00:00
Severity: High Impact
Services Affected: Sydney - Network

Event Description
Data Center network engineers at SAU have identified a serious congestion issue affecting the core network in Sydney.
To resolve this issue, the engineers will be performing emergency changes and upgrades to ensure the continued stability of the network.

During the above times, some customers may see a short service disruption, however due to the redundant nature of the Sydney core network most services will not see any interruptions as traffic finds an alternate route.

The result of this work will be increased capacity and stability for all customers in Sydney. This issue has been identified in the last 24 hours, and is therefore classed as emergency maintenance due to the severity of the issue.

Telstra Network Issues to Landlines and Mobil (Resolved) Critical

Affecting Server - Business Service

  • 02/02/2017 13:31 - 22/02/2017 16:12
  • Last Updated 02/02/2017 14:34

We don't have much information at this point but have been made aware that a fire at a Telstra Chatsworth exchange NSW.
Its Apparently staring to affect Mobiles calls nationally too and impacting some services to/from mobile and fixed nationally.

We will update you further if we receive more information.

UPDATE - 14:34
Telstra Outage:
Please be aware that Telstra have notified us that the fire at the Chatswood Exchange in Sydney this afternoon has been contained, but damage has been caused to some power infrastructure.
Anecdotally we are hearing that some services are already back up

Issue with 61390344xxx range not conecting (Resolved) Critical

Affecting Server - Business Service

  • 16/01/2017 08:45 - 20/01/2017 10:28
  • Last Updated 17/01/2017 12:28

We have been advised that there is an issue with some number ranges with the carriers on the number range 039034xxxx

This issue has already been escalated to the carriers involved earlier this morning and it is being looked at.


We can see some numbers ranges are back online but it seems that what left 0390344xxx seems to take longer than anticipated.

We will update this info as soon as it comes available from the carriers.

UPDATE - 11:15

Via Telstra the number range 0990344xxx confirmed and resolved.
What is not working yet is via Optus and Vodafone.
AAPT is being checked at this moment

UPDATE -
17/01/2016 - 12:27

We have just received:
Optus have corrected their data for 03903448XX range. So this should be completely fixed now.

Issue with Inbound calls and some outbound ca (Resolved) Critical

Affecting Server - Business Service

  • 21/10/2016 11:01 - 03/11/2016 11:57
  • Last Updated 21/10/2016 15:32

We are having some issues with inbound calls and some outbound calls.
Engineers are working on the issue
We will update as soon as we have more information

UPDATE - Issue resolved at 11:15 for incoming calls

UPDATE - 11:25 Issue fully resored all inbound and outbound working

LAST UPDATE- 15:21

Details of incident are as follows:

-------------------------------------------------------------------
Type: Un-Planned
Severity: Critical
Period From: 2016-10-21 11:01:00
Period To: 2016-10-21 11:16:00
Location: National
Services Affected: Voice Issue - Resolved
Detail:

In Summary we had a core voice failure across the network between 11:01pm and 11:16pm.
This was accidentally caused by human error while attempting to add further monitoring to the voice network.
Due to the nature of the outage, the backup voice gateway did not automatically kick in.

The problem was identified straight away and manual intervention was required to bring up the backup voice gateway.
Due to the flood of connections flowing back into the network after the voice gateway was brought back online caused the intrusion protection to
kick in and began blocking some of the connection attempts.
Our engineers responded by white listing the connections as quickly as possible to restore all voice connectivity back to working order.

Timeline of events
------------------
11:01-11:02 - Primary voice gateway lost connectivity to network
11:02-11:08 - Secondary Voice Gateway brought online manually
11:08-11:10 - Intrusion Prevention Alarms began alerting
11:10-11:15 - Engineers began white listing blocked IP's

All voice network connectivity restored @ 11:16
Total outage time: 15 minutes
------------------

This outage did highlight a type of fail scenario that we did not envisage, this has been noted by the engineers who will add further checking to pro-actively detect
such an outage and automatically fail over.

On a second more positive note we have just completed as of yesterday a full day high level voice workshop with our voice and networking
engineers and have got a high level action plan to bring the voice network back to 100% stability and to even higher increased levels
of reliability that customers can depend on.

Each engineer has been tasked to complete their work over the next four weeks with the most important tasks to be completed over the
next week to bring a high level of stability across the network with a number of very important sub projects completed one week at a time.

The main aim and service we hope to provide all our clients when we have completed this project is the following:

===4 week time line===
-A highly stable service from our core Voice network to deliver always on inbound and outbound services.
-A fail-over scenario proven and tested that we can fail-over across multiple pieces of equipment and still deliver inbound and outbound services to all our clients.
-A more scalable and robust Hosted PBX service.

Once the above schedule has been completed we will email out and advise accordingly of our progress.


===End of first quarter 2017===
-A complete redundant setup that will operate out of a separate location in order that you can rely on inbound and outbound services close to 100% of the time.


The above is one of the most critical upgrades we will be embarking upon and we understand how important your inbound and outbound services are with us
and I ensure you I am personally running this project with our engineers to ensure we eradicate any further major outages once these additional measures and precautions are taken.

I am also happy to come out and assure each one of our clients of our project plan and process and how this will be implemented.

We do apologise for today's event and assure you we are moving as fast as we can on our project not only to deliver and improve a more stable voice network but to deliver additional services, monitoring and notifications that we have not done before.

Data Traffic from Data center OFFLINE (Resolved) Critical
  • 22/09/2016 12:11 - 22/09/2016 12:33
  • Last Updated 21/10/2016 11:10

We would like to inform you that there is an issue in the data center .It seems that SAU are having some problem with the data flow.

Our engineers are in contact with the data center

UPDATE 12:40pm - Data center routes are all online at 12:32:50. Conection should be fine now.
We will be asking for a full report as to what has happened and remediation
We will also update this info in due time (when report is recieved)

MELBOURNE INBOUND ONLY - UPSTREAM OPTUS ISSUE (Resolved) Critical

Affecting Other - MLB Phone Numbers

  • 24/06/2016 12:30 - 24/06/2016 15:00
  • Last Updated 28/07/2016 18:37

Dear Customer,

 

OPTUS is having MELBOURNE wide inbound issues .

Hence your Melbourne DID may have issues .

Engineers are working on this issue to be rectified asap.

Currently the upstream carrier has not given us an ETR .

 

Please be advised that this is NOT a WorldDialPoint issue , its higher up with the carrier OPTUS.

We will update you as soon as we have any more information.

If you have any questions, concerns or issues - please don't hesitate to contact our support team

UPDATE - 24/06/2016 15:10 We just received that issue is now resloved.

Inbound calls Interruption (Resolved) Critical

Affecting Server - Business Service

  • 31/03/2016 14:14 - 31/03/2016 14:20
  • Last Updated 02/05/2016 21:26

Inbound calls Interruption at 14:14 on 31/03/2016 for several minutes.

We have received the report of what has happened at 2:14 pm today.
Issue inbound calls stopped working

ISSUE

There was a switch failure at the IX-NSW peering exchange at 14:14:50. There was no failure in our network.

IX-NSW is a peering exchange where various ISPs interconnect. There would have been a brief disruption to connectivity between us and other ISP's/Carriers's who peer on IX-NSW. In addition to the peering we have two upstream transit carriers. Connectivity would have been restored once the BGP routing reconverged and activated routing via another path

UPDATE 14/04/2016 - RFO issued from IX Australia Download here

Upgrade Wholesale SIP Infrastructure (Resolved) Critical
  • 31/03/2016 21:00 - 31/03/2016 21:30
  • Last Updated 14/04/2016 09:43

Affecting Server – Ivoice Wholesale Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2016-03-31 21:00:00 AEST

Period To: 2016-03-31 21:30:00 AEST

Location: SYD (Ivoice Network)

Services Affected: All VoIP services – Upgrade to X8

Detail:
At 21:00 on the 31/03/2016 we will be upgrading our wholeslae softswitch from version X7 to version X8.
All services on this service will be unavailble for approx 30mins.
No other services will be interupted on the other 2 networks.

We will update this notification once upgrade is finished

UPDATE 31/03/2016 21:40 - Upgrade complete and tests completed

Inbound Calls to NSW 02 Numbers (Resolved) Critical

Affecting Other - OPTUS OUTAGE

  • 17/03/2016 09:44 - 31/03/2016 16:14
  • Last Updated 17/03/2016 10:55

We would like to advise that and inbound calls to any 02 numbers are having issues.
Issue has been investigated and excalleted to Optus
No Eta was issue yet

We will keep updating as more info comes in hand

UPDATE 1 17/03/2016 10:31am - Issue seems to be Resloved at 10:05 am and 02 incoming calls are working again  as we still did not receive as yet Confirmation that is fixed for good.

UPDATE 2 17/03/2016 10:43 am - We received confirmation that issue is now resolved. We have requested a RFO report from Vocus who will in turn request one from Optus. We expect that Optus in particular will take some time to prepare their report. As soon as we have the details from the upstream carriers we will provide you with further details of the outage.

Emergency Cloud Migaration (Resolved) Critical
  • 04/03/2016 18:25 - 07/03/2016 17:35
  • Last Updated 04/03/2016 18:33

Emergency Cloud Migaration needed to be done at 04/03/2016 at 18:25 to an other Virtualazation.

Voice services will be lost for approx 30 seconds to 120 seconds


UPDATE 04/03/2016 at 18:30 Migration Finished

.

Evoice Cloud System reboot required (Resolved) Critical
  • 23/02/2016 23:00 - 04/03/2016 18:21
  • Last Updated 23/02/2016 15:54

Affecting Server – Evoice Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2016-02 - 22 23:00:00 AEST

Period To: 2016-02 - 22 23:05:00 AEST

Location: SYD (Evoice Network)

Services Affected: All VoIP services – Reboot Required

Detail:

We would like to inform everyone that tonight 22/02/2016 at 23:00 we would need to apply a software patch to update the cloud infrastructure. This require us to reboot our system and thus all services will be unvailable for approx 3 mins.

We apologize for this inconvenience.

UPDATE 23/02/2016: Software Patch has been moved for tonight at 23:00

Routing issues with Data Centre (Resolved) Critical
  • 11/12/2015 14:15 - 22/02/2016 16:17
  • Last Updated 18/12/2015 13:25

Affecting Server – Evoice Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2015-12 - 12 14:15:00 AEST

Period To: -------------------------

Location: SYD (Evoice Network)

Services Affected: All VoIP services – Upgrade

Detail:

We are having some issue in the data center at Equinix.
We are in direct contact with them to resolve the issue ASAP.

What we are told is that they are having routing issues.

UPDATE - 15:08 - Data Center is looking as to why routing is broken

UPDATE - 15:57 - Data Center inform us that the issue is IDENTIFIED and working on a resolution

Update 17:35 - Issue resolved at 16:37 and Comfirmation received that routing is stable from within SAU core network in SYD1 and SYD 2 in Equinix.
We will be posting the RFO from SAU as it is received.

Update 18/12/2015 13:25 - RFO has now been issued and you can view it via this link (Needs to be logged in into your account) https://wdpvoip.net.au/control/dl.php?type=d&id=16

WA Inbound Numbers 086365 numbers (Resolved) Critical

Affecting System - Inbound Phone numbers

  • 20/08/2015 07:50 - 20/08/2015 09:57
  • Last Updated 20/08/2015 10:00

We are currently experiencing an unplanned disruption to inbound calls from Telstra services to the following numbers:

 

61863651800-61863651899

61863651900-61863651999

61863652150-61863652199

 

This problem has been raised with our carrier and is being pushed with Telstra at the highest priority.

 

We apologise for the disruption and will advise you as soon as we have an update

UPDATE - 20/08/2015 at 09:57 am - Issue has been resolved by Telstra


Emergency Upgrade - WorldDialPoint Network Up (Resolved) Critical
  • 07/08/2015 23:30 - 08/08/2015 03:07
  • Last Updated 08/08/2015 03:07

WorldDialPoint Network Upgrade

UPGRADE Finished

THIS IS AN EMERGENCY UPGRADE

Affecting Server – Evoice Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2015-08-07 23:30:00 AEST

Period To: 2015-08-08 01:30:00 AEST

Location: SYD (Evoice Network)

Services Affected: All VoIP services – Upgrade

Detail:
Upgrading the SIP Software to Version 3.7

 

Tonight Engineers will be performing an emergency upgrade to our SIP servers. This needed to be done to eliminate some software bugs that were found lately effecting Hosted PBX and Email2Fax. 

During this VoIP upgrade calls will be interupted for at least 30 mins.

A completion notice will be published on our website once we have completed our internal testing to confirm the completion of the upgrade.

 

Please contact our helpdesk via support@wdpvoip.net.au if you experience any issues after the upgrade.

We apologise for any inconvenience this may cause and we appreciate your ongoing support.

Update 1 - (07/08/2015 at 17:07) - Engineers will be performing the upgrade as stipulated as it has just been re-confirmed by them. 

Update 2 - (07/08/2015 at 20:12) - Upgrade moved to start at 23:30 Sydney AEST/13:30 UTC 

Update 3 - (07/08/2015 at 23:45) - Upgrade Started

Update 4 - (08/08/2015 at 01:30) - Upgrade Finished

Issue with calls dropping (Resolved) Critical
  • 11/05/2015 08:30 - 10/07/2015 16:19
  • Last Updated 11/05/2015 09:37

We would like to let you know that calls are dropping out we already have an engineer looking into the issue and we will updade as soon as possible

UPDATE - 09:30 - Issue resloved

WorldDialPoint Network Upgrade (Resolved) Critical
  • 08/05/2015 23:59 - 10/07/2015 16:20
  • Last Updated 09/05/2015 04:06

WorldDialPoint Network Upgrade


UPGRADE FINISHED

Affecting Server – Evoice Service | Priority - Critical

Type: Planned

Severity: Critical

Period From: 2015-05-08 22:59:00 AEST

Period To: 2015-05-09 04:00:00 AEST

Location: SYD (Evoice Network)

Services Affected: All VoIP services – Migration and Upgrade

Detail:

1. Moving Voice Platform into a Private Cloud

2. Upgrading the SIP Software to Version 3

 

WorldDialPoint will be migrating our voice platform to a new Private Cloud on the 8th of May 2015 (Friday) starting from 11:59PM (AEST) until 04:00AM, 9th May 2015 (Saturday).

During this migration the VoIP service will be disrupted for up to 4 hours while the migration takes place.

You might be required to restart your CPE equipment onsite the following day if your services do not automatically register.

A completion notice will be published on our website once we have completed our internal testing to confirm the completion of the migration and upgrade.

 

Please contact our helpdesk via support@wdpvoip.net.au if you experience any issues upon receiving the completion notice.

We apologise for any inconvenience this may cause and we appreciate your ongoing support.

UPDATE - 08/05/2015 22:40 - Upgrade will start in 5 mins

UPDATE - 09/05/2015 15:30 - Migration ansd upgrade sucessfully finished at approx 01:30am . All testing done finished at 03:30am.

Data Loss on SYD BGB (Resolved) Critical
  • 04/03/2015 12:30 - 04/03/2015 19:20
  • Last Updated 05/03/2015 08:50

We just received from the data center an Outage Notice.

The data center in Equinix are having some issues with BGB

Intermittent SYD BGP Issue
Effect Level Critical
Type Name Unscheduled
Subject Intermittent SYD BGP Issue
Time Started 2015-03-04 12:30:00
Time Finished 0000-00-00 00:00:00
Services Effected Co-Location SYD, Ethernet
Status Name Open
Notes 12:30 04/03/15

Engineers are aware and are working on an Intermittent SYD BGP Issue and will update when available.

We will update the Notice as soon as it is fixed

UPDATE RECEIVED 14:50 - Data Center is working on BGB routing issue and No ETA has been given yet

UPDATE RECEIVED 15:00Engineers continue to work on the issues which are still causing intermittent problems to those located within this location. We apologise for the inconvenience this may be causing

UPDATE RECEIVED 19:20 - Issue Resoved

Data Center Upgrades in Equinix and Tuggerrah (Resolved) Critical
  • 20/02/2015 22:00 - 04/03/2015 15:44
  • Last Updated 18/02/2015 13:20

Dear Customers

We have received from the Data Centers where we have our services co-located an Advisory Notice, that some upgrading of their infrastruture will be happening in the next few days.
These upgrades should only impact our services for a total of 30mins downtime or less for each occassion.

Please read the Notification that has been issued:

Outage Type:

Scheduled

Services Affected:

Co-Location SYD, Co-Location TUG, Dedicated SYD, Dedicated TUG, IP Transit

   

Start Time:

2015-02-20 22:00:00

End Time:

2015-02-21 02:00:00

Outage Status:

Open


Advisory Notes

In the last few weeks, we have made numerous changes to our SY1 core network. These changes were needed due to ongoing issues with our old core network.

To complete these core migrations there are a few services left that still need to be moved over. Please see the below schedules for further information.

Friday 20th Feb 2015 - 10.00pm - 10.59pm (No Downtime expected)
Our engineers will be on-site at Equinix SY3 replacing a failing top of rack switch.

Customers co-located in SY3:L1H4:100:808 connecting to SW-6-12 will experience up to 15 minutes of service unavailability in the above window, during the removal and installation of the new top of rack switch.


Friday 20th Feb 2015 - 11.00pm - 11.59pm (Expecting some downtime)
Our engineers will be on-site at Equinix SY1, moving the uplinks of 2 top of rack switches to our new core network.

Customers located in SY1:271 and SY1:1749 will experience up to 3 brief 60 second outages whilst spanning tree re-converges and traffic is moved to their new paths through our new core switches in the above window.


Saturday 21st Feb 2015 - 12.00am - 12.30am  (No Downtime expected)
Our engineers will be migrating our PIPE IX service to our new core routers, to ensure customers are always seeing the best paths into our network from all providers.

There is no expected impact to any customer services.


Saturday 21st Feb 2015 - 12.30am - 1.30am (Expecting some downtime)
Our Australia engineers will be migrating the last of the links to our Tuggerah facility. Customers currently running over our primary link will be moved to our redundant path via our new core.

Once all traffic has been migrated to our backup path, we will be moving our primary path to Tuggerah to the new core network to facilitate further redundancy and greater throughput.

Customers located in Tuggerah will see up to 3x 60 second outages whilst traffic is being moved. There will be no impact to customers whilst the primary path is being moved.

Upgrading routers at NextDC (Resolved) Critical

Affecting Server - Business Service

  • 16/02/2015 23:00 - 16/02/2015 23:59
  • Last Updated 18/02/2015 13:08

Type: Planned

Severity: Critical

Period From: 2015-02-16 23:00:00 AEST

Period To: 2015-02-16 23:59:00 AEST

Location: National

Services Affected: Transit

Detail:

Dear Customer,

During the above maintenance window one of our main carriers will be upgrading routers at NextDC.

The outage will cause a brief disruption while traffic switches to a redundant path.

Please contact our support team should you have any questions, comments or concerns regarding this notification.

Network Issues via our links (Resolved) Critical
  • 30/01/2015 13:16 - 30/01/2015 14:00
  • Last Updated 05/02/2015 16:08

We are receiving errors from routes accross on our netwrk and engineers are trying to rerout traffic via other carriers.

We apologise for this and will update again within 15 mins


UPDATE 13:29- We have received back from the data center

Unfortunately we are experiencing an issue in our Sydney Datacenters which is affecting Connectivity.

Engineers are working on the issue with highest priority.

UPDATE 14:00 - Issue resloved at 13:51.
We are waiting for a report back from the data center.


UPDATE 14:35 - Data Center confirmed that it was a Broadcast Storm and still investigating who was at fault and why the datacenter go effected.
This is what we dealt with today http://www.webopedia.com/TERM/B/broadcast_storm.html 

UPDATE 05/02/2015 16:03 - Data Center issued Incident Report and can be found here https://wdpvoip.net.au/doc/IncidentReport2015011801.pdf 

SIP connectivity Issue (Resolved) Critical
  • 20/11/2014 22:30 - 02/12/2014 10:03
  • Last Updated 20/11/2014 23:17

We are aware of a problem with SIP connectivity issue and Engineers are working on it.

Will update more in the next 30 mins

UPDATE - 23:10 Issue is now resolved. 
Engineers reported packet loss in Tugerah Data Center. This was confirmed by the Data Center and found a problem with an Optus link. As soon as this link was pulled Offline. Tugerrah Service started behaving normally and services resumed.

Issues with Inbound Numbers (Resolved) Critical
  • 16/07/2014 12:15 - 12/11/2014 16:11
  • Last Updated 17/07/2014 15:19

We would like to advice that the inbound carrier that supply us with numbers are having issues will all inbound traffic.

Engineers are already investigating the issue and will advice ASAP

UPDATE -  Issue resolved at 12:46

UPDATE - Notice of Fault Sent to us today 17/07/2014 at 15:10

Period From: 2014-07-16 12:15:00 AEST

Period To: 2014-07-16 12:46:00 AEST

Location: National

Services Affected: Voice network

Detail:

 

We received alarm notifications from our monitoring system on the 16/7/2014 about packet loss occurring on the Voice network.

Our engineers instantly started investigating the issue, but due to the substantial packet loss occurring from the switches we were unable to remotely recycle the switches.

An engineer immediately went to the data center to recycle the power.

 

On further investigation it looks like a software bug had occurred on the IOS of the Cisco 3750 network cluster.

Now we are presently investigating the issue with Cisco our vendor.

An emergency maintenance service has been scheduled this evening to make some some modifications to the environment that should be non-customer impacting between 22H00 - 23H00.

We do apologise for the inconveniece that occurred yesterday

BBVOICE Service - Upgrading Switches (Resolved) Critical

Affecting Server - Business Service

  • 07/06/2014 22:00 - 18/06/2014 21:33
  • Last Updated 05/06/2014 18:02

Details are as follows:

 

-------------------------------------------------------------------

Type: Planned

Severity: Warning

Period From: 2014-06-07 22:00:00 AEST

Period To: 2014-06-07 23:00:00 AEST

Location: National

Services Affected: All Voice Services 

Detail:

During this outage window there will be disruptions to all voice services.

During the scheduled window it is estimated that the disruption to voice services will be for less than 15 minutes.

 

We apologise for any inconvenience this may cause.

-------------------------------------------------------------------

BBVOICE Service - Upgrading Switches (Resolved) Critical
  • 07/06/2014 22:00 - 18/06/2014 21:33
  • Last Updated 05/06/2014 18:02

Details are as follows:

 

-------------------------------------------------------------------

Type: Planned

Severity: Warning

Period From: 2014-06-07 22:00:00 AEST

Period To: 2014-06-07 23:00:00 AEST

Location: National

Services Affected: Inbound Voice Services and International Calls 

Detail:

During this outage window there will be disruptions to Inbound Voice Services and International Calls.

During the scheduled window it is estimated that the disruption to voice services will be for less than 15 minutes.

 

We apologise for any inconvenience this may cause.

-------------------------------------------------------------------

Some Inbound calls from Optus andVodafone (Resolved) High

Affecting Other - Carriers

  • 24/04/2014 14:50 - 05/06/2014 13:52
  • Last Updated 24/04/2014 16:12

Please be advised that there is currently an issue with inbound calls from Optus and Vodafone.

The issue was first reported at 2:50pm AEST today.

We will update further information once it comes to hand

UPDATE 24/04/2014 16:10 - We just received that issue has been resolved

bbvoice Network (Resolved) Critical

Affecting Server - Business Service

  • 19/03/2014 08:15 - 29/03/2014 09:18
  • Last Updated 19/03/2014 08:50

We are presently encountering an outage on our voice network and our engineers are working to rectify this issue.

Currently we do not have an ETA .

We will update you asap.

UPDATE 08:47: Voice Services are back to NORMAL.

We are waiting for a report from the carrier as to what happened and any remediation.

Incoming Calls to network from Certain Carrie (Resolved) Critical
  • 19/03/2014 08:15 - 29/03/2014 09:19
  • Last Updated 19/03/2014 08:48

We are having some issues with Incoming calls coming from from the carrier and engineers are already looking into in.

We will update ASAP.

UPDATE: 08:47 Incoming calls are now OK .

Vocus DC IP Transit (Resolved) Critical

Affecting Other - IP Transit

  • 20/02/2014 00:00 - 24/04/2014 15:25
  • Last Updated 19/02/2014 10:14

Advisory Notes

This is a notice that Vocus will be performing maintenance in the Vocus Data Centre in Sydney. We will move traffic from this facility to flow via an alternative path, we anticipate there to be NO interruptions to any service and this is a hazard notice only.

Issue with SSL Certificate (Resolved) Low

Affecting Other - www.wdpvoip.net.au

  • 28/01/2014 08:24 - 08/02/2014 08:29
  • Last Updated 28/01/2014 11:27

There is an issue with an SSL Cerificate for wdpvoip.net.au

This is being look at by one of the engineers and we should have a fix soon for it.

Appologies form Managment

Data Center DDos Attacked (Resolved) Critical
  • 02/01/2014 08:26 - 03/01/2014 10:19
  • Last Updated 02/01/2014 09:20

We just had reports from the Data center that were we host out Servers is Getting DDos Attacked and engineers are trying to stop the attack

We will update as soon as possible

UPDATE - 08:37 - Equinix DDos Attack mitigated and now stablized
                            Tuggerah still under DDos Attack and Engineers are trying to mitigate
                            Service should be stabized

UPDATE - 08:58 - Equinix DDos Attack Started again Engineers are trying to stabize again

UPDATE - 09:04   Equinix Stabilized and Tugerrah Stabilized Now
                            Services are back to normal.

UPDATE - 09:19   We will be monitoring for any more issues in the next 24hrs and if DDos Attack starts agian on a specific Datacenter, we will stop that specific service so that the other service will not be effected and at least run on one Datacenter.


Issue with BGB Routing from various Carriers (Resolved) Critical
  • 25/11/2013 14:45 - 25/11/2013 15:00
  • Last Updated 25/11/2013 15:04

We are expierincing some issues with BGB Routing from various carries.
This is getting looked at from Vocus Network and an ETA of 15 mins was given

UPDATE - Issue resolved at 15:00

Issue with incoming Phone numbers (Resolved) Critical

Affecting Server - Business Service

  • 01/11/2013 17:34
  • Last Updated 12/11/2013 21:41

We just have been informed from our carrier that they are experiencing some issues on the network and that calls to all incoming numbers are temporarily unavailable.

 

Outgoing calls to Australian Destinations should be fine via Evoice system but not available via bbvoice system

We are trying to divert calls from the evoice system to a different carrier for all international links.

International Incoming Phone numbers should not be affected.

17:42 Update: Engineers are still investigating the issue

17:44 Update: Engineers despatched to Dryburg ST DC

17:55 Update Issue Resolved all numbers are back and bbvoice network online

 We are now Waiting for full incident report

Outage at data center in Equinix (Resolved) Critical
  • 03/10/2013 17:17 - 01/11/2013 17:34
  • Last Updated 03/10/2013 18:58

Report of Outage from the data center

Advisory notes: as of 18:54

At 5:20pm SA detected a broadcast storm between the Equinix SY3 Data Centre and the Equinix SY1 Data Centre, storm control detected this and unfortunately shut down the ports to both our redundant feeds between the facilities. Technicians investigated the storm and stopped it ASAP, however the SY3 end needed to be cleared which took the most amount of time.

SA have identified this as being a huge failure in our network design and the way that broadcast storms are handled, as such we are now going to perform a full investigation into this outage and also implement a new way to handle such situations in the future. We apologise to any customers that were affected by this outage in our Sydney SY3 Data Centre

migrating of voice platform and VMWare infras (Resolved) Critical

Affecting Server - Business Service

  • 21/09/2013 23:00 - 01/10/2013 20:02
  • Last Updated 22/09/2013 15:21

Type: Planned

Severity: Critical

Period From: 2013-09-21 23:00:00 AEST

Period To: 2013-09-22 03:00:00 AEST

Location: MLB (bbvoice Newtwork)

Services Affected: All VoIP services - Migration

Detail:

WorldDialPoint will be migrating their voice platform and VMWare infrastructure to a new Data Centre on the 21st of September (Saturday) from 11:00PM (AEST) until 03:00AM, 22nd September (Sunday).

During this migration the VoIP service will be disrupted for up to 4 hours while the migration takes place.

You might be required to restart your CPE equipment onsite the following day if your services do not automatically register.

A completion notice will be published here once we have completed our internal testing to confirm the completion of the migration.

 

Please contact our helpdesk via support@wdpvoip.net.au if you experience any issues upon receiving the completion notice.

We apologise for any inconvenience this may cause and we appreciate your ongoing support.

Regards,

WorldDialPoint Network Operations

UPDATE: migration completed  as of 22/09/2013 Time 02:00:00

Upgrade to Porta System Sip Softswitch (bbvoi (Resolved) Critical

Affecting Server - Business Service

  • 27/07/2013 00:00 - 27/07/2013 06:00
  • Last Updated 27/07/2013 07:58

Details are as follows:

-------------------------------------------------------------------

Type: Planned

Severity: Info Only

Period From: 2013-07-27 00:00:00 AEST

Period To: 2013-07-27 06:00:00 AEST

Location: National

Services Affected: VoIP Service - Upgrade

Detail:

 

We will be performing a software upgrade in conjunction with our vendor on Saturday from 00:00 AM 27th of July 2013 (AEST) until 06:00AM.

 

In preparation for the software upgrade there will be a few intermittent restarts of the services outside of office hours on Friday morning the 26th July to pre-stage the upgrade.

The restarts will happen between 00:00AM - 04:00AM.

During the actual software upgrade, the VoIP service will be disrupted for up to 3 hours while the upgrade takes place.

You might be required to restart your CPE equipment onsite the following day if your services do not automatically register.

A completion will be published once we have completed our internal testing to confirm the completion of the upgrade.

Please contact our helpdesk via our support channel support at wdpvoip.net.au  if your service experiences any issues upon receiving the upgrade completion notice.

We apologise for any inconvenience this may cause and we appreciate your ongoing support.


UPDATE 07:55 am - Upgrade successful and all services are fully operational


Inbound Calls Issue (Resolved) Critical

Affecting Other - Telecorp

  • 25/07/2013 20:11 - 25/07/2013 21:11
  • Last Updated 25/07/2013 21:14

We just received reports from Telecorp that some inbound numbers are not working.

Telecorp has advised that they have some issues with some cards and they are replacing them.

We are expecting that this issue will be resloved within the next 1hr.

UPDATE - 21:10 We just received confirmation that inbound numbers that were not working has now been fixed

High Pings on Vocus Network (Resolved) High
  • 16/06/2013 09:20 - 16/06/2013 09:35
  • Last Updated 16/06/2013 09:45

We are expierincing high pings via Vocus Networks, engineers at Vocus Networks are already investing what is happing and trying to resolve it as soon as possible.

This issue has to do nothing about our service as it effecting multiple providers in Australia.

We are hoping that Vocus find a resulution as soon as possible.

UPDATE 09:37 am: Vocus reported that one of the switching that our links is was having some issues and it has now been resolved at 09:35 am

Issues with Evoice System (Resolved) Critical
  • 04/03/2013 09:05 - 18/03/2013 08:07
  • Last Updated 05/03/2013 10:01

We would like to notify that we are having some Technical issues with our system and Enginners are working at this moment on it.

At the moment Engineers are checking the File System and Databases for any file corruption.

 

More to follow

Update - 10:22 -  System is Partially up and Incoming and outgoing calls should be fine.

Module access is still desabled.

 

Update - 10:36 - Online Module is now back Online

Update - 10:50 - Previous Issue resurfaced again Engineers are again looking into the issue

Update - 10:50 - Issue found and enginners are paching up should be fine in the next 20 minutes

Tonight Engineers will be performing Urgent Maintinance and we will notify as soon as it is all good

Update - 05/03/2013 9:00 - Maintenance all went good last night. Tonight at around 21:00 engineers will be performing a swap of Service onto Equinix Service. This will result in appprox 2 minutes downtime and after that work will continue in Tuggerah Data Center to finalize the Maintenance.

Data Outage, in Equinix (Resolved) Critical
  • 14/02/2013 15:38 - 04/03/2013 09:07
  • Last Updated 14/02/2013 17:18

We would like to advise that our equipment in the 2 datacenter (Equinix and Tuggerah) is expierencing a data outage and engineers are working on it.

 

Update - 15:42 Data Center informed us that all services should be up and roning in the next 15mins

Update - 15:54 Just been informed that Data in the 2 data Centers has now been restored

Network Status Update (Resolved) Medium

Affecting System - BBVOICE Network

  • 08/02/2013 23:00 - 08/02/2013 23:59
  • Last Updated 10/02/2013 13:40

 

Details are as follows:

 

-------------------------------------------------------------------

Type: Planned

Severity: Info Only

Period From: 2013-02-08 23:00:00 AEST

Period To: 2013-02-08 23:59:00 AEST

Location: Melbourne Network ONLY

Services Affected: VoIP

Detail:

Engineers will be installing new clustered Cisco switches on the voice network to facilitate increased resiliency.

The upgrade is expected to take VoIP services offline on Friday the 8th February 2013 at 23:00 AEDST for approximately 15 - 30 minutes.

 

Voice Service will NOT be available for APPROXIMATELY 15-30 mins

 

Routing Issues with some major Providers (Resolved) Critical
  • 03/02/2013 10:13 - 03/02/2013 12:35
  • Last Updated 03/02/2013 12:53

We would likre to inform our clients, that we are seeing some routing issues with majow Carriers.

Because of this some ISP's cannot route their traffic to our Servers.

 

This is being investigated by our carriers as we speak and hoping a fix will be done soon.

Servers are fully operational but not recachable from some carriers at the moment.

 

UPDATE - 12:35 Carrier has informed us that the Issue fully resolved.

We are waiting for a full report and once we have it, we will publish here

WorldDialPoint Network Update and More featur (Resolved) Medium

Affecting Server - Business Service

  • 08/12/2012 00:00 - 08/12/2012 00:30
  • Last Updated 25/12/2012 02:17

 

WorldDialPoint Network Maintenance on the bbvoice network

Details are as follows:

-------------------------------------------------------------------

Type: Planned

Severity: Info Only

Period From: 2012-12-08 00:00:00 AEST

Period To:      2012-12-08 00:30:00 AEST

Location: National

Services Affected: Voice and Data

Detail:

Engineers will be installing two new dual powered switches on the network to facilitate increased resiliency of Voice and Data services.

The upgrade is expected to take services offline on Saturday the 8th December 2012 at 00:00 AEDST for approximately 15 - 30 minutes.

Just a Precaution - If your services do not automatically connect back to the network please power cycle your CPE.

Thank you for you co-operation during this upgrade.

-------------------------------------------------------------------

With the latest upgrade approx. 3 weeks ago some new features have been added and activated.

These are

1.      Customized CLI for outgoing calls (proof of ownership is required)

Hunt Groups (if you have multiple extensions)

 

High Availability Firewall Installation (Resolved) High

Affecting Server - Business Service

  • 30/11/2012 11:30 - 30/11/2012 11:59
  • Last Updated 30/11/2012 22:31

Details are as follows:

-------------------------------------------------------------------

Type: Planned (Resheduled)

Severity: Hazard

Period From: 2012-11-30 23:30:00 AEST

Period To: 2012-11-30 23:59:59 AEST

Location: Nationwide

Services Affected: All Voice Services

Detail:

The following servers will be migrated to a pair of High Available Firewalls on Thursday 29/11/2012 @ 23h00:

Porta Voice Gateway

AV/Anti-Spam mail server

Primary Name server

Radius Server

Voicemail/IVR server

The downtime is not expected to be longer than 10 minutes during the maintenance window (Thursday 29/11/2012 23h00 - Thursday 29/11/2012 23h59).

A number of post tests will be applied to confirm the changeover.

Update: 29/11/2012 23:09 - Started Maintenance


UPDATE 29/11/2012 23:28 - Service UP had to roll back configuration Firewall HA Failed. More to come


UPDATE 29/11/2012 23:53 - The planned upgrade has been suspended due to an unforeseen issue arising on the configuration. Once this is resolved we will schedule a new time for the upgrade.

 

UPDATE 30/11/2012 12:05 - Reschedule for tonight at 23:30 AEST

 

UPDATE 30/11/2012 23:13Started Maintenance

 

UPDATE 30/11/2012 23:21Maintenance finished

Issue with Caller ID on Forwarded calls / SIP (Resolved) Low

Affecting Server - Business Service

  • 12/11/2012 08:21 - 17/11/2012 02:30
  • Last Updated 28/11/2012 16:25

Issue with Caller ID on Forwarded calls / SIP Trunks

Because of the Upgarde on the weekend and new features added to the Forwarding rules CLI forwarding is not showing the Callee number on an incoming call.

This issue is being investigated by the software manufacturer and a fix will be applied hopefully within the next 24 hrs


ETA - A fix has been scheduled for  16/11/2012 23:00


NOTICE


WorldDialPoint Network status Update BBVOICE ONLY (203.176.185.10)

Details are as follows:

-------------------------------------------------------------------

Type: Planned

Severity: Warning

Period From: 2012-11-16 23:00:00 AEST

Period To: 2012-11-17 02:30:00 AEST

Location: National bbvoice Network

Services Affected: VoIP Services

Detail:

Our software vendor has requested that we apply a critical patch to our voice network on Friday evening in order to fix a CLI (display) and a call forwarding issue that is occurring on some CPE equipment.

The main outage window planned is 120 minutes with an additional 90 minutes for testing.

Just a Precaution - Please reboot your CPE ONLY if your voice services do not automatically connect back to the network.

Further Notification:

We will Update our Notification Pages here in real time https://wdpvoip.net.au/control/networkissues.php

We will also Update the Twitter Feeds here http://twitter.com/WorldDialPoint


UPDATE 23:20Maintenance starting soon. Service will be unavailable for the next 120 mins

UPDATE 02:35 - Maintenance Finished and Testing Started

UPDATE 03:34 - Testing Finished

Network Status Update (Evoice) (Resolved) Low
  • 15/11/2012 16:25 - 17/11/2012 02:30
  • Last Updated 17/11/2012 01:40

Network Status Update (Evoice)

Start Date - 16/11/2012 23:00

Finish Date - 17/11/2012 02:30

Notice

International Calls will not be available during the time

Incoming and Outgoing Faxes is not possible during this time


Incoming Calls is not Possible

Our Software vendor will be preforming some maintenance on one of the core notworks in Melbourne.

Further Information

All other service will NOT be affected

Outgoing Calls to Mobile / Landlines and Emergency Services are NOT Effected



We will Update our Notification Pages here in real time https://wdpvoip.net.au/control/networkissues.php

We will also Update the Twitter Feeds here http://twitter.com/WorldDialPoint

Regards,

WorldDialPoint

UPDATE 23:20 - Maintenance starting soon.

UPDATE 02:35 - Maintenance Finished all up and running again

Issue with Caller ID on Forwarded calls / SIP (Resolved) Low
  • 12/11/2012 08:21 - 15/11/2012 15:59
  • Last Updated 12/11/2012 15:55

 

Issue with Caller ID on Forwarded calls / SIP Trunks

Because of the Upgarde on the weekend and new features added to the Forwarding rules CLI forwarding is not showing the Callee number on an incoming call.

This issue is being investigated by the software manufacturer and a fix will be applied hopefully within the next 24 hrs

 

ETA - A fix has been scheduled for midnight today 12/11/2012 23:59

 

VoIP Network Upgrade - Phase 3 (Scheduled) (Resolved) Critical

Affecting Server - Business Service

  • 09/11/2012 23:00 - 10/11/2012 09:00
  • Last Updated 10/11/2012 10:24

 

Type: Planned

Severity: Info Only

Period From: 2012-11-09 23:00:00 AEST

Period To: 2012-11-10 07:00:00 AEST

Location: National

Services Affected: VoIP Network Upgrade - Phase 3

Detail:

A major upgrade to the VoIP network is planned for Friday the 9th November 2012 starting at 23:00 and ending at 07:00 on Saturday 10th November 2012.

 

This is the last major upgrade on the Voice network for 2012.

In summary we have installed a new voice switch which is currently running in parallel with the production VoIP network.

 

During the upgrade process we will be moving all VoIP services over to the new voice switch. All incoming and outgoing calls will not be possible during this time.

 

Further Notification:

We will post through a further update once testing has been completed via our Network status Page https://wdpvoip.net.au/control/networkissues.php and via Twitter https://twitter.com/WorldDialPoint.

 

As this is a major upgrade you will be expected to restart your equipment if it does not automatically register

 

10/11/2012 07:00 Upgrade looking good and all system should be online within the next 1hr

LAST UPDATE - 10/11/2012 10:22am: Testing Finished and all is good

 

VoIP Network Upgrade - Phase 3 (Scheduled) (Resolved) Critical
  • 09/11/2012 23:00 - 10/11/2012 09:00
  • Last Updated 10/11/2012 10:24

 

Type: Planned

Severity: Info Only

Period From: 2012-11-09 23:00:00 AEST

Period To: 2012-11-10 07:00:00 AEST

Location: National

Services Affected: VoIP Network Upgrade - Phase 3

Detail:

A major upgrade to the VoIP network is planned for Friday the 9th November 2012 starting at 23:00 and ending at 07:00 on Saturday 10th November 2012.

 

This is the last major upgrade on the Voice network for 2012.

In summary we have installed a new voice switch which is currently running in parallel with the production VoIP network.

 

During the upgrade process we will be moving all VoIP services over to the new voice switch. All incoming and outgoing calls will not be possible during this time.

 

Further Notification:

We will post through a further update once testing has been completed via our Network status Page https://wdpvoip.net.au/control/networkissues.php and via Twitter https://twitter.com/WorldDialPoint.

 

As this is a major upgrade you will be expected to restart your equipment if it does not automatically register

 

10/11/2012 07:00 Upgrade looking good and all system should be online within the next 1hr

LAST UPDATE - 10/11/2012 10:22am: Testing Finished and all is good

 

BBvoice Network (Resolved) Critical

Affecting Server - Business Service

  • 19/10/2012 08:55 - 19/10/2012 09:53
  • Last Updated 20/10/2012 09:04

Customers are experiencing no connection to bbvoice servers and cannot receive and make calls out.

We are at the moment investigating the issue

We will update as the info comes in


Update 19/10/2012 09:53 - Services restored



Update 19/10/2012 10:44 - Experienced a power outage at our data centre where some of our key equipment is housed. All services have been restored. There is dual power and equipment for services but this power issue took out more services than expected.

 

An investigation will be done and we will prepare an incident report which will be posted with details on what happened.

Our sincere apologies for the outage that has occured this morning

 

Incoming calls Failing (Resolved) Critical
  • 19/10/2012 09:00 - 19/10/2012 09:53
  • Last Updated 20/10/2012 09:04

We are experiencing some issues with incoming calls nationwide from our carrier.

Engineers are looking into the issue.

Out going calls should not be effect at this time.

We will update the info as soon as it come in.


Update 19/10/2012 09:53 - Services restored


Update 19/10/2012 10:44 - Experienced a power outage at our data centre where some of our key equipment is housed. All services have been restored. There is dual power and equipment for services but this power issue took out more services than expected.

 

An investigation will be done and we will prepare an incident report which will be posted with details on what happened.

Our sincere apologies for the outage that has occured this morning

Change of IP Address for WorldDialPoint SIP/V (Resolved) Critical
  • 21/09/2012 22:00 - 24/09/2012 23:59
  • Last Updated 19/10/2012 09:11

 

On the Monday 24/09/2012 23:00 Engineers will disconect the Old IP address Range so if you still did not change to sip.evoice.net.au, from Tuesday morning your service will stop working unless you chage it.

Please note that the Change of IP address will happen on 21/09/2012 at 22:00.

In the meantime we have managed to increase the links to our network and this will resolve any bandwidth congestion issue that we experienced in peak times during last week.

Start - 21/09/2012 22:00

Finish - 21/09/2012 23:59

During this time outgoing and incoming calls will not be possible on the evoice system. This should return to normal after 2 hrs.

Please note that due to Caching of DNS with your internet provider, there could be some delay from when our New Ip address range be picked up by your ISP but should return to normal with 24-48hrs

 

 

 

Change of IP Address for WorldDialPoint SIP/VoIP Proxy

Urgent Attention Emergency work required in the next 96hours

As you may be aware we have been working for the last few months in upgrading the network infrastructure to add more resilience to our offerings while this has happened already we have encountered an issue with growth and some of you may have experienced some issues because of congestion in the last 4 days on our existing links from our supplier. We do apologise for this late notice, but this change is needed so that we can supply you with a better service.

As a part of this, we need to change the IP Address that the SIP Voice Proxy is on.

If your equipment (hardware / software to make calls) connects to our system via the IP Address 202.130.35.67, you will need to change the Proxy address to sip.evoice.net.au

If your equipment also has a firewall or router which is setup to send traffic/accept traffic to/from the old IP Address of our Proxy, you will also need to change the details on this equipment/software as well. More information will be posted

WorldDialPoint will be changing the IP Address behind sip.evoice.net.au from 202.130.35.67

Further to this, as we will be implementing new load balancing configurations, it is advised that you use SIP Proxy URL of sip.evoice.net.au, otherwise the load balancing may not work correctly, if pointing to the IP.

 

To help with this change over we also have reduced the DNS time to 4hrs, so if you are connecting to sip.evoice.net.au the DNS to the new Ip Address will not take long. If for any reason your equipment will not connect automatically with 24hrs, please switch of your equipment (including router / modem and voice equipment) and restart again. This should fix the connection issue.

If you need help changing your IP address please let us know and we can help you make the change.

We will also post Notification on our Announcment page / Network Page and Twitter

Should you have any further queries in relation to this matter, please email us at support@wdpvoip.net.au any time (target response time is 4hrs)

 

Evoice System (Resolved) Critical
  • 26/09/2012 18:00 - 26/09/2012 19:40
  • Last Updated 26/09/2012 19:52

It has come to our Attention that Vocus Is Having problems with their network. It looks that vocus in SYD Equinix is not responding.

 

We are waiting for their links to come up

 

Some customers that are connecting with certain ISP may find that they cannot connect to our service.

UPDATE - 26/9/2012 - 19:40 - Links to Vocus are up and running.

Outgoing Faxes Failture (Resolved) Critical
  • 21/09/2012 21:45 - 24/09/2012 22:59
  • Last Updated 26/09/2012 14:49

Outgoing Faxes Failture

 

This issue has been investigated and enginneers confirmed that outgoing fax server is still on Old IP address range It seems that something did not work as expected due to DNS catching on the Fax Server.

This issue should be resolved by Monday 24/09/2012 23:00 when the engineers will disconnect the Old IP address range.

ISSUE RESOLVED

Incoming calls Failing (Resolved) Critical
  • 23/09/2012 10:57 - 23/09/2012 17:36
  • Last Updated 23/09/2012 17:37

 

We would like to advise customers that there is an issue with incoming calls going directly to voicemail instead of the phones ringing.

Engineers are investigating the issue that is causing and we are expecting an ETA within the next 3 hours

Apologies for this inconvenience caused

 

Issue Resolved

 

Network Status Update (Resolved) Low

Affecting Server - Business Service

  • 06/09/2012 23:00 - 06/09/2012 23:59
  • Last Updated 10/09/2012 21:03

 

Type: Planned

Severity: Low

Period From: 2012-09-06 23:00:00 AEST

Period To: 2012-09-06 23:59:59 AEST

Location: Nationwide

Services Affected: All Voice Services

Detail:

The following servers will be migrated to a pair of High Available

Firewalls on Thursday 06/09/2012 @ 23h00

This will effect -

Porta Voice Gateway

Primary Nameserver

Radius Server

Voicemail

 

The downtime is not expected to be longer than 10 minutes during the

maintenance window (Thursday 06/09/2012 23h00 - Thursday 06/09/2012

23h59).

 

Residential and Business Scheduled Maintenanc (Resolved) Critical

Affecting Other - Evoice and BBvoice System

  • 25/08/2012 22:00 - 26/08/2012 06:00
  • Last Updated 26/08/2012 12:29

 

Update - 26/08/2012 at 8:13


The major upgrade has been completed.

Our Voice Vendor has been working in conjunction with our engineers since 10pm last night.

The old Voice network was officially shut down at 6am today to bring online the new servers.

All new servers are online and in production.

We will be conducting our own tests for the next several hours.

 

 

---------------------------------------------------------------------

 

Upcoming Major VoIP Software Upgrade - UPDATE – 25/08/2012 at 12:43

We have been in discussions with the software vendor and have managed to arrange the continuation of the original upgrade.

The restart of the upgrade will start tonight at 22:00 AEST (UTC +10)

We will still allow for the same time window to complete.

Start Date: Saturday 25th August 2012, 22:00 AEST (UTC +10)

End Date: Sunday 26th August 2012, 06:00 AEST (UTC +10)

Once the upgrade has been completed we will notify you accordingly so you can perform your own tests.

Engineers will be on standby to assist with any major issues that you might have in relation to the upgrade.

If you have any problems connecting back please send us a support ticket ASAP to support@wdpvoip.net.au

Announcements and Network issues will be posted on Twitter and our website https://wdpvoip.net.au/control/

 

------------------------------------------------------------------------

 

WorldDialPoint have been planning three major upgrades on the VoIP network over the past several months. We have now scheduled a go live date for the Second upgrade.

This is purely a courtesy email advising that we will be performing the upgrade on Friday the 24th August 2012 between the hours of 22H00 and should be completed by the 25th August at 06H00 on Saturday morning.

The official outage should not last longer than 1-2 hours but this larger time window will allow us time to complete the required testing.

As we are migrating the core systems to new hardware your VoIP equipment/handsets might need to be restarted the following day if any of your equipment does not automatically register to the BBVoice Network.

Evoice Network during this time will not be able to receive and Make calls

Bbvoice Network during this time will not be able to receive and Make calls

If you are having any VoIP difficulties the following day registering to the bbvoice network please contact us to assist in resolving the matter via support@wdpvoip.net.au and we will look into the problem immediately. Our engineers will be on standby to assist with any major issues that you might have in relation to the upgrade.

Please also refer to our Twitter feed https://twitter.com/WorldDialPoint where we will post any applicable updates subsequent to the upgrade

 

Vocus Outage on Certain Inbound numbers (Resolved) Critical

Affecting Other - Vocus Network

  • 10/05/2012 13:05 - 10/05/2012 14:19
  • Last Updated 10/05/2012 14:21

 

Please be advised that Vocus is currently experiencing an unscheduled outage with their Voice network.

Due to this Vocus numbers would be having issues receiving incoming calls. All Vocus Engineers are working on it and the updates will be passed on as we get it.

 

Update 10/05/2012 @ 13:36 - We are now seeing incoming calls via Vocus Network as it looks like issue is fixed but at this moment Vocus still have never sent comfirmation.

We will update as soon as possible

Update 10/05/2012 @ 14:19 - We just had comfirmation from Vocus that issue with certain inbound Australian Phone numbers is now RESLOVED

Equinix Service (Main Servers some Issues) (Resolved) Critical
  • 24/04/2012 17:59 - 00/00/0000
  • Last Updated 08/05/2012 08:07

 

Severity Critical
Type Unplanned
Period From 2012-04-24  15:44 AEST
Period To 2012-05-08 3:00 AEST

Location National
Services Affected Equinix Services
Details :Sydney

Equinix service at the moment is Offline and Engineers are looking into it. The service should no be effected as failover to the other data center kicked in as expected and fully active

We had a 2 short outages till the buckup servers kicked in Tuggerah.

Update 24/04/2012 18:00 Enginneers at this moment are checking for any hardware failture and some other things.

Update 24/04/2012 20:50 Engineers have found that 1 of the disks on one of the servers needs replacment. The Equinix Service is now OFFLINE as some repairs and ingerity checking will be done. Once all this is done Equinix service should be back online.

At this moment we do not have an ETA

Update - 26/04/2012 16:15 After all checks that have be done, it was found that one of the Servers Raid Cards includind one of the disks for some unkown reason failed. Due to this issue we have order a new server that will repalce one of the old ones and now expecting delivery.

More to follow

Update 4/05/2012 19:55 After a long awaited week, Equinix service is back online and final testing has commenced. This will take approx an other 24 hrs, to bring back a fully Geographical Reduntant System.

More to follow

Update 5/5/2012 21:30 Please note that Equinix will go live 7/5/2012 at 21:00. Most likely there will be a downtime of approx. 3mins as Equinix Service will need to catchup with Tuggerah Service

More to follow

Update 8/5/2012 8:00 Equinix is now back on Primary and system is running at 100%

Module Access Limited (Resolved) Low
  • 20/03/2012 22:14 - 00/00/0000
  • Last Updated 04/04/2012 21:07

Please be advised that access to Online module is linited at the moment, we are worlking with the software vender to diagnose and fix.

Please note that calls are NOT effected.

We expect the issue to be resolved soon and will advise with any progress.

Thankyou

WorldDialPoint Team

Nation Wide Telstra Outage effecting Intercon (Resolved) Critical

Affecting Other - BGB Routing with Telstra

  • 23/02/2012 14:15 - 23/02/2012 16:40
  • Last Updated 23/02/2012 16:39

 

We have been advised of a major outage affecting 3rd party broadband services in NSW, VIC, and QLD.

 

ISP’s affected include Telstra and potentially traffic traversing Telstra networks

 

This is NOT an Issue with our Service

 

Update : 23/02/2012 16:40

internet outage seen earlier this afternoon - we have been advised that the issue has been resolved.

Nation wide Telstra Outage effecting ISP's (Resolved) Critical

Affecting Other - BGB Routing with Telstra

  • 23/02/2012 14:15 - 23/02/2012 16:40
  • Last Updated 23/02/2012 16:39

 

 

We have been advised of a major outage affecting 3rd party broadband services in NSW, VIC, and QLD.

 

ISP’s affected include Telstra and potentially traffic traversing Telstra networks

 

 

This is NOT an Issue with our Service

 

Update : 23/02/2012 16:40

internet outage seen earlier this afternoon - we have been advised that the issue has been resolved.

International Calls Fail (Resolved) Critical

Affecting Server - Business Service

  • 20/02/2012 23:18 - 21/02/2012 10:50
  • Last Updated 21/02/2012 10:52

 

We are getting reports that International destinations are all getting an engaded tone.

We also had a report that there is a problem in Global Switch Data Center in Sydney. We believe that this is the place where interconnection happens to all International destinations.

We are investigating the issue with 2 different carriers and we have sent a please explain what is happening. We are waiting for any outcome.

We would like to inform everyone that this issue is not related with our Network and calls should be fine within Australia.

 

Update - 10:50 am Issue Resolved and International Links are back ONLINE

International Calls Fail (Resolved) Critical
  • 20/02/2012 23:10 - 21/02/2012 10:50
  • Last Updated 21/02/2012 10:51

We are getting reports that International destinations are all getting an engaded tone.

We also had a report that there is a problem in Global Switch Data Center in Sydney. We believe that this is the place where interconnection happens to all International destinations.

We are investigating the issue with 2 different carriers and we have sent a please explain what is happening. We are waiting for any outcome.

We would like to inform everyone that this issue is not related with our Network and calls should be fine within Australia.

 

Update - 10:50 am Issue Resolved and International Links are back ONLINE

Problem with Login (SSL) (Resolved) High
  • 19/02/2012 20:37 - 20/02/2012 09:18
  • Last Updated 20/02/2012 09:18

We are expierencing some problems with the SSH certificate for Evoice Billing System. This is being investigated by IT.

Calls are NOT effected

We appologise for any inconvenience caused

 

 

Packet Loss (Resolved) Critical
  • 19/02/2012 21:00 - 19/02/2012 21:16
  • Last Updated 19/02/2012 21:17

We are now expierencing packet loss of 5% on our intire network. This issue is with Equinix / SA Network and we are working with them to resolve the problem ASAP.

 

Issue is now resloved

Packet Loss (Resolved) Critical
  • 10/02/2012 16:20 - 10/02/2012 17:10
  • Last Updated 10/02/2012 18:05

We are expierencing 5% packet loss on 2 different links. Engineers are already investigating the issue

 

Update - Issue is now resolved