Comodo Certificate Authority (Resolved)
  • Priority - High
  • Affecting Other - SSL certificates
  • Scheduled Maintenance

    Comodo Certificate Authority


    Title: Comodo CA Scheduled Maintenance Saturday 13th, October 2018 22:59 UTC

    Planned Start: 10/13/2018 10:59pm (+0000)
    Expected End: 10/13/2018 11:14pm (+0000)

    Affected Infrastructure
    Components: ComodoCA, Website
    Locations: EmailQueue, Primary Data Center, Signing

    Details: The live web servers will be going down at
    22:59 UTC Saturday 13th October 2018
    23:59 BST Saturday 13th October 2018
    18:59 EDT Saturday 13th October 2018
    17:59 CDT Saturday 13th October 2018
    16:59 MDT Saturday 13th October 2018
    15:59 PDT Saturday 13th October 2018
    for an outage of approximately 15 minutes in order to add additional functionality to the system.

    October 13, 2018 10:59PM UTC
    [Update] Scheduled maintenance is starting.

    October 13, 2018 11:14PM UTC
    [Update] Scheduled maintenance is complete.

     

     

  • Date - 13/10/2018 22:59 - 13/10/2018 23:14
  • Last Updated - 14/10/2018 07:22
LHC Data Centre Core Router upgrade (Resolved)
  • Priority - Medium
  • Affecting Other - Network Router Upgrades
  • Planned maintenance Friday July 13th 21.00hrs to Saturday July 14th 01.00hrs UTC

    LHC Data Centre Core Router upgrade will be conducted. Where each Router, in turn will be gracefully removed from the topology and then returned once maintenance has been completed.

    Issues should not be experienced during this time. But, it may be possible to experience some internet paths becoming slower or unavailable for a moment or two.

  • Date - 13/07/2018 21:00 - 14/07/2018 01:00
  • Last Updated - 14/07/2018 06:47
Datacentre Planned maintenance (Resolved)
  • Priority - Medium
  • Affecting Other - Network Router Upgrades
  • Enfield Data Centre Core Router upgrade will be conducted. Where each Router, in turn will be gracefully removed from the topology and then returned once maintenance has been completed.

    Issues should not be experienced during this time. But, it may be possible to experience some internet paths becoming slower or unavailable for a moment or two.

  • Date - 30/06/2018 23:00 - 01/07/2018 03:00
  • Last Updated - 01/07/2018 09:33
Packet Loss (Resolved)
  • Priority - Critical
  • Affecting Server - echo
  • We experienced severe packet loss between Spain and the servers in London starting 23:30 on the Tue 6 Mar 2018

    Tests by Coreix networking engineers determined that this was isolated to connections from Spain, and the servers were accessible from the rest of the world without problems.

    The network seems to have been restored to 0% packet loss this morning 08:30 Wed Mar 7 2018 but we shall continue to monitor the incident for a few hours to see if the network is stable.

    Our best guess is the packet loss was caused by maintenance work on the network overnight.


    ****UPDATE****

    I haven't seen any packet loss all day, so I am marking this incident as Resolved. Any new events overnight will be treated as separate incidents.

  • Date - 06/03/2018 23:30 - 07/03/2018 19:38
  • Last Updated - 07/03/2018 18:38
Network Service difficulties (Resolved)
  • Priority - High
  • Affecting Server - echo
  • Network Engineers are currently working to mitigate this incident & hope to have it resolved shortly.

    Monitoring - The incident has been controlled & is currently being monitored to ensure that it has been resolved. Once resolved, a report will be generated.

    UPDATE - All systems are now operational

    The service difficulties this morning related to an unusually large DDoS attack which required us to mitigate the attack at our upstream providers. Due to the size of the attack it took just over 10 minutes to achieve full mitigation across all uplinks.

  • Date - 23/02/2018 09:51 - 23/02/2018 10:13
  • Last Updated - 23/02/2018 10:15
Off server backups service (Resolved)
  • Priority - High
  • Affecting Server - echo
  • We are currently experiencing difficulties with our off-server backup service.

    The service provider has, so far, been able to supply us with a module that is compatible with our new kernel that protects us against the Meltdown and Spectre exploits.

    We are working closely with the backup service provider to resolve this issue as soon as possible.

    In the interim, we should like to remind our clients that we also take daily backups to a separate hard drive on the server which has not been affected by this incident, and is working normally.

    We appreciate your patience and understanding whilst we work to resolve this issue.


    **UPDATE Tue Jan 9 2018 **

    The backup software supplier has provided us with a working solution, and we are monitoring the backups to ensure they complete successfully over several days.


    **UPDATE Tue Jan 16 2018 **

    After the kernel update today, we successfully rebuilt and activated the backup module and ran a full backup to test the system, which completed without issues.

    If we get successful backup reports for the rest of this week, we shall consider this incident closed, and open a new one if we have any new problems.

     
    ***UPDATE Tue Jan 16 2018 23:15 UTC***

    We received a security alert from the kernel developers instructing us to revert back to the 2.6.32-896.16.1.lve1.4.49.el6.x86_64 kernel as there was a possibility that there was an issue in the  2.6.32-896.16.1.lve1.4.50.el6.x86_64 kernel which makes it vulnerable to Spectre variant 1 attack.

    The system has been rebooted into the 2.6.32-896.16.1.lve1.4.49.el6.x86_64 kernel and the backups tested again to completion without issues.


    ***UPDATE Thu Jan 18 2018 23:09 UTC***

    Kernel was updated to version 2.6.32-896.16.1.lve1.4.51.el6.x86_64


    This kernel is a rework of the 2.6.32-896.16.1.lve1.4.50.el6.x86_64 kernel that we had to withdraw as it re-introduced some of the problems that which made it vulnerable to Spectre variant 1 attack.

    The system has been booted into the 2.6.32-896.16.1.lve1.4.51.el6.x86_64 kernel, and we are monitoring the backups to ensure they will complete successfully.


    All the backups to date have completes sucessfully, so this incident is being marked as resolved.


  • Date - 08/01/2018 17:49 - 21/01/2018 00:00
  • Last Updated - 20/01/2018 14:03
Core5 Abnormal behavior (Resolved)
  • Priority - Medium
  • Affecting Other - Incident Report for Coreix Networking
  • Investigating
    Coreix are observing high cpu and abnormal behavior on one of the Core router (Core5.enf). This may be causing some high latency for some of our customer being routed via Core5. Coreix engineers are investigating the issue and will update shortly.

    Update 15:21 UTC
    Monitoring -
    A fix has been implemented and we are monitoring the results.

    Update 15:27 UTC
    The Core5 router is now operating within normal parameters. All traffic and BGP sessions are also showing as normal. Coreix will continue to monitor Core5, scrutinizing the information gathered thus far. Anomalies will be investigated & further information provided. In the interim, should you experience any other connectivity problems, please do not hesitate to contact support

    Update 17.16 UTC
    This incident is now resolved.

  • Date - 09/10/2017 15:05 - 09/10/2017 17:16
  • Last Updated - 09/10/2017 17:25
Degraded routing performance (Resolved)
  • Priority - High
  • Affecting Other - Transit providers
  • Degraded routing performance

     
    Incident Report for Coreix
     
    New Incident Status: Resolved
    Some issues were experianced with a few of the Coreix transit providers between 09:45 and 10:05 BST (GMT+1) which caused degraded routing between some non UK and Europe destinations, mainly Asia and US. The issue has been identified and resolved. The service level has been normal since 10:05 BST, but if you're experiencing high latency or packet loss please open a ticket to support@rpvw.com

  • Date - 30/09/2017 08:45 - 30/09/2017 09:05
  • Last Updated - 02/10/2017 06:23
Necessary Network Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Network
  • Our data-centre has indicated that they will be performing necessary preventative maintenance to be conducted Tuesday 26th September 00.30hrs to 01.30hrs British Summer Time, (GMT +1). This will affect customers connected via EDGE 1. Possibly resulting in a minor loss in connectivity for a few minutes. It is possible, that customers connected via alternate EDGES may experience the same.

    **** UPDATE  Tuesday 26 September 2017 01:16 BST ****

    The scheduled maintenance has been completed successfully. According to our logs and monitoring, we didn't experience any downtime during the maintenance activity, but please feel free to contact our support if you do experience any issue.

  • Date - 25/09/2017 23:30 - 26/09/2017 10:50
  • Last Updated - 26/09/2017 10:52
SquirrelMail (Resolved)
  • Priority - Critical
  • Affecting System - Web-Mail interface
  • Due to reports of a potential remote code execution vulnerability exploit in the SquirrelMail web-mail service, we have withdrawn this option from our web-mail services until we can ascertain if the service is vulnerable on our servers.

    As soon as we have received confirmation that the service is safe to run, or we get a patch or updated version, we shall reinstate the service.

    Our apologies to users of the SquirrelMail interface. Please use one of the other web-mail services until such time as we can reintroduce the SquirrelMail service.

    Users should be aware that we have seen no indications of any exploits, nor even exploit attempts against this service, but we are committed to ensuring the best possible security for our customers, which is why we have temporarily withdrawn the service.

    *** Update Tue Apr 25 07:42:07 2017 ***
    The cPanel security team has opened a case for this issue - CPANEL-12702.

    *** Update Tue Apr 25 16:45:50 2017 ***
    It looks like the instance of SquirrelMail offered through cPanel isn't affected by this vulnerability. However, as mentioned, internal case CPANEL-12702 is open to assess that report and ensure the instance of SquirrelMail offered through cPanel is updated to patch against this vulnerability.

    We shall update this Issue as new information becomes available.


    *** Update Friday 05 May 2017 ***
    cPanel are assuring us that the version of SquirrelMail included in our cPanel WebMail interface is not vulnerable to this exploit as we have the connection to sendmail disabled in the configuration.

    As a result, we have today re-enabled SquirrelMail for our users.

  • Date - 24/04/2017 21:53 - 05/05/2017 12:00
  • Last Updated - 05/05/2017 10:01
Maintenance (Resolved)
  • Priority - High
  • Affecting Server - echo
  • Our Datacentre has advised us of upcoming maintenance to our backup network. You may be aware that we have experienced sporadic issues with some of the switches that are deployed in this element of the network. After consultation with the manufacturer these units are being retired from our network and replaced with alternative units supplied by the manufacturer. We will be pre-mounting the new switches in each cabinet so the downtime is expected to minimal per port as cables are rerouted.

    Date: 10/04/2017
    Maintenance Window:
    23:00 10/04/2017 to 01.30 11/04/2017
    Expected outage per port: < 5 minutes.
     

    Thank you for your cooperation and apologies for any inconvenience this might cause you.

  • Date - 10/04/2017 23:00 - 11/04/2017 01:30
  • Last Updated - 11/04/2017 08:12
Planned Maintenance (Resolved)
  • Priority - Low
  • Affecting Other - Installation of multiplexers
  • Planned maintenance to be conducted to upgrade Coreix’s Metro infrastructure. The maintenance will not be service impacting as alternate routes will be in effect. However, there may be some instances of small blips due to STP re-calculations.
    
    
    Location = London (6/7 Harbour Exchange Sq), LHC and Telehouse East
    
    Start: 2016-10-20 01:00:00 GMT
    End: 2016-10-20 06:00:00 GMT
    
    Duration: 5 hours
    
    Impact: 120
    Reason: Installation of multiplexers between Telehouse East, Harbour Exchange and LHC.
    
    We apologise in advance for any inconvenience this might cause.

  • Date - 20/10/2016 01:00 - 20/10/2016 06:00
  • Last Updated - 20/10/2016 08:15
Planned Maintenance (Resolved)
  • Priority - Low
  • Affecting Other - Infrastructure Backups
  • Planned maintenance to be conducted to upgrade Coreix’s Metro infrastructure. The maintenance will not be service impacting as alternate routes will be in effect. However, there may be some instances of small blips due to STP re-calculations.
    
    Location = London (6/7 Harbour Exchange Sq), LHC and Telehouse East
    
    First Backup Start: 2016-10-21 01:00:00 GMT
    First Backup End: 2016-10-21 06:00:00 GMT
    
    Duration: 5 hours
    
    Impact: 120
    Reason: Installation of multiplexers between Telehouse East, Harbour Exchange and LHC.
    
    We apologise in advance for any inconvenience this might cause.

  • Date - 21/10/2016 01:00 - 21/10/2016 07:53
  • Last Updated - 12/10/2016 15:57
Extra Backup 1TB drive (Resolved)
  • Priority - Critical
  • Affecting Server - echo
  • We are going to take the server off-line for about 15 minutes to install an extra 1TB hard drive to use for multiple backups, and we should like to  apologize in advance for any inconvenience this may cause.


    If you need any further information, please do not hesitate to contact us.


    Please note - times are UTC


    Maintenance completed 2016-06-20 12:32:07 CET

  • Date - 19/06/2016 10:00 - 19/06/2016 10:15
  • Last Updated - 20/06/2016 10:32

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Server Name HTTP FTP POP3 IMAP SMTP Server Load Uptime
echo