[Linux-aus] Recent Linux Australia website issues - VPAC Closure
Linux Australia Secretary
secretary at linux.org.au
Thu Apr 28 23:13:48 AEST 2016
Dear Linux Australia Colleagues,
In line with our values of openness and transparency, we'd like to
recent developments regarding the recent VPAC/V3 Alliance closure [^1]. The
closure has had some impacts to Linux Australia’s infrastructure, and
to describe what they are, what we're doing about them, and what you might
In November 2015, Linux Australia was made aware of the announcement of the
closure of the Victorian based eResearch co-operative 'V3 Alliance'. Two of
Linux Australia's servers at the time were hosted by the Victorian
for Advanced Computing (VPAC), a member of the co-operative. Linux Australia
contacted VPAC, who indicated that funding existed for VPAC to continue to
operate through to 30 June 2016.
Following discussions between the Linux Australia Council and the Admin
the Admin team was authorised to locate new hosting for these servers as a
matter of priority. Discussions were started with a department within a
university located within Victoria to seek hosting space and support as
In early December, contacts at VPAC in December advised Linux Australia
the majority of staff had moved onto other organisations, VPAC would now be
auspiced by the University that had hosted their offices and equipment
effective 1 January 2016.
As a result the pace of negotiations increased, this was however
postponed due to impending Christmas and New Year holiday periods. The Admin
team ensured the servers located at VPAC were clearly marked with
information, and backups of the devices continued. New equipment was
to facilitate the rapid deployment of new hardware once a hosting agreement
could be negotiated.
In March 2016, the conversations regarding hosting were restarted, and
began for a staged and managed migration away from the old hardware to
server hardware when the equipment had a new home.
#### April 19 2016
At 3pm, the Admin team received notifications from the monitoring system
the two servers at VPAC, and the virtual machines that run on them, were no
longer reachable. As the VPAC website was also offline, the Admin team
Council that it was highly likely the servers had been powered off. Through
conversations with previous staff of VPAC, it was confirmed the site was
decommissioning process, and the number for the contractor
room was obtained. The contractor was notified of the equipment in the
negotiations began for its recovery. The contractor worked with the project
manager from the hosting institution, and an agreement was reached to
servers to be brought back online for 2 hours the following morning. It was
immediately identified with the loss of these servers that the
accounts, such as hostmaster, council and domain admin, were offline until
service could be restored.
The Admin team subsequently undertook the following steps;
- Council was updated on the status of the servers
- Backups of the servers hosted at VPAC were analysed, and a number of files
were found to be missing from backups due to permissions on the host server.
- Several of the files found to be missing were ones required to update
DNS glue for the linux.org.au and linux.conf.au domains. Replacement
requested with the expectation these would be delivered the next morning
the servers came online.
- Plans were made to change the delegation of the two domains, however,
the nature of change requests on the linux.conf.au domain, the Admin team
reached out to Jo Lim, our contact at AUDA to determine the best alternate
method to change the delegation should it be needed.
- New VMs were prepared on LA servers in Canberra to take over the roles
previously provided by the servers in Melbourne.
#### April 20 2016
At 8.30am, the Admin team were notified the servers were back online,
jobs were started to recover the missing files. Once these were
directory trees on both servers were compressed and copied to a remote
While these jobs were running, a DNS server was stood up and configured
zone files from the previous DNS server hosted in Victoria. The codes
for DNS glue changes were also recovered at this time.
The last files were recovered from the servers at 11.05am, and the
powered down by the contractors and removed from the racks. Backups of
Australia wed, DNS and mail systems were prioritised over other sites, which
meant that the LCA2010 and LCA2012 sites were not backed up from their host
servers before power was removed.
Around 2pm, the replacement DNS server was ready, and root glue for the dns
server 'russell.linux.org.au' was changed with the registrar. Once this was
live, the secondary DNS was updated around 3.30pm to pull from the new
and DNS was returned for the linux.org.au and linux.conf.au domain.
Whilst plans had been well underway to migrate the @linux.org.au email
to a new
system, the Admin team proposed, and Council wholeheartedly agreed, that the
risk at this point was too high, and the legacy configuration would be
on a new VM.
The Admin team focussed their work on this service for the remainder of the
#### April 21 2016
The legacy mail server was bought online at 10.30am, and the majority of
mail flow returned. The logs were monitored and issues were resolved as they
were identified. Around 6pm the Admin team started to restore the
LUG websites from backups, however following discussion within the team, the
decision was made to postpone this work to give team members a break and
burnout or critical mistakes from occurring, and also to deal with personal
matters that required their attention..
Linux Australia reached out to our friends at AARNet to recover the
the Datacentre on our behalf at the same time they recovered their
#### April 22 2016
Work re-commenced on the recovery of websites, continues at this time.
Australia blog aggregation site “planet.linux.org.au” is managed by a
team than the Admin team, and they will work on returning this functionality
over the coming week. AARNet notified the Admin team that the servers
recovered, and plans are underway to have the equipment shipped to the Admin
team so that a final backup can be taken
### Current situation - April 28 2016
During Wednesday 20th April you may have observed the following symptoms;
- Slow or non-delivery of email to some Linux Australia mailing lists
- Slow or non-delivery of email to Linux Australia email addresses (ie
president at linux.org.au)
- Sporadic outages of Linux Australia web based properties such as
www.linux.org.au, www.linux.conf.au and MemberDB.
At the time of writing, the current status of infrastructure is as follows;
- DNS services have been restored onto new infrastructure.
- DNS secondary services, kindly provided by Andrew Pollock, have now been
updated and are retrieving records correctly
- linux.org.au is resolving and online
- linux.conf.au is resolving and online
- Non-core sites such as some Linux Australia hosted LUG sites, Linux
Planet, radio and hosted sites etc are offline as their respective
still in the process of being restored.
### How was this allowed to occur?
Whilst Linux Australia had undertaken all possible means to ensure the
was identified as belonging to the organisation, ownership was mistakenly
attributed to a Victorian linux users group, who also had equipment
the room. The room was decommissioned at 9 weeks ahead of the last
given, which meant that migration works had not yet been completed.
### Was any data lost?
It is too early to determine this, however once the servers have been
by the Admin team, a complete analysis of the servers will be undertaken
update given to Council.
Was any personal data leaked?
Linux Australia believe the chain of ownership (VPAC to HPC Contractor
to LA) has protected any personal information held on the servers.
### Why does this keep happening to Linux Australia?
Linux Australia has previously relied on the good-will of the community or
commercial organisations to host the server infrastructure. These
often non-binding, and, in some previous situations, have been revoked
notice following a change in business ownership. Starting in 2012, hosting
agreements were prepared for new servers that ensures Linux Australia
plenty of advance notice, and also outlines a clear communication path
changes to the hosting situation. All new LA servers are clearly marked
organisational details, as well as at least 2 contact numbers should an
### What can I do to help LA?
Linux Australia servers are currently hosted at sites connected to the
network, as these sites are connected onto a high speed white space
transfers between these sites are considered to be unmetered. If you
work at a
higher-education institution that is connected to the AARNet network,
open to hosting several rack-units of equipment and have IP space available,
please contact the Council.
### Planned future actions
The Admin Team, with Council's approval, is not going to migrate mail to
system immediately. Instead, a new server environment has been built to
legacy mail configuration, and the migration planning has been picked up
where it was left.
Linux Australia has long pursued a strategy of controlling our own virtual
machines and the underlying hardware. This approach gives us the ability to
resolve hardware issues much faster than a hosting company. It also provides
greater flexibility for scheduling maintenance. Linux Australia has usually
partnered with Universities for hosting, given that transfers on the AARNet
network are un-metered, reducing the cost of data transfers. Our current
are for 2RU of rack space, a /28 block of IP space (14 usable addresses)
filtering, and a large amount of data transfer.
The Admin Team is currently liaising with some Universities located in
to identify if it's possible for them to host the Linux Australia server
equipment. Council and Admin Team will review alternative options for
due course to ensure alignment with Linux Australia's ongoing needs in this
space. Any transition will be well planned.
As always, we warmly welcome your comments, queries and feedback.
Sae Ra Germaine
secretary at linux.org.au
Linux Australia Inc
GPO Box 4788
Sydney NSW 2001
ABN 56 987 117 479
More information about the linux-aus