[Linux-aus] Linux Australia server breach

Sat Apr 4 16:42:38 AEDT 2015

Dear Linux Australia Members and Conference Attendees,

In accordance with our values of transparency and openness, we wish to inform
you of a security breach of Linux Australia's servers. This incident has
resulted in the possible, but not confirmed, release of personal information.
This communication provides full disclosure of the nature of the breach, the
actions undertaken by Linux Australia to remediate it, and the actions we now
request you undertake to secure your personal data.

In line with guidelines provided by the Office of the Australian Information
Commissioner, specific information regarding the data breach, and the data
which may have been disclosed, is outlined below.

What was the nature of the breach and how did the breach occur?
---------------------------------------------------------------

Between 0600 and 1100hrs (AEDT) on the 22nd of March, a large number of error
reporting emails were sent by the Conference Management (Zookeepr) hosting
server. This server hosted the conference systems for linux.conf.au 2013, 2014
and 2015, and for PyCon Australia 2013 and 2014.

The error emails were generated by the automatic deployment of code merges to
the various Zookeepr instances, and it is not uncommon for large numbers of
these to be generated as generalised network routing or other issues occur.

Upon investigation of the source of these error emails on the 24th of March,
it was discovered that between 0400 and 0600hrs (AEDT) on the 22nd of March,
the server was subject to an attack by a malicious individual. It is the
assessment of Linux Australia that the individual utilised a currently unknown
vulnerability to trigger a remote buffer overflow and gain root level access
to the server.

A remote access tool was installed, and the server was rebooted to load this
software into memory. A botnet command and control was subsequently installed
and started. During the period the individual had access to the Zookeepr
server, a number of Linux Australia's automated backup processes ran, which
included the dumping of conference databases to disk.

In response, Linux Australia have undertaken a number of steps to minimise the
immediate damage. These are outlined below.

Whilst there is no indication that personal information was removed from the
server, the logical course of action is that we operate on a worst case
situation, and proceed on the belief that this has occurred.

What type of personal information was possibly disclosed?
---------------------------------------------------------

The database dumps which occurred during the breach include information
provided during conference registration - First and Last Names, physical and
email addresses, and any phone contact details provided, as well as a hashed
version of the user password.

As Zookeepr uses a third party credit card payment gateway for credit card
processing, the database dumps do not contain any credit card or banking
details. The payment processing process on the Zookeepr system was
specifically designed to send minimal information to the payment gateway, and
as a result only receive back a payment success or failure code. All other
payment details are handled by the payment provider's systems. Therefore,
credit card information was not disclosed.

How was the breach identified, investigated and validated?
----------------------------------------------------------

Linux Australia's experienced and respected Admin Team implements a separated
three-person response protocol for all incidents. In this methodology, after
the initial notification, one member of the Admin Team is removed from the
assessment, and is not briefed on what the other two find in their
collaborative assessment.

Once this is complete, the third person then investigates the reported
incident and develops their own assessment, without bias or prejudice. The
results are then discussed and any anomalies are scrutinised and assessed by
all three members, and, if needed, undergo further investigation until all
members of the team have confirmed the discoveries and results.

What are the implications of the security breach and what should I do?
----------------------------------------------------------------------

Whilst Linux Australia do not believe this was a targeted attack against the
Zookeepr conference management system, nor an attempt to harvest details from
the system, we are taking the necessary precautions to review, remediate and
minimise the risk of exposure to attacks similar to this.

How did Linux Australia respond to the breach?
----------------------------------------------

 - The Admin Team immediately suspended all non-admin system accounts on the
   Zookeepr server to quarantine all information relating to the attack.

 - The remote access software and botnet software were isolated and the init
   scripts removed from the system for later assessment.

 - The 'rkhunter' software was installed for the first time, and multiple test
   scans were run.

 - The system underwent a number of reboots to ensure the software installed
   by the attacker was removed.

 - The modification time of shell history files were checked, and then the
   file contents were inspected to ascertain the activities of the attacker
   [0].

 - Logs were checked in an effort to ascertain the method the attacker used to
   gain access.

 - All other Linux Australia servers hosted on the hardware were assessed and
   where required, their security measures were increased.

What steps were taken to prevent the threat of a similar breach in the future?
------------------------------------------------------------------------------

 - The compromised host is being decommissioned.

 - A new host was built, and the PyCon Australia 2015 production instance was
   re- deployed onto the new Zookeepr host.

 - This new host is enforcing key-based logins only, and a number of other
   security measures have been applied to attempt to limit the attack surface.

 - The new host will have tighter restrictions for services facing the
   internet

 - The new host will have a far more rigorous operating system updating
   schedule applied to it.

 - Logs are duplicated to a central log server where a log analysis tool has
   been installed, this will alert the Admin Team to suspicious activity when
   detected.

 - System user accounts on the new server will be expired 3 months after the
   conference ends (with special arrangements for PyCon Australia's 24-month
   cycle).

 - linux.conf.au and PyCon Australia sites will be converted to HTML copies 6
   months after the conclusion of the conference. The conference's Zookeepr
   database will then be archived and stored on a separate server, and the
   database deleted from the ZooKeepr server.

I'm a previous linux.conf.au/PyCon Australia attendee. What should I do?
------------------------------------------------------------------------

For your security, we strongly encourage you change your passwords on other
web services if the same password may have used when registering for our
conferences. This would also include your Mozilla Persona accounts if you have
chosen to use this method for authentication[1]. In the interests of improving
your online security, it is recommended that a one time password service be
used in the future for any accounts you may create on any web services
including Linux Australia's conference websites.

Has assistance been offered to Linux Australia?
-----------------------------------------------

No assistance has been offered, however Linux Australia is interested in
working with relevant Australian based Computer Emergency Response Teams
(CERT) or accredited computer security experts to determine the method the
attacker utilised to gain access to the system.

Who should I contact for more information?
------------------------------------------

Thank you for your patience, understanding and support. If you have any
questions, concerns please do not hesitate to contact the Linux Australia
Council at council at linux.org.au or if you would like speak in camera please
contact the Secretary at secretary at linux.org.au [2].

Signed, The Linux Australia Council

[0] Shell history files do not appear to have been modified or removed by the
attacker, as a result of which the Admin Team were (to a large extent) able to
interpret the activities undertaken on the machine during the incident.

[1] Whilst Mozilla Persona provides a central authentication token that
verifies the user identity in a way that does not expose the password to the
system being authenticated to, it was necessary for users of Mozilla Persona
to set a password on the Zookeepr system to be able to edit the conference
wiki.

[2] Please note that this is an archived email address but steps will be taken
to protect your privacy.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.org.au/pipermail/linux-aus/attachments/20150404/f4e54345/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linux.org.au/pipermail/linux-aus/attachments/20150404/f4e54345/attachment-0001.sig>