March 17 2015 Incidence Report
At 9:27 UTC an unauthorized withdrawal for 150 BTC was sent from Coinapult's hot wallet to this address: 12LszeXACdj9bdETzv8BkXyWeabZ1151aA
The transaction id is:
As of 15:41 UTC these coins have not been spent.
Coinapult Team Involved
Ira – CEO with physical and SSH access to servers
Zach – IT Admin with physical and SSH access to servers
GP – CTO with SSH access to servers
Cindy – Developer and forensics expert with SSH access to servers
Justin – COO with no access to servers
Robinson – Customer service with no access to servers
There are 2 people with physical access to the affected servers. The servers are kept in a tier 3 data center with layers of physical security.
There are 4 people with SSH keys to the production finance server where the hot wallet is kept. To connect to the finance server, one of these would have to log into the company VPN, then use their SSH keys to log in to the server via command line.
Each of the production key holders' laptops were inspected by the others for network activity from the time window with nothing suspicious found, however Zach's laptop was exhibiting strange behavior reminiscent of a MITM attack. While we were all using the same local network, his IP address showed Gabon while the others all showed Panama. After we discovered this, Zach powered down his laptop and we are taking the hard drive out for forensic analysis.
On Friday March 13th two things happened which are potentially related to this incident.
1. The data center where the finance server is hosted experiences an all day outage. The Panamanian government websites and numerous other local businesses were also down. The phones at the data center were down. Zach logged in to almost every server in the data center that day, as part of our recovery from the outage.
2. GP created and emailed Justin, Ira, and Zach a plan to transition all IT services to different servers outside of the data center. This may have been a last chance notice for the attacker, as their penetration work would be undone in the transfer.
In addition to the March 13th events, the previous 2 weeks have been unusually problematic for system stability. We have had hard drive problems, CPU problems, and all sorts of side effects. While the causes of most of these are known, it is possible some of it was malicious and/or masking other activity.
Clues from Finance server
/var/log/auth.log appears to be modified. There is a blank line around 10:24-10:27 UTC and auth.log.1 is empty. Auth.log.1 should be full of data from the past few days.
/root/.bash_history was last modified on Mar 14 19:38 UTC. The last 4 entries are:
These commands appear to have been run by the attacker, as the pattern does not match Coinapult usage. It looks like he opened these key log files with the intent to doctor them after he leaves. The lack of other commands and evidence suggests the possibility of a root kit. This may show up in forensics analysis of the hard drives. The use of 'nano' is also odd, as most hackers would chose a different text editor.
Clues from API & Data servers
API server takes 30 minutes to send a message to the transaction processing engine (on the data server) at around the time of the attack. This is what caused <Customer 1> to reach out to Robinson at 9:15 UTC. The cause of this is currently unknown. Compromising either of these servers would not allow direct access to the hot wallet or allow Finance server access.
Clues from SaaS servers
Coinapult runs a second set of production servers to rent out on a SaaS model. These SaaS servers have the same access rights as Coinapult's own. (2 physical access, 4 virtual) At the time of the theft, the SaaS finance server had over 500 BTC in it, which were not touched. This indicates the hacker did not know about that wallet. Of course, the funds have been moved and are now safe.
All times below are UTC-5.
1:49 – Ira requests hot wallet top off with 100 BTC from Bitfinex.
2:36 – Ira logs in to VPN (according to his syslog)
2:36 – Ira logs in to Finance server (according to server log)
2:37 – Ira runs sendmany to split outputs for optimal sending performance during the night. This was unnecessary as the 100 BTC had not shown up yet, but Ira did not notice that. Sendmany transaction:
3:55 – Bitfinex sends 100 BTC withdrawal: 4691cf88a74cc752c3777ec9952500cc04bd02ea6f979e326d3550b1581723c3
4:15 - <Customer 1> notifies Robinson about improperly canceled transactions
4:27 – Withdrawal by hacker is made:
4:54 – Robinson sends out email about <Customer 1> transactions being stalled and HW being suspiciously low
4:58 – Robinson calls Zach and Zach starts trying to connect to VPN (according to his syslog)
5:17 – Zach successfully logs in to VPN (according to his syslog)
5:22 - Zach logs in to Finance server (according to server log)
5:31 – Zach sends email saying processes are running but can't assess HW on his own
8:42 – Ira has done enough investigation to identify that there has been 150 BTC withdrawn to an unknown address. Emails this info to the group.
9-12 Majority of funds are withdrawn from HW (minus change to watch). Customers (i.e. <Customer 1>) are notified and public notice is placed on our website. Team investigates and identifies the contents of this report.
Robinson was first on the scene, and so far, all evidence and corroborations agree with Robinson's timeline. Robinson had neither access to the servers nor the technical knowledge required to perpetrate the attack.
10:11:50 AM Ira hey, what exactly happened this morning?
10:11:53 AM can you give me the timeline please?
RO 10:12:15 AM Robinson sure.
10:13:02 AM <Customer 1 employee> sent a message about 2 <Customer 1> transactions at 4:15 this morning
10:14:30 AM I saw the payments were canceling incorrectly, but I didn't know the reason
IR 10:14:56 AM Ira you were awake?
RO 10:15:33 AM Robinson yea, I woke up at 3, couldn't sleep, was doing some journaling when the <Customer 1> message was received
10:16:25 AM the HW low balance emails started being sent at 4:30
10:17:30 AM I messaged both you and justin about that
10:18:16 AM justin pointed out that 100 BTC had been withdrawn from Bitfinex, but the volume in admin didn't add up to a low HW
10:18:31 AM that is when I sent the email and started calling you
IR 10:18:46 AM Ira justin was up to?
RO 10:18:46 AM Robinson after I tried you a couple of times I called zach
10:19:45 AM he said he was already awake
10:20:46 AM I spoke with Zach for ~2 minutes, mainly to tell him to check the email I sent
IR 10:23:09 AM Ira then he sent his email saying everything is fine?
RO 10:26:17 AM Robinson he said on the phone he would log in to investigate
10:26:48 AM he said in the email he doesn't know how to troubleshoot the hw
IR 10:27:22 AM Ira to clarify
10:27:28 AM Justin was already awake or Zach was?
RO 10:27:45 AM Robinson Justin said he was already awake.
10:28:15 AM When i called Zach, it sounded like I woke him up
We have powered down and isolated all of the hardware in the data center. The plan is to disassemble them then run forensics on the hard drives to see if we can recover data from the manipulated logs or anywhere else.
Zach is disassembling his laptop to run forensics on it.
March 19 2015 UPDATE: The Gabon IP address issue was identified as a buggy holaVPN client. It is not likely related to this incident.
While we are moving the hardware out of the data center, we will ask for access logs and/or surveillance footage relevant to our situation. In addition, we will gather more information about the March 13th outage they experienced.