If you're interested in a more comprehensive source on setting up these sorts of malware analysis labs I would recommend Tony Robinson's "Building Virtual Machine Labs: A Hands-On Guide"[1] It is a very detailed guide and also supports multiple hypervisors.
This is pretty neat, however I wonder about the risks.
Is there any malware that detects that it's running under something like VirtualBox and changes is behaviour? Are there any security exploits in VirtualBox that malware could take advantage of to infect the host OS?
Regardless of the risks, this is still a neat way to analyze an unknown system (whether a single program or a suite of programs that run across multiple machines).
Lots of malware will detect it's running virtualized and change behaviour (generally to "do nothing") and there are mitigations around this detection (removing virtualbox/vmware specific serial numbers, virtualization drivers, etc) but exploits are hard, they are generally tied to a very specific version of very specific software in order to be successful, versions and products the author can't predict (are the hostile analysts running VMWare? Virtualbox? Xen? Mysteries...) and the target surface is likely very low since most malware is just going to be classified by some overworked junior analyst at an antivirus as "GENERIC.W32.RAT" well before it gets any sort of in-depth analysis and binned.
Ultimately you shouldn't worry about VM escapes by any malware you're likely to come across, they're an academic threat.
Weirdly, as a side effect of the first point, simply installing the virtualbox & vmware drivers on your real actual machine can stop a small subset of malware because it thinks it's running virtualized and shuts down
I don't know that I'd say VM escapes are an academic threat -- more that they're used for targeted attacks and not generic malware (usually).
Xen, VMWare, and presumably the others seem to get about 1 escape a year that's a serious vulnerability (and 2-3 other ones that could potentially be, but don't seem easily exploitable).
Of course, that's publicly disclosed ones, which means there may be more that we're unaware of.
Using a VM is the standard way to analyze malware. There is a threat, but its not considered to be significant at this point. If you were analyzing a malware sample that you knew for a fact performed VM escape, you would run it in a VM on a system specifically designated for that purpose.
There are lots of ways that malware can detect the presence of a virtual environment, but part of reverse engineering malware involves defeating these mechanisms.
Here's just a few simple ways that malware can detect the presence of a VM:
- Size of the hard drive. (people frequently use smaller amounts of storage space for virtual environments).
- Presence of a debugger.
- Scanning for common analysis tools (regshot, process
monitor, ollydbg, etc.
- Processes associated with VMware
- Network connectivity (like we recently observed)
If anyone's interested in learning malware analysis, practical malware analysis is a great book for someone that's already decent at programming. I'd also recommend learning assembly pretty well before you get started.
Thanks for the feedback :-) As I mentioned at the end of the post, there are indeed malwares that detect being run in a virtual environment and who change their behavior (e.g. do nothing). I'm not an expert at all on the topic, but I was recommended https://byte-atlas.blogspot.com/2017/02/hardening-vbox-win7x... on the subject. I'm interested if somebody has additional resources to provide.
About 20% between 2012 and 2014 [1], I actually believe it was much more earlier when few used VMs for production. That meant that a virus could assume that if it were executing in a VM context they were being monitored and altered their behavior to either play nice or distract.
Nowadays a VM target could be just as valuable as any other.
There have been many examples of malware detecting being run in a virtual machine to discourage analysis. (I would note that one obvious sign would be the presence of the guest additions and/or the fact that the PCI bridge will almost always be an i440fx or Q35...)
As for VM escapes... Can't guarantee there won't be a sample that comes along eventually that'll try.
As we saw with the recent Ransomware attack malware doesn't behave differently if it realises it is in a sandbox, however to date that has usually been to deactivate itself to reduce the likelihood of it being analysed. I imagine a virtualbox 0 day would be useful in these situations but probably only to state level actors or malware authors looking to get one up on security firms.
"We used a JavaScript engine bug within Microsoft Edge to achieve the code execution inside the Edge sandbox, and we used a Windows 10 kernel bug to escape from it and fully compromise the guest machine," Qihoo 360 Executive Director Zheng Zheng wrote in an e-mail. "Then we exploited a hardware simulation bug within VMware to escape from the guest operating system to the host one. All started from and only by a controlled a website."
I wonder if there's a need for a malware tear down service; submit a JSON request with the binary defined as part of the request, a spot instance spins up, allows the malware to run, and then all of the MITM network traffic and the instance memory snapshot is dropped into S3 for forensic exploration and collaboration.
That's a very interesting idea! Certainly it's an interesting take on the sandbox, and might allow for access to analysis tools even for those without the required tools or skillset. Malware is a threat to us all, and anything offering to democratize analysis is well worth considering.
With that said, I can perhaps see a couple of potential sources of additional complexity. First, a spot instance can't really offer the required level of sandboxing. A typical analysis scenario exploits the privileged position a VM host has over the guest to further analysis and safety. Attempting to do so from a peer position is more difficult and may put you in a more limited position to observe all potential behavior as you lose the ability to easily fake DNS, alter the clock, and more.
Second, it's possible that cloud hosting service providers might run services designed to look for and shut down malware-infected hosts. It would certainly be in their general interest and probably in the general interest of most of their customers. In such a scenario, it's very possible that a service like that you describe might find all analysis attempts shut down instantly rather that be able to even attempt to gather useful behavioral evidence.
Your idea is great! It's well worth exploring and holds amazing potential. It may perhaps be best done with the above caveats in mind.
Thank you! I'll explore the use of ephemeral VPCs for this tonight; that should allow for the level of network isolation (and possibly traffic routing through another VPC with network traffic inspection tooling) necessary.
I'd like to work on this. Do you want to collaborate?
I think running it on e.g. DigitalOcean is fine. In particular, "Playing with malware in public is not something to be done lightly, and potentially very dangerous (to others as well as you!)" is rather overblown. Doing it locally via Docker, on the other hand, is asking for trouble. Docker isn't a security mechanism.
We could offer a service where we can spin up a honeypot image (e.g. a fairly typical Windows installation + a few installed apps and some browsing history). The deal is, you can submit your malware to us, we'll spin up the image, run the malware, and provide real-time feedback about what it's doing. One strategy for pricing would be $hosting_fee*markup per hour.
Honestly it's one of the better ideas that's been tossed around, so if you're considering this as potentially more than a side project, hit me up.
Indeed! It's such a great idea, that services such as https://malwr.com/ already exist! As do VirusTotal and several other sandbox-driven services. With such services, the deal is that you submit your malware, analysis is performed in a variety of ways, and results are offered.
You're absolutely right that Docker isn't a security mechanism. It might be worth considering why it might have been suggested, and what local virtualization offers that AWS or DigitalOcean might not in a context where a typical analysis toolset has been dismissed out of hand.
For my own part, I've engineered services that offer precisely the featureset you've described. We found compelling reasons to not do so on AWS or similar. While your conclusions may obviously differ substantially, it may be wise to arrive at them with full knowledge of why others made their decisions.
If I'm mistaken, then just point out why. I'd rather know.
It might be worth considering why it might have been suggested, and what local virtualization offers that AWS or DigitalOcean might not.
What does local virtualization offer that AWS or DO don't?
For my own part, I've engineered services that offer precisely the featureset you've described. We found compelling reasons to not do so on AWS or similar. While your conclusions may obviously differ substantially, it may be wise to arrive at them with full knowledge of why others made their decisions.
Such as?
I'm not doubting you, so please don't take it as such. On the other hand, if you don't have time to describe your findings then no worries. I'll likely rediscover the wisdom independently.
I led the team for WildFire (the most widely used commercial sandbox and, I believe, the first one in the cloud) at Palo Alto Networks. If anyone from FireEye or Lastline is around, maybe they'll chime in as well.
The design of the analysis system itself determines your deployment constraints. For example, will you give the malware an open line to the Internet? If not, you'll miss out on most of its activity[1], but won't be limited in terms of hosting. If, on the other hand, you give the malware free reign, AWS wants no part of that (we tried it).
1. See the (speculated) anti-analysis kill switch in WannaCry. This sort of thing is common. But more fundamentally, malware also tends to work in stages, and often won't do much if it can't phone home.
Please accept my apologies. I was under the mistaken impression that I had outlined some of the difficulties previously upthread. Allow me to remedy this error.
Having control over the virtualization as the host offers a lot of power to protect yourself, contain your infected virtual machines, and perform analysis. Using EC2 instances as your victims means giving up much of that.
For example, I don't know of a way offhand to perform a full memory dump of a running EC2 instance without its cooperation. Memory dumps can be very useful forensic data, but Volatility cannot be run against an EC2 instance. Other items that come to mind include the ability to falsify clock behavior, more or less arbitrary simulated hardware, and the ability to pause execution. All of these can be very useful in malware analysis.
Also, needless to say, running live malware so you can passively watch its live behavior without isolating it from the world is perhaps not ideal behavior. No responsible engineer wants to see their analysis machine involved in an attack. Such things have happened when insufficiently cautious engineers, with the purest and greatest of intentions, hooked up live malware to the internet.
I hope these points have helped you understand. Please don't hesitate to ask if you have further questions.
I fear you misinterpret me. You may want to consider doing this in a strictly local manner with Docker hosts first. Playing with malware in public is not something to be done lightly, and potentially very dangerous (to others as well as you!) in situations where your control is any less than nigh-total.
I beg of you, please do not in any way, shape, form, or manner interpret my above comment as encouragement to place the systems of others at risk.
What you have described leads me to think that you want to execute malware in a live cloud environment where you do not have a host/guest advantage and trust to man-in-the-middle attacks on live traffic.
I was channeling Dale Carnegie so as to not completely discourage someone. I can see that this was the wrong approach.
It is no surprise to see articles and lots of comments. I was in a class where people wanted to set up Cuckoo Sandboxes with the controller system in a VM as well, not just malware testbeds and the the controller system in Python installed system wide in the host.
It makes this stuff annoyingly difficult. I was also very very interested in Drakvuf, a system which inspects the VM from the Xen host on the host side with special shims for memory and OS introspection.
[1]: https://www.amazon.com/dp/B071G4SCB4