Any reason this type of work couldn't be done with mininet (http://mininet.org/)?
I can see this was developed for NRL research, so not trying to say it's a knock off or anything. Seems to have been developed a few years later, but close to the same time, one of those things like Deep Impact and Armageddon both coming out at the same time.
Anyway, having used mininet quite a bit back in the day in grad school, this kind of thing is great. Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies. It's really cool, stuff you would normally need to work for AT&T with access to millions of miles of fiber to have a chance to do.
Shadow developer here! Disclaimer - I joined the project a little over a year ago. I'm not as well versed in some of the related work as my colleagues, but I can take a shot -
Mininet looks cool!
It looks like Mininet lets programs run pretty close to natively, using network namespaces to pipe data over its simulated network, and some other lightweight techniques for isolation and some extent of reproducibility (e.g. using the fifo scheduler).
Shadow intercepts the libc API via LD_PRELOAD, allowing finer grained control over the process's view of the world. In particular it lets us emulate time - we can run simulations with tens of thousands of endpoints on a single physical machine; even if we don't have the CPU power to run it in real-time, the simulation behaves as if we do. Conversely, small simulations can be run faster than real-time.
The down-side is that it's more difficult to support arbitrary software, since there are various ways of "escaping" LD_PRELOAD, and we don't support the full libc API. We're in the process of switching to a ptrace-based approach that is more robust, at the cost of some additional runtime overhead.
> Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies.
It sounds like mininet has a lower-level model of the network itself. Shadow simulates links between endpoints, including bandwidth limitations and drop rates. It doesn't model individual routers along the path, so yeah I don't think you could use it to simulate BGP.
A question: I tried poking around with `tc` (qdisc) a while back to limit speed and emulate latency, and I found the implementation seemed to simulate delay by waiting a bunch of time then dumping all the data at once, for example if I asked for 5s of latency I'd be sitting at 0 bytes/sec for 5 seconds then it'd be like INCOMING and I'd get a giant pile of data all at once.
I do absolutely have horrendously little networking experience :) so I was probably fiddling with the knobs wrong (and fiddling with wrong expectations from the mechanics), but I didn't see any tunables for ramp-up et al.
For a while I've been idly curious how to iterate on applications that need to respond favorably to everything from 10Gbps and "what latency?" to flaky 3.5G with "give up" latency and worse jitter. At the end of the day I know I need real networks to play with, but I've been curious what software will get me the closest approximation in terms of microsecond-to-microsecond network behavior.
TBH this question is me being somewhat lazy :) - I'm not yet ready to start working on the above ideas, so I'm yet to actually sit down and do some basic prerequisite research... but I thought I'd jump on the opportunity to ask while I could.
Sorry, I'm not well versed in the network model; most of my work has been on the new ptrace-based syscall interposition. That sure sounds like a bug, though.
We recently added a discussion section to Shadow's GH repo: https://github.com/shadow/shadow/discussions. That'd be a good place for your questions. More knowledgeable folks are pretty responsive there, but aren't on HN :)
Apologies for the ambiguity and lack of context - to clarify, I fiddled around with raw tc/qdisc (part of Linux's network stack) a while back, and was wondering if Shadow had more accurate timing simulation.
The discussion section sounds like exactly the place to follow up on this sort of thing, thanks for the reference. (Once I verify what I'm looking for is actually exotic :P)
We've started looking into eBPF a bit - IIUC eBPF by itself doesn't give us the ability to service or arbitrarily manipulate the traced process's syscalls.
We have recently learned of an interesting technique that dettrace [1] uses of combining seccomp with an eBPF filter and ptrace. Instead of generating a ptrace-stop for every syscall (as we do now, using PTRACE_SYSEMU), they use a seccomp policy with an eBPF filter, s.t. a ptrace-stop is only generated for syscalls that violate the policy, allowing them to emulate the result of those syscalls. syscalls that don't violate the policy are allowed to execute natively, saving a lot of overhead.
This works great for them since they want to emulate a relatively small subset of syscalls. In our case we want to emulate most syscalls, so it's not as clear-cut of a win. We have found though that if we use an LD_PRELOAD'd shim in the target process to intercept syscalls and then service them via IPC, that's substantially faster than catching them with ptrace. That runs back into the problems with LD_PRELOAD in general of there being various ways of missing syscalls. but, we may be able to use that technique along with ptrace+seccomp+ebpf to intercept any syscalls that we'd otherwise miss. The seccomp technique would allow us to exempt the syscalls that our shim itself is making to do the IPC.
Both Emulation and Simulation are valid tools to perform network research.
Mininet is a network emulator designed to run real network software. It emerged as a leading tool to evaluate software defined networking solutions. Mininet leverages openvswitch which is the same software that would run on the networking boxes. So the TCP/IP stacks are emulated. This means that the system has very high fidelity making it behave close to the real system. Mininet runs in 'wall-clock' time. (or real-time)
On the other hand, simulation has a more difficult time achieving the same fidelity due to the difference in the execution modes. Discrete event simulation works by maintaining a queue of events. Each event has a timestamp that advances the simulation clock forward. Each event can create 0 or more events. The behavior of the network (adding delays / dropped packets / etc.) are simulated with models. This simulation allows a process to run real application code by invoking it from an event handler. A discrete event simulation uses virtual clocks which can be faster or slower than wall-clock time.
As a general rule we can say that simulation is more scalable than emulation (10s vs 1000000s of nodes) while emulation has higher fidelity (more accurately captures behavior of real system).
Glad to see mininet mentioned here. I used mininet academically for a graduate level course in computer networks, in combination with Pyretic (http://frenetic-lang.org/pyretic/), and it was a pleasure.
Spinning up a network topology simulation using mininet was effortless, and it proved to be an invaluable tool for network research. It's good to see other projects such as Shadow being developed in this space as well.
Shadow's primary use-case is simulating the Tor network. Shadow's tor plugin has a step-by-step for getting a tor network simulation in particular running: https://github.com/shadow/shadow-plugin-tor/wiki
For anyone wondering how this might actually work, "Shadow dynamically loads these libraries to natively execute the application code. Shadow intercepts a selective set of system calls to enable seamless integration of an application to the simulated environment. In this way, the application may be unaware that it is running in the simulator and will function as if it was running in a standard UNIX environment." https://github.com/shadow/shadow/blob/main/docs/0-Design-Ove...
It then uses the an event queue to transfer the packets between virtual hosts, and applies rate-limiting, bandwidth, jitter, packet loss, and TCP stack delays and CPU processing delays, among other things I'm sure. Very cool!
A quick look (by searching for bgp) suggests that this handles the upper layers, but doesn't try to simulate some of the deeper layers of a network such as bgp. Which something that emulators like core can do [0].
Imunes looks more similar to Mininet[1]. The networking stack in Shadow looks simulated so you could do more large scale research. It seems that Imunes claims one use case is product testing so more likely they are targeting fidelity.
I can see this was developed for NRL research, so not trying to say it's a knock off or anything. Seems to have been developed a few years later, but close to the same time, one of those things like Deep Impact and Armageddon both coming out at the same time.
Anyway, having used mininet quite a bit back in the day in grad school, this kind of thing is great. Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies. It's really cool, stuff you would normally need to work for AT&T with access to millions of miles of fiber to have a chance to do.