Hello there.
We have a bunch of obsd firewalls, 8 at the moment, all working nice and

so forth. But we

need to add about another 4 in there for new connections and networks,

which means more

machines to find room for.
So basically I have been asked to investigate running all these

firewalls in two big boxes, with lots

of NIC's, with a bunch of openbsd vritual machines on them. One main box

for the primary firewalls,

one for the secondary. Each virtual machine getting its own physical NIC.
Personally I dont really like the idea, I can see things going wrong,

lots of stuff balancing on a

guest os and box.
Can someone please inform me if this is a really bad idea or not,

ideally with some nice reasoning?
Cheers,

    Josh

Hi Josh,
VM is great and I use it a lot for test and development. It's a

marvellous tool. I also think it's very good to make virtual

web/ftp/whatever servers, it eases maintenance and add a layer of

reliability.
That being said it doesn't make sense to virtualize the whole network,

especially firewalls, on the same machine. Your virtual OpenBSD is

just going to be a fancy and heavy weight personal firewall. Personal

firewalls are useful but they don't offer the same level of security.
Your call.
-- 
Die Gestalt

I'm going to vote this idea as "really bad" for two reasons.

Considering the technology out there at the moment, I'm going to

guess you're looking at vmware.
As everyone keeps saying, the security and reliability of your

systems now inherits these characteristics from the VM host, and in

some cases, the other guests. I personally can't trust the host after

having worked on OpenBSD drivers (mpi and vic) inside virtual

machines. In the mpi case I do know they took shortcuts in their

emulation of that hardware which required extra tweaks in my driver.

Even worse though, when I was working on vic I used to be able to

crash the host by getting something wrong in the guest.
I don't want a bug in one of the VMs bringing the whole lot down.
Secondly, the performance of your guests aren't going to be as good

as the performance of OpenBSD running directly on the same hardware.

If you look at the path an arbitrary packet takes on the way out of

an OpenBSD firewall its loosely something like (this path be

accurate, but that's not the point): routing table, pf, enqueue,

driver, hardware. If you look at that same packet in a VM: routing

table, pf, enqueue, driver, virtual machine, vm input, vm routing, vm

enqueue, vm driver, hardware. The amount of work per packet increases

so the time taken to deal with a packet goes up, which means that the

hardware can't deal with the same number of packets that a real

OpenBSD install can deal with.
Just my two cents,

dlg

Check out the HP c-Class BladeSystems offerings. It is sad that HP is

marketing it with virtualization via Vmware. Just disregard the vmware

affair.

I don't like the idea of virtualiazing the firewalls either.  It's

just asking for trouble.
What happens when the host OS gets hacked?  Better I think to get some of these:

http://www.netgate.com/product_info.php?cPath=67&products_id=369
and some soekris boards.  You'll be able to fit 2 firewalls per 'u'.

Then either use VLANS, or put a nic on each segment.
-Bryan

Some commercial firewalls (i.e. Juniper/NetScreen ScreenOS-based gear)

have been offering virtual-systems for years now.  I think the negative

comments received here may be appropriate when sharing the system with

non-secure guest OSs, but it seems that it might be alright if its

nothing but firewalls
Cheers,

Kent

It sounds to me like the comments here are largely appropriate,

virtualizing firewalls in the limited context that has been explained

probably isn't a real good idea...at least due to perceived load.

Additionally, if there are that many fireuwalls being ran, instead of

numerous interfaces in a fewer number of machines, you're going to

continue to have problems being able to virtualize enough hardware

network interfaces.
However, I don't fully agree with the sentiment that running a firewall

in a virtual machine (let's be specific, VMWare ESX) guest environment.

I'm running my firewall on a ESX 3.0.2 guest, and it works perfectly

fine.  That being said, you have to be aware of the VM configuraton.

The majority of vulnerabilities in VMWare are patchable (so yes, someone

needs to do maintenance), but are also issues that affect the VMKernel

or service console, and with careful planning, the vulnerabilities can

largely be prevented for being used as exploits on external interfaces.
And one final note...although I am a fan of virtualization (I work for

the company that owns VMWare), I really, really wish they did not have

so many freaking patches...

I don't understand the logic of having multiple firewalls on one box.

If one box can handle the throughput requirements of all the NICs, why

not just one big firewall?
Doug.

Douglas A. Tutty wrote:
There are lots of places where multiple firewalls are better than a

single firewall.  If one believed in the idea of "a perfect VM

environment", it could make sense to do that.
1) Unrelated projects: If Project A and Project B are not related,

keeping them on separate firewalls can simplify the rule sets and

administration.
2) Separate administration: If you run a data center with lots of

different people managing different systems, "They" can administer their

systems without having access to (or messing with) "My" systems'

firewall.  When they screw up their rules, they don't break my systems

(and I guess it works the other way, too. :)  Note this has some

cross-training benefit, too. I can be the Firewall Deity, but I do want

to go on vacation. Fred may be a Firewall Jester, but with a bit of

practice, he could possibly back me up very effectively.  So, Fred

manages a firewall for his projects, when he screws up, he learns

lessons on a simple system, and when I am not there, he can babysit

the "big" firewall, and if I get run over by a bus, he knows how to

keep all the systems running.
3) Isolation: I had set up a firewall for a web app some time back.  I

had ZERO trust in the skills of the web developers, and even less for

their security programming skills (and similar trust in my skills to

audit their code).  So, I stuck their app on its own firewall,

completely isolated from our production environment.  I also made sure

that the various machines in the thing were each attached to their own

leg of the firewall, so that we really had several layers of security

between the Internet (bad guys) and the database (the valuable stuff).

You would have to knock over Apache, then the app, then the DB to get to

the data.  Even then, they get to a DB Server which had ONLY THE BARE

MINIMUM data required to accomplish the task at hand.  If it wasn't for

this design, you can be sure that database server would end up serving a

lot of things as, $18k Oracle licens...
[ message continues ]

Hi Nick.
I understand your reasons.  To me they look like reasons for separate

firewalls on separate boxes.  In the scenarios you mention, would you

put separate firewalls on one machine?   
If I was going to put them all on one machine, I'd separate the

administration of the box itself (me) from the people responsible for

rule sub-sets.  E.g. if one sub-firewall is dealing with traffic between

NICs 1 & 2 (call it channel A), another between NICs 3 & 4 (call it

channel B), I'd have the channels A and B admins submit rules sub-sets

via rsync to the box.  My script would then sanity check (ensure that

they only dealt with the interfaces they were assigned) then incorporate

all of them into a master rule-set that would then get tested and then

put on-line.  I would think that this, being only one firewall, would be

simpler than several firewalls in VMs on one box;  possibly more secure

given the comments in this thread about the porus isolation between VMs.
That's just how I would think of it.  OTOH, I've never done any

virtualization and never been into a proper data center.
Doug.

Douglas A. Tutty wrote:
That's where you are supposed to 1) recognize that my mysteriously

mangled e-mail address is me and 2) Read back to my previous statement

where I stated that I don't feel VM technology is suitable for

externally exposed apps or security critical apps and 3) catch the

implied sarcastic sneer in "If one believed in the idea of 'a perfect

VM environment'"
Yes, very separate is what I was recommending: no VM, keep them as

separate as possible.  When appropriate, of course.
VMware and related technologies look cool, but it's an extra layer

of complexity and security vulnerabilities.  It is also a technology

where the track record is "Coolness first, security when they catch

us with our pants down".  It is also something that is rarely done

properly (for my definition of "properly"), but that's a different

discussion for a different list.
Nick.

Thanks Nick.  I don't catch sarcastic sneer much in person, yet alone

via email.  
Doug.

Overlapping IP address space.
ciao
Luca

someone just needs to sit down and add the code to put interfaces into

alternate routing tables and arp running there and you can have that on

openbsd.

ok, it is a bit of work (that I am not very interested in).

but the hard part (introduction of multiple routing tables) is already

done.
--

Henning Brauer, hb@bsws.de, henning@openbsd.org

BS Web Services, http://bsws.de

Full-Service ISP - Secure Hosting, Mail and DNS Services

Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

That's why god created competant network admins and NAT.
--

"This officer's men seem to follow him merely out of idle curiosity."

-- Sandhurst officer cadet evaluation.

You are not always in control of all things. Powerful technology is

about choice, not about one absolute right way. BTW, NAT sucks.
ciao
Luca

And VRF.

We are talking about OpenBSD here, and support for VRF is not there.
ciao
Luca

That may change faster then you expect

These are great news. If the implementation will allow to assign

interfaces to different VRFs it would solve the virtual router/firewall

setup without the need for OS virtualization.
ciao
Luca

On 22.09-02:06, Luca Corti wrote:
i have a feeling that the funds currently available for your virtualisation

project would improve the quality and delivery of these requirements.

If I had such project and funds I'd certainly contribute. In the

meantime I have assigned part of my limited resources to buying the CDs

for the new release...
ciao
Luca

On 23.09-01:35, Luca Corti wrote:
my apologies, i had you confused with the original poster.
--

        t

 t

                 w

Oh, please....
/Tony

(I'd hoped you would have prefaced that with a statement like "these

are my stock options talking, but...")
This is the kind of bad advice that virtualization companies (and

naive users of those technologies) need to stop spreading. This

security model is flawed, and people should not rely on these virtual

machine environments to provide firewall services.
Here's an entirely realistic scenario at this point:
- Administrator pays loads of money for VMware ESX; for better ROI, he

intends to replace several systems on the network with one big system

running a number of VMs. Maybe there is a full DMZ (say, 10 hosts) on

this box. One virtual machine is configured as a firewall, intended to

provide packet filtering and other network security services for the

other DMZ VMs.

- A vulnerability is discovered that allows an attacker who has

presence in one VM to execute arbitrary code on the host OS, or

transfer files between guest and host. (Both of these have happened

already. In fact, VMware Tools seems to be the perfect bit of flawed

gateway software to make this even easier.) Virtualized segmentation

is compromised at this point.

- Attacker now has presence on host OS and can fully control all 10 of

the VMs running on the host. VM segmentation was supposed to prevent

this, remember? This includes the firewall which he can now play fun

games with such as overwriting the ruleset. He can sniff network

traffic for all the VM hosts since he has direct access to the host

interface. In one short subversion, 10 (11) systems have been

compromised through one flawed security model. A weakness in one VM

becomes the thing that makes compromising all the others dramatically

easier. Why subject your firewall to that?
At least in a traditional non-virtualized firewall model, the attacker

would have to pull out real exploits and attack real (secured)

services to compromise the firewall, and it wouldn't fall at the same

time as the other hosts.
Yes, these kinds of of flaws have (so far) been ...
[ message continues ]

so what do you recommend?  running all 10 services on the same

non-virtualized machine?

When I provided patch support for Solaris 10, the number of times that a

patch would not add to the global zone, thus affecting all local zones

on the host, was fairly common. This affected airlines, banks and oil

companies that you have heard of......
I know an OS is different to a VM platform, but you are still relying on

someone else to do their bit. And just because you have a lottery ticket

worth of support contract does not mean that it will actually get

patched in a timely manner, from experience....

I have no detailed knowledge of those devices, but I'm pretty sure their

virtual

firewalls are not accomplished by virtualizing the OS, but by supporting

multiple routing tables and rules-sets.
Virtual and virtual may be different in reality.
/Tony

Read this:

http://advosys.ca/viewpoints/2007/04/fuzzing-virtual-machines/

Read the paper linked there as well.  Always good to go back to original

source material.
Anyone who told you VM technology and security had anything to do with

each other was full of doo-doo.
After reading that, I'd not want to put any "externally exposed" apps on

a VM system.  Granted, OpenBSD might not be the best entry point for a

VM attack, but the foundation VM design is based on isn't as solid as

people think.
Nick.

Ironically, today's ISC handler's diary entry talks to this as well.
http://isc.sans.org/diary.html?storyid=3411&rss
DS

I'll echo Nick's statements here. Virtualization does not provide

reliable enough segmentation to rely on for security assurance. Do not

buy into the market smack the vendors are putting out about it.
As far as that goes, the more time goes on, the weaker the assumption

of virtualized segmentation becomes. Research from IntelGuardians and

other groups appears to be coming closer to completely unraveling

virtualization security, at least in terms of how it's implemented in

VMware for example. See also CVE-2007-0061, CVE-2007-0062,

CVE-2007-0063, and CVE-2007-4496.
DS

Virtual equals complex.
Network devices are supposed to be reliable.
Complex does not equal reliable - just ask anyone that has served in the

military!

What type of throughput is required between each segment?  If you've

been around here much, you've probably heard me espouse on the

benefits of VLANs.  This is certainly more elegant and secure than

running a number of virtualized OpenBSD systems on non-OpenBSD

virtual host.
There's nothing wrong with running multiple firewalls where your

physical topology dictates it.  Virtualizing numerous firewalls in

the same chassis just doesn't make sense.
---

Jason Dixon

DixonGroup Consulting

http://www.dixongroup.net

Well, heck, if he's thinking of putting in lots of interfaces

(probably to the tune of 1 interface per firewalled segment), why not

just run ONE or TWO firewalls?  Either vlan the things or dedicate one

interface per network segment, both work well.
Actually, use the two boxes, and carp them for failover.
--

"This officer's men seem to follow him merely out of idle curiosity."

-- Sandhurst officer cadet evaluation.

Because we have no idea what his requirements are.  That's exactly

why I asked for them.  Obviously, CARP is good in any scenario, but

it only provides redundancy.  It has virtually nothing to do with his

network design.
---

Jason Dixon

DixonGroup Consulting

http://www.dixongroup.net