Intrusion Detection Systems

Whenever a company connects its network to the Internet, it opens up a whole can of worms regarding security. As the network grows, it will play host to numerous bugs and security loopholes of which you have never heard—but you can bet intruders have.

Many organizations are recognizing the value of a good security policy to define what is and is not allowed in terms of network and Internet access. Then they deploy a number of tools to enforce that security policy—usually in the form of a firewall or two.

Firewalls may be billed as commodity items, but the "shrink wrap" element certainly doesn't extend to their configuration. A detailed knowledge of what a hacker can do and what should and shouldn't be allowed through the firewall is required before embarking on the configuration adventure, and a slip of the mouse is all it takes to open up a hole big enough for your average hacker to drive the proverbial bus through. The problem is, a badly configured firewall can be worse than no firewall at all, since it will engender a false sense of security.

To protect an organization completely, therefore, it is necessary to provide a second line of defense, and in order to achieve this, an entire category of software exists in the form of Intrusion Detection Systems (IDS). When it comes to computer and network security, there are a number of analogies that can be drawn with the "real world". Such analogies are particularly useful for answering such questions as "I already have a firewall, why do I need Intrusion Detection Systems as well?"

Depending on how you approach the security of your home, for example, you may opt for high quality locks on your doors and windows. That will help to keep intruders out, and could be thought of as the equivalent of the firewall—perimeter defenses. It's nice to feel secure, but the determined burglar can often find ways around these measures. He can always throw a brick through your back window, for instance, and get in that way—or perhaps you simply forget to lock your door one day.

Once he is inside your home he is free to wreak havoc, perhaps making it obvious he has been there by stealing or wrecking things, or perhaps simply taking copies of any keys he finds so he can come and go later at his leisure. Whatever happens, you don't want your first knowledge of the break-in to be when you return home to the ransacked contents.

That is why many people install a burglar alarm as well. Should the intruder gain access through the perimeter defenses, the burglar alarm alerts you or your neighbors to the break-in immediately, and provides an additional deterrent to the would-be thieves.

IDS, therefore, are the equivalent of the burglar alarm. To be used alongside firewalls, they are a recognition of the fact that you can never have a 100% secure system. However, should someone be clever enough to breach your perimeter defenses, you want to know about it as soon as possible. It would also be nice to know what they have been up to while they were inside too.

Intrusion Detection and Vulnerability Assessment are becoming increasingly important as the stakes become higher. In the 1980s and early 1990s, denial-of-service (DoS) attacks were infrequent and not considered serious. Today, successful DoS attacks can shut down e-commerce-based organizations like online stockbrokers and retail sites.

Within the IDS marketplace there are three broad categories of product:

• Host IDS (HIDS)
  These employ an agent that resides on each host to be monitored. The agent scrutinizes event logs, critical system files, and other auditable resources looking for unauthorized changes or suspicious patterns of activity. Whenever anything out of the ordinary is noticed, alerts or SNMP traps are raised automatically.
  
  For instance, they will monitor attempted logins and take note of when an attempt is made to access an account with an incorrect password. If the attempt fails too many times within a short time span the system may conclude that someone is trying to gain access illegally and an alarm can be raised.
  
  Another thing they can do is monitor the state of system and application files, or the Windows Registry. They do this by making an initial pass of a clean system and storing a condensed "snapshot" of how that system should look. If an intruder—or some sort of Trojan Horse—does manage to gain access to the system and make changes, the IDS will spot this (maybe not in real time) and raise an alert. There are systems on the market that specialize in this type of operation, and they tend to be referred to as File Integrity Assessment (FIA) products.
  
  Most host-based systems tend to be reactive rather than proactive—that is they often have to wait until something has actually happened before they can raise the alarm. Some, however, attempt to be proactive, monitoring and intercepting system calls to the kernel or APIs in order to prevent attacks as well as log them. They may also monitor data streams and the environment specific to a particular application (file locations and registry settings for a Web server, for example) in order to protect that application from generic attacks for which no "signature" yet exists.
  
  These products sometimes go by the description "intrusion prevention", since their aim is to stop intrusions dead, rather than simply report on them as, or after, they occur.
  
  
• Network IDS (NIDS)
  These monitor traffic on the wire in real time, examining packets in detail in order to spot denial of service attacks or dangerous payload before the packets reach their destination and do the damage. They do this by matching one or more packets against a database of known "attack signatures", or performing protocol decodes to detect anomalies, or both. These signature databases are updated regularly by the vendors as new attacks are discovered.
  
  When suspicious activity is noticed, a network-based scanner is capable of both raising alerts and terminating the offending connection immediately (as are some host-based scanners). Some will also integrate with your firewall, automatically defining new rules to shut out the attacker in future.
  
  Most of the network-based IDS available to date work in what is known as "promiscuous mode". This means that they examine every packet on the local segment, whether or not those packets are destined for the IDS machine (much like a network monitor, such as Sniffer). Given that they have a lot of work to do in examining every single packet, they usually require a dedicated host on which to run due to their heavy use of system resources.
  
  For instance, most attacks are not based on the contents of a single packet, but are made up of several, sometimes sent over a lengthy period of time. This means that the IDS has to store a number of packets in an internal buffer in order to track established sessions and compare groups of packets I with its attack signature database. This is known as "maintaining state", and allows IDS to compare new packets against its signature database in the context of what has happened previously in a particular session, rather than examining every packet in isolation.
  
  You will also need one Network IDS sensor per segment, since they are unable to see across switches or routers, and some have problems keeping up with heavily loaded Fast Ethernet segments (never mind Gigabit).
  
  
• Network Node IDS (NNIDs)
  Recently we have seen a new type of "hybrid" IDS agent appear which overcomes some of the limitations of the network-based IDS.
  
  This new agent works in a similar manner to the network-based IDS in that it takes packets from the wire and compares them against database entries. However, this new "micro agent" is only concerned with packets targeted at the network node on which it resides, thus giving rise to the term Network Node IDS (NNIDS) (also sometimes referred to as a Stack-based IDS).
  
  Rather confusingly, it is also occasionally referred to as "host-based", but usually only by those who are looking at the industry purely from a Network IDS viewpoint. For the purposes of this article, Host IDS is concerned with monitoring of log files and behavioral analysis, whereas both Network and Network Node IDS are concerned with TCP analysis—the only difference is that one (NIDS) is running in promiscuous mode whilst the other (NNIDS) is not.
  
  The fact that the NNIDS system is no longer expected to examine every single packet on the wire, however, means that it can be much faster and take less system resources, and this allows it to be installed on existing servers without imposing too much overhead. It also makes it particularly suitable for heavily loaded segments, switched network environments, or VPN implementations with encrypted traffic on the wire, all areas where traditional network-based IDS have problems.
  
  Obviously it is necessary to install a number of these NNIDS agents—one for every server to be protected—and they will all have to report back to a central console.
  
  Many organizations may opt for a combination of the two—NNIDS on individual servers in switched server farms, and traditional NIDS on less heavily used segments, where a single IDS can protect a large number of hosts.

For those with a reasonable security budget, we would recommend purchasing a firewall and at least one product from each of the aforementioned categories. The firewall guards your perimeter, whilst the IDS' monitor what is happening on your network, guarding against slip-ups by the firewall as well as internal mischief-makers.

IDS devices can also be installed outside a firewall in order to detect the "doorknob rattlers", attempted scans and other probes and attacks that are normally dealt with by your firewall and thus would not be detected by an internal IDS. This is purely a management issue, and depends on the human resources available to scan the resulting log files—those from an IDS installed outside a firewall are likely to be voluminous to say the least.

Both host-based and network-based scanners are worth investing in, since they each have their own strengths. Network-based IDS will monitor the wire for suspect packets and are adept at spotting Denial of Service type attacks and unwelcome probes—usually from outside the network. Host-based systems, on the other hand, are watching the "crown jewels"—the actual data on the file servers, monitoring for inappropriate logins or changes to critical files from unauthorized sources.

Although network-based products seem to get most of the publicity at the moment, given that the FBI figures still point to over 70% of hack attacks coming from inside a network, the host-based system can sometimes be more valuable. We would still recommend deploying both technologies wherever possible.

To begin with, deployment of IDS requires careful thought and planning if you are to get the most from it. What kind of resources are you protecting? If you have a DMZ containing only Unix-based Web servers, you could disable all IDS signatures to do with Windows NT or DNS servers. Some IDS products will go so far as to try and perform this step automatically for you depending on what operating systems and services are found during a network scan.

To do the job properly, however, you should acquire some good Vulnerability Assessment tools and scan your own network. This should provide a good starting point in determining which machines need protecting, and from what sort of attacks.

Are you worried about intruders breaking through your firewall and launching DoS attacks against your machines? Then perhaps a Network or Network Node IDS would be a good buy. Or are you more concerned about Trojans on your desktops and file servers? File Integrity Assessment would be the best bet. Or perhaps you are worried about your own users making off with company secrets? In which case, Host IDS would be the right choice.

If attacks against your Web servers are your chief concern, however, intrusion prevention systems might be more appropriate. Most likely you will actually need a combination of all these technologies.

Having selected your products, where are you going to install them? Host and Network Node IDS are simple—you put them on the hosts that need protecting. But Network IDS is another matter. Placing a network interface card into promiscuous mode generates an awful lot of material to be processed. All the traffic passing by on the subnet being monitored has to be collected and examined by the IDS sensor and compared against a database of known attack signatures.

Nor is it enough to simply look at this traffic on a packet by packet basis, since some attacks are spread over several packets, or may be fragmented deliberately to confuse the IDS. This necessitates a time- and memory-consuming process of buffering packets and maintaining state tables to keep track of individual sessions. To defeat the common IDS evasion techniques such as packet fragmentation, out-of-order fragments, and so on, it is also necessary to perform some heavy-duty processing within the IDS itself to reassemble packets back into the form that will eventually be seen by the target host (or capture the packets at a higher level in the TCP stack of the sensor machine). Or perhaps a full protocol decode is performed. All of these options are likely to slow the detection process, but only then can the true packet contents be determined and compared against the signature database.

And all this must be done whilst eliminating—or at least keeping to a minimum—the risk of the "false positive". In an ambiguous situation, there is no point in raising an alert "just to be on the safe side", otherwise the IDS runs the risk of crying wolf once too often and eventually being ignored altogether. Quite a bit is involved behind the scenes then, and whilst most IDS products could quite happily keep up with traffic on a 10 Mbit network, leased-line, or DSL connection, it is much more difficult to do all this at wire speed on a 100 Mbit network.

Gigabit presents even more problems. Not only are we now looking at a significant increase in bandwidth—and thus a significant increase in the volume of traffic to be analyzed—but we have also moved into the realms of the purely switched network. Don't forget that the promiscuous mode sensor can only see traffic on its own segment, and in a switched environment, every connection to the switch is, effectively, a single segment. In the 10 Mbps or 100 Mbps world, this can be overcome by the use of network taps or mirroring all the switch traffic to a span port, to which is attached an IDS sensor.

But with Gigabit, the result would be a seriously overloaded sensor. Building IDS technology into the switch hardware itself, allowing the sensor to grab traffic directly from the backplane, may be one solution. Another is to move to a pure Network Node IDS implementation, where the agents are concerned only with the traffic directed at the host on which they are installed.

Companies that make extensive use of VPNs will also find problems with Network IDS, since the traffic picked from the wire will obviously be encrypted, thus rendering any pattern matching or protocol analysis completely useless.

If it is not acceptable to decrypt data at the network border, then once again the only solution is to install Network Node IDS, so that the IDS agent has access to the data stream as it is decrypted at the VPN end-point.

Going forward, then, Network Node IDS is likely to be the favored technology in high-speed, switched networks, or networks that carry a lot of encrypted traffic. Does that mean that Network IDS is a redundant technology?

Not at all. With careful selection of the appropriate product, NIDS still provides the most cost-effective solution for intrusion detection on unencrypted networks—both switched (using taps or span ports) and shared—up to 100 Mbps. And Gigabit products are now appearing which may take us beyond even that level.

Although this may seem like using a sledgehammer to crack a nut, it does have the advantage of highlighting anomalies in packet contents much more quickly than doing an exhaustive search of a signature database. It also has the advantage of greater flexibility in capturing attacks that would be very difficult—if not impossible—to catch using pure pattern-matching techniques, as well as new variations of old attacks. These are attacks which—although changing only slightly from variant to variant—would normally require a new signature in the database for the "traditional" IDS architecture, but which would be detected automatically by a complete protocol analysis.

Consider how an IDS might detect RPC exploits. A pattern-search system looks for patterns on ranges of ports where RPC programs typically run—for example, examining ports in the range 634 through 1400 for the AMD exploit. In contrast, a state-based system can "remember" the ports on which the AMD service is running, and only test the AMD signatures on those specific ports. If no system on the network is running AMD, then a state-based system will never test network traffic for those signatures.

The theory behind this is that a pattern-search system has no knowledge of the contents of the packets, and must search each packet for many different patterns. In contrast, a protocol-analysis system knows the contents of the packet, and only tests signatures that apply to those contents. Although this may sound like a huge advantage, it should be remembered that full protocol decodes take significantly more processing power than simple pattern matching, rendering the actual differences in performance less than one might imagine.

The second part of the theory is that for pattern-search systems, a continual increase in the number of signatures will have an adverse effect on performance. Some vendors of pattern-matching IDS sensors recommend pruning the signature database of "unwanted" signatures in order to improve performance—but how do you determine which signatures are unwanted? This should not be an issue for a stateful lDS architecture.

Another example of the advantage of protocol decodes concerns searching Telnet login strings for one of the many well-known login names that rootkits tend to leave behind on the system. A pattern-search system might scan all Telnet traffic for all these patterns, in which case the more patterns you add, the slower it becomes.

In contrast, a protocol-analysis system will decode the Telnet protocol and extract the login name. It can then perform an efficient search in a binary-search tree or a hash table for that login name, which should scale much better as new signatures are added.

In theory, therefore, state-based protocol-analysis should offer more efficient processing of traffic and improved scalability as more signatures are added, compared to a pure pattern-matching solution. In reality, pattern-matching solutions rarely opt for a "brute force" approach, and so the differences are not always as marked as the marketing people would like us to believe.

Note also, that a protocol analysis IDS can only go so far with its protocol decodes before it too will be forced to perform some kind of pattern matching, albeit against a theoretically smaller subset of "signatures". Beware the marketing hype in this particular area—no matter what architecture is used, the performance figures in a live deployment will speak for themselves.

Yet another approach is to forget about trying to identify the attacks directly, and concentrate instead on ignoring everything that is considered "normal". This is known as "anomaly-based"—as opposed to "signature-based"—IDS, and the basic principle is that, having identified what could be considered "normal" traffic on a network, then anything that falls outside those bounds could be considered an "intrusion"—or at the very least, something worthy of note.

The primary strength of anomaly detection is its ability to recognize previously unseen attacks, since it is no longer concerned with knowing what an attack looks like—merely with knowing what does not constitute normal traffic. Its drawbacks, of course, include the necessity of training the system to separate noise from natural changes in normal network traffic (the installation of a new—perfectly legitimate—application somewhere on the network, for example).

Changes in standard operations may cause false alarms while intrusive activities that appear to be normal may cause missed detections. It is also difficult for these systems to name types of attacks, and this technology has a long way to go before it could be considered ready for "prime time".

IDS is a valuable component of an organization's security plan, but it is just that—a component. The first point of defense may well be the firewall, and behind the Network and Network Node IDS system may well be additional port monitoring and File Integrity Assessment products to alert you as to when an intrusion attempt has been successful.

All of these components must operate within the confines of a strict security policy, which should determine what is and is not allowed on the corporate network. This, in turn, will help specify how the individual components are to be deployed and configured, as well as offer guidelines as to how alerts are to be handled. There is no point in using the latest IDS technology only to have it log intrusion attempts to a file that is only examined once a month.

There should be differing levels of importance assigned to different types of intrusion attempts, and the alert and response procedure should be scaled accordingly. There is little point in raising the roof should you discover your network is being port scanned by someone using nmap—that happens all the time, and it is enough to log that for periodic examination just to make sure it is not happening too often within a set time interval.

On the other hand, a successful BackOrifice ping that elicits a reply from somewhere within your organization indicates a serious compromise, and for that you may deem it appropriate to page the security administrator any time day or night. Between those two extremes are various other possible responses such as e-mail, SNMP alerts, session termination, firewall reconfiguration, and so on—use them to the full, and make sure that your security staff takes the time to examine the various log files at regular intervals to keep tabs on the more mundane intrusion attempts.

Intrusion Detection Systems are good at sounding alarms, but unless there is someone around who is prepared—and trained—to respond, it is no better than a car alarm that everyone ignores. An effective response is every bit as important as detecting the attack in the first place.

Maintenance is equally important. Security is certainly not static, and new vulnerabilities are being discovered and exploited all the time. This should result in new signatures for your IDS and VA tools, patches for your operating systems, and updates for your firewalls. Even so, it is a wise security administrator that keeps an eye on the various underground hacking sites and security alert mailing lists for himself. Between the point of discovery to the point where a patch is issued and applied, there is a window of opportunity for the hacker to exploit. It is up to the security administrator to minimize this window as much as possible.

If an OS vendor is slow in bringing out a patch for a new vulnerability, for example, perhaps the administrator can reconfigure the corporate firewall to eliminate the sort of traffic that could exploit that vulnerability, or add a temporary custom signature to the IDS in order to detect it. Certainly as soon as new IDS signatures are made available from the vendor, they should be downloaded and deployed to every appropriate sensor on the network.

Finally, the administrator should be using Vulnerability Assessment tools on a regular basis (or employing outside consultants to do the same) in order to continually test defenses and update security policies accordingly. A VA scan may well highlight additions or changes to the network and its applications which might necessitate a rethink on how IDS sensors are deployed and which signatures are monitored by each sensor.

Monitor, evaluate, modify. Then back to the beginning. It is a cycle that must be repeated over and over if you want to keep your network as secure as possible. Only by continual vigilance and refinement will you stay one step ahead of (or at least no more than one step behind) the hackers.

Bob Walder, a leading authority on network security, is one of the founders of The NSS Group. Since leaving behind the world of IT management in 1991, Bob has been at the cutting edge of new technology and has invested much of his time in advising on, testing and certifying a range of security products on behalf of end user organisations, vendors and certification bodies. He is also a regular contributor of technical articles to the major networking and security titles.

The NSS Group is Europe’s foremost independent network and security testing facility. With labs in Cambridge in the UK and Carcasonne in the south of France, The NSS Group offers a range of specialist networking and security services to vendors and end user organisations throughout Europe and the United States. For more information, visit www.nss.co.uk or e-mail info@nss.co.uk.