Hierarchical State Machine Architecture for Regular Expression Pattern Matching Cheng-Hung Lin
Hsien-Sheng Hsiao
National Taiwan Normal University, Taipei, Taiwan
National Taiwan Normal University, Taipei, Taiwan
[email protected]
[email protected] patterns. Take a regular expression “.*AB.*CD” for example where the metacharacters “.*” denotes zero or more instances of any character. Figure 1 shows the DFA to recognize the regular expression pattern “.*AB.*CD”, where State 0 is the initial state, and state 4 is the final state indicating the matching of the regular expression pattern.
ABSTRACT Regular expression has been widely used in network intrusion detection system to represent attack patterns due to its expressive power and flexibility. However, the traditional memory architecture suffers from the problem of memory explosion for certain types of complex regular expressions. In this paper, we propose a hierarchical state machine architecture which can significantly reduce the memory required to accommodate complex regular expression patterns. The experiments demonstrate a significant reduction in memory for the complex regular expression patterns commonly used in network intrusion detection systems.
[^A]
[^C]
A B
1
0
C C
2
D
3
[^CD]
4
C
[^AB]
[^C] Figure 1: Deterministic finite automaton of “.*AB.*CD”
Categories and Subject Descriptors C.2.0 [Computer Communication Networks]: General-Security and protection (e.g., firewalls)
It’s well known that when m patterns are compiled into a single DFA, the matching time complexity for matching m patterns against the input texts of length n is reduced from O(mn) to O(n). Therefore, compiling multiple regular expression patterns into a single composite DFA is an efficient way to match multiple regular expression patterns. However, there are specific regular expression patterns which cause the composite DFA to grow exponentially. For example, using Flex[18] for compilation, the two patterns in Linux L7-filter, “.*membername.*session.*player” and “.*rdpdr.*cliprdr.*rdpsnd” contain 47 and 43 states, respectively. When compiling these two patterns into a composite DFA, the number of states is up to 232. The increasing number of states come form the multiple sets of metacharacters “.*” in these patterns. Figure 2 illustrates the reason why compiling such regular expression patterns into a composite DFA causes the corresponding state machine to grow exponentially. It shows the composite DFA for matching the regular expression patterns, “.*AB.*CD” and “.*MN.*PQ” where partial edges are omitted for easy description. In Figure 2, states 0, 1, 2, 3, and 4 belong to the original state machine of “.*AB.*CD” and states 5, 6, 7, and 8 belong to the original state machine of “.*MN.*PQ”.
General Terms: Algorithms, Design, Security Keywords: Pattern Matching, Regular Expression, State Machine
1. INTRODUCTION In a signature-based network intrusion detection system (NIDS), the input packets are matched against thousands of attack patterns to identify malicious packets. To efficiently represent attack patterns, regular expressions has been widely adopted in NIDS systems such as Snort[1], Bro[2], and Linux L7-filter[3] because of its better expressive power and flexibility, when compared with explicit string representation. Because the regular expression matching is the most computative task in NIDS systems, many hardware approaches are proposed to accelerate regular expression matching. The hardware approaches may be classified into two main categories, the logic [4,5,6,7] and memory [8,9,10,11,12,13] architectures. Basically, the logic architecture is based on Nondeterministic Finite Automaton (NFA) while the memory architecture is based on Deterministic Finite Automaton (DFA). From the perspectives of re-configurability and scalability, memory architectures are attractive because memory architecture is flexible and scalable. The memory architecture for matching regular expression pattern works as follows. First, regular expression patterns are compiled into a Deterministic Finite Automaton (DFA). Then, the corresponding state transition table is stored in memory to recognize regular expression
[^MC]
[^AM] 0
A
1
B
C
2
D
3
M 9 M
N
11
P
12 N
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GLSVLSI’09, May 10–12, 2009, Boston, Massachusetts, USA. Copyright 2009 ACM 978-1-60558-522-2/09/05…$5.00.
D C
10
B 5
4
C
13
A
Q 6
P
7
Q
8
[^AP] Figure 2: Deterministic finite automaton of “.*AB.*CD” and “.*MN.*PQ”
133
The additional states 9, 10, 11, 12, and 13 come from the correlation of two patterns. We observe that the additional states are determined according to the order and the lengths of four fragments, “AB”, “CD”, “MN”, and “PQ”. In other words, the additional states are duplicated from the sub-patterns segmented by the metacharacters “.*”. With the increasing number of such regular expression patterns to be compiled, the additional states will increase tremendously. Because the memory requirement is proportional to the state number, reducing the state number can efficiently reduce the memory requirement and, therefore the power consumption and cost. In this paper, we propose a hierarchical state machine architecture composed of two state machines, called master and slave, to match these complex regular expression patterns. The master state machine is used to match against the segmented sub-patterns while the slave state machine is used to identify the correlation of the segmented sub-patterns. Compared to the traditional DFA, the hierarchical state machine architecture can significantly reduce the number of states and therefore the memory size. Experimental results show that the hierarchical state machine architecture achieves an average of 95% of state reduction and 97% of memory reduction for the complicated regular expression patterns in Snort, Bro, and L7-filter systems.
R1 R1’
with one “.*” in each pattern needs O( ( ∑ ni ) ⋅ 2 i =0
β α
γ δ interε θ result
Rm Rm’
Master State Machine
ε
γ
δ θ
match vector
Slaver State Machine
Figure 3: Hierarchical State Machine Architecture We now illustrate the hierarchical state machine architecture using an example. Considering the following two patterns, “.*AB.*CD” and “.*MN.*PQ”, we first construct a master state machine to match input characters to the segmented sub-patterns, “AB”, “CD”, “MN”, and “PQ”, as shown in Figure 4(a) where partial edges are omitted for easy description. In Figure 4(a), the outputs, called inter-results, of final states 2, 4, 6, and 8 are represented asα, β, γ, and δ, respectively. Except for final states, the outputs are 0. Then, we construct a slave state machine to match the inter-results to the new regular expression “.* α.* β” and “.* γ.* δ” as shown in Figure 4(b).
In this paper, the complex regular expression pattern denotes the regular expression pattern with multiple sets of metacharacters “.*” such as the two patterns in L7-filter, “.*membername.*session.*player” and “.*rdpdr.*cliprdr.*rdpsnd”. Snort and Bro have similar complex regular expressions. As described in the introduction, when compiling multiple complex regular expression patterns into a composite DFA, the correlation of those segmented sub-patterns causes the state machine to grow exponentially. Basically, compiling m patterns of length ni (i=0…m) m
R2 R2’
input streams
2. COMPLEX REGULAR EXPRESSION
m
α β
[^ACMP] 0
0 1 0 3 0 5 0 7
A C M
) states.
P
Furthermore, compiling m patterns of length ni (i=0…m) with x
B
2
D
4
N
6
Q
8
α β γ δ
m
instances of “.*” in each pattern needs O( ( ∑ ni ) ⋅ (m + x) m ) states.
(a) Master State Machine
i =0
In other words, with the increasing number of metacharacters “.*” and the number of patterns, the composite state machine will grow exponentially due to the duplication of segmented sub-patterns. Therefore, if we can reduce the duplication of segmented sub-patterns, we can reduce the number of additional states and therefore the memory size.
0
0 0
α
γ
3. HIERARCHICAL STATE MACHINE In this section, we propose a hierarchical state machine architecture to handle such complex regular expression patterns. Our basic idea is that if we can use two concurrent state machines, where one state machine is for matching the segmented sub-patterns, while the other state machine is for identifying the correlation of the segmented sub-patterns, then we can efficiently reduce the additional states. For m regular expressions, “.*R1.*R1’”, “.*R2.*R2’”,…, and “.*Rm.*Rm’”, Figure 3 shows the hierarchical state machine architecture which are composed of two parallel state machines, master and slave state machines. The master state machine is used to match input streams to the segmented sub-patterns, such as R1, R1’, R2, R2’,…Rm and Rm’. Then, the matching results, α,β,γ,δ,…,ε, andθ of the master state machine, called inter-result is fed to the slave state machine which is used to identify the correlation of the segmented sub-patterns and to output final results, called match vector.
1 γ α 3
β
2
0 5 δ
β δ 4
0 (b) Slave State Machine Figure 4: Master and Slave State Machine Consider the example in Figure 5 which matches the input string “ABMNCD” to the regular expressions “.*AB.*CD” and “.*MN.*PQ”. First, as the master state machine reads “AB”, we can see that the master state machine moves from state 0 to state 2 and outputs “α”. At the same time, the slave state machine reads the inter-results “00α” and moves from state 0 to state 1. Then, the master state machine reads “MN”, and moves from state 2 to state 6 and outputs “γ”. Concurrently, the slave state machine reads “0γ”
134
and moves from state 1 to state 5. Finally, the master state machine reads “CD”, moves from state 6 to state 4, and outputs “β”. On the other hand, the slave state machine reads “0β”, and moves from state 5 to state 2 which represents the final state of the pattern “.*AB.*CD”.
In the following theorem, we show a necessary condition for the exception problem. Therefore, we can safely merge the complex regular expressions without the exception problem if the necessary condition is not satisfied. Consider a hierarchical state machine architecture which merges m regular expressions, “.*R1.*R1’”, “.*R2.*R2’”,..., and “.*Rm.*Rm’”.
Input character: A B M N C D State of master: 0 Æ 1 Æ 2 Æ 5 Æ 6 Æ 3 Æ 4 Inter-results: 0 0 α 0 γ 0 β State of slave: 0Æ 0Æ 0 Æ 1 Æ 1 Æ 5 Æ 5 Æ 2
Definition: A sub-pattern, R’ is called an overlapped-expression if R’ both matches the suffixes of R1, R2, …, or Rm and the prefixes of R1’, R2’, …, or Rm’.
Figure 5: State transitions of matching “ABMNCD”
Consider the same example in Figure 6, R1 and R1’ denote “ABC” and “BCD”, respectively while R2 and R2’ denote “MNO” and “NOP”, respectively. The sub-pattern “BC” both matches the suffixes of R1 and the prefixes of R1’. Therefore, the sub-pattern “BC” is an overlapped-pattern. Similarly, the sub-pattern “NO” is also an overlapped-pattern.
4. EXCEPTION PROBLEMS Although the hierarchical state machine architecture can effectively reduce the memory requirements of merging such complex regular expression patterns, it creates a new problem, called exception problem when merging certain types of complicated regular expressions. The exception problem may lead to false positive results. We now illustrate the exception problem using an example. Figure 6 shows the master state machine of two patterns, “.*ABC.*BCD”, “.*MNO.*NOP”. Consider an input string containing a substring “ABCDE”. When “ABC” is fed, the master state machine moves to state 3 and outputs “α” which causes the slave state machine to move to state 1. When the next “D” is fed, the master state machine moves to state 6 and outputs “β”. Then, the slave state machine moves to state 2 which represents the final state of the pattern “.*ABC.*BCD”. Actually, the matching result is a false positive because the input string “ABCD” shouldn’t match the pattern “.*ABC.*BCD”. We can see that the input “BC” both matches the suffixes of pattern “ABC” and the prefixes of pattern “BCD,” thus the exception problem arises.
[^AC] A 0 B M N
α 3 0 0 β C D 5 4 6 0 0 γ N O 8 7 9 0 0 O P δ 11 10 12 0 1
0 2
B
Theorem: The hierarchical state machine architecture does not have the exception problem if there is not any overlapped-pattern both matches the suffixes of R1, R2, …, or Rm and the prefixes of R1’, R2’, …, or Rm’. Proof: The exception problem arises when there exists an overlapped-pattern. Suppose an input string contains an overlapped-pattern of R1 and R1’, the master state machine matching R1 also matches the prefixes of R1’. If the R1’ is matched successively by the master state machine without moving back to ■ initial state. The exception problem arises.
5. EXPERIMENTAL RESULTS We implement the hierarchical state machine architecture on the regular expression patterns of Snort, Bro, and L7-filter. The results are compared with the traditional DFA approach, as shown in Table 1. Columns one and two show the rule set and the number of complex regular expression patterns. Columns three and four show the state number and the memory size of the traditional DFA. Columns five and six show the state number and memory requirements of the master state machine. Columns seven and eight show the state number and memory requirements of the slave state machine. For example, in the first row of the Table 1, the Snort rule sets contain 71 complex regular expressions. Constructing a composite DFA needs 211,438 states and 153,321,472 bytes. Applying our hierarchical state machine architecture, the master state machine needs 2,914 states and 1,694,720 bytes while the slave state machine needs 2,724 states and 1,498,624 bytes. The state and memory reductions are 97% and 98%, respectively. Furthermore, when applied to Bro and Linux L7-filter, we get 99% and 95% memory reduction, respectively. The results show that the hierarchical state machine architecture is very efficient in memory reduction for compiling multiple complex regular expression patterns.
C
D
P
(a) Master State Machine
0 0
α
γ
0 1 γ α 3
β
2
0 5 δ
β δ 4
0 (b) Slave State Machine Figure 6: Example of Exception Problem
135
Table 1: Experimental results on Snort, Bro and L7-filter rule sets Traditional DFA Rule set
# of patterns
SNORT
Hierarchical state machine architecture Master state machine Slave state machine state memory # of memory reduction # of states (bytes) states (bytes)
# of states
memory (bytes)
71
211,438
153,321,472
2,914
1,694,720
2,724
1,498,624
97%
98%
BRO
88
300,498
324,534,272
3,367
1,731,072
4,051
1,943,552
98%
99%
L7-FILTER
11
9,614
7,362,560
758
335,616
118
7,120
91%
95%
Total
170
521,550
485,218,304
7,039
3,761,408
6893
3,449,296
95%
97%
1
1
1.35%
0.78%
1.32%
0.71%
AVG.
6. CONCLUSIONS Because there are specific regular expression patterns in NIDS which cause the memory requirement to grow exponentially, reducing the memory requirement of such complicated patterns has become imperative. In this paper, we have presented a hierarchical state machine architecture to accommodate complex regular expression patterns. By using two parallel state machines, the hierarchical state machine architecture achieves significant improvements on state and memory reductions while maintaining the same matching time complexity. In addition, as the hierarchical state machine architecture is based on general memory architecture, it can also be applied to processing simple regular expression patterns, such as exact string patterns. The experiments demonstrate a significant reduction in memory for the complex regular expression patterns commonly used in NIDS systems.
[8]
M. Aldwairi*, T. Conte, and P. Franzon, “Configurable String Matching Hardware for Speeding up Intrusion Detection,” in ACM SIGARCH Computer Architecture News, 2005, pp. 99–107
[9]
S. Dharmapurikar and J. Lockwood, “Fast and Scalable Pattern Matching for Content Filtering,” in Proc. of Symp. Architectures Netw. Commun. Syst. (ANCS), 2005, pp. 183-192
[10] Y. H. Cho and W. H. Mangione-Smith, “A Pattern Matching
Co-processor for Network Security,” in Proc. 42nd Des. Autom. Conf. (DAC), 2005, pp. 234-239 [11] L. Tan and T. Sherwood, “A high throughput string matching
architecture for intrusion detection and prevention,” in 32nd Ann. Int. Symp. on Comp. Architecture, (ISCA), 2005, pp. 112-122 [12] H. J. Jung, Z. K. Baker, and V. K. Prasanna, “Performance of
FPGA Implementation of Bit-split Architecture for Intrusion Detection Systems,” in 20th Int. Parallel and Distributed Processing Symp. (IPDPS), 2006.
References [1]
Snort official website, “http://www.snort.org/”
[2]
Bro official website, “http://www.bro-ids.org/”
[3]
Linux L7-filter official website, “http://l7-filter.sourceforge.net/”
[4]
R. Sidhu and V. K. Prasanna, “Fast regular expression matching using FPGAs,” in Proc. 9th Ann. IEEE Symp. Field-Program. Custom Comput. Mach. (FCCM), 2001, pp. 227-238.
[5]
Memory reduction
[13]
F. Yu, Z. Chen, Y.Diao, T.V. Lakshman, and R.H. Katz, “Fast and Memory-Efficient Regular Expression Matching for Deep packet Inspection,” in Proc. ACM/IEEE Symp. Architectures Netw. Commun. Syst. (ANCS), 2006, pp. 93-102
[14]
A. V. Aho and M. J. Corasick. “Efficient String Matching: An Aid to Bibliographic Search,” in Communications of the ACM, 18(6):333–340, 1975.
B.L. Hutchings, R. Franklin, and D. Carver, “Assisting Network Intrusion Detection with Reconfigurable Hardware,” in Proc.10th Ann. IEEE Symp. Field-Program. Custom Comput. Mach. (FCCM), 2002, pp. 111-120.
[15]
B. Brodie, R. Cytron, D.Taylor, “A Scalable Architecture for High-throughput Regular Expression Matching,” in Proc. 33rd Int’l Symposium on Computer Architecture (ISCA), 2006, pp191-202.
[6]
C. R. Clark and D. E. Schimmel, “Scalable Pattern Matching for High Speed Networks,” in Proc. 12th Ann. IEEE Symp. Field-Program. Custom Comput. Mach. (FCCM), 2004, pp. 249-257
[16]
[7]
J. Moscola, J. Lockwood, R. P. Loui, and M. Pachos, “Implementation of a Content-Scanning Module for an Internet Firewall,” in Proc. 11th Ann. IEEE Symp. Field-Program. Custom Comput. Mach. (FCCM), 2003, pp. 31–38.
S. Kumar, S.Dharmapurikar, F.Yu, P. Crowley, and J. Turner, “Algorithms to Accelerate Multiple Regular Expressions Matching for Deep Packet Inspection,” in ACM SIGCOMM Computer Communication Review, ACM Press, vol.36, Issue. 4, Oct. 2006, pp. 339-350.
[17]
P. Piyachon and Y. Luo, “Compact State Machines for High Performance Pattern Matching,” in Proc. of Design Automation Conference, (DAC), 2007, pp.493-496
[18]
Flex, The Fast Lexical Analyzer, http://flex.sourceforge.net/
136
