DISTRIBUTED SPAM FILTERING TEAM BROKEN SHACKELS MEMBERS ANAND KULKARNI MANAN DOSHI GUIDE PROF.ALAN KAMINSY

INDEX TITLE

PAGE NO.

INTRODUCTION

3

IP ADDRESS BASED SPAM CLASSIFIER

4

ANALYSIS OF RESEARCH PAPERS

5

DEVELOPERS MANUAL

15

USERS MANUAL

17

DESIGN

19

SCREEN SHOTS

24

FUTURE ENHANCEMENTS

26

TEST CASES

27

INVESTIGATIONS

29

DISTRIBUTED SPAM FILTERING We plan to implement a prototype of a content independent , distributed spam classifier that classifies spam based on the origin of the mail ( ip address ) and also on the time stamps on which the mail was sent. The design tries to block the spam which has been generated in a very short span of time ( This phenomena has been observed in most of the spam that gets generated by bot-nets) We investigate the performance of this design on client-server architecture and on p2p architecture and compare their performances. Our design can be implemented over exisiting email system without any special hardware requirements or without any changes to the underlying protocols.

REASON FOR IP ADDRESS BASED SPAM CLASSIFYING ALONG WITH TIME THRESHOLDS -

Researchers have made the observation that in order to be effective, bots must send a relatively large number of e-mails in a short amount of time. This stems from the fact that most computers in a botnet are only up for a short period each day; in the evenings when the average user comes home and turns on their machine. Consequently, if an e-mail is received from an “unknown” source that has sent many e-mails in a short period of time, then the likelihood of this being spam is high.

-

Large percentages of spam originate in well defined areas of the globe.

-

Various spam black lists available on the Internet such as : 1) The SBL is a realtime database of IP addresses of verified spam sources maintained by the Spamhaus Project team and supplied as a free service to help email administrators better manage incoming email streams. 2) The Mail Abuse Prevention System (MAPS) is an organisation that provides anti-spam support by maintaining a DNSBL. They provide five black lists, categorising why an address or an IP block is listed.

ANALYSIS OF RESEARCH PAPERS ANALYSIS OF RESEARCH PAPER 1 TITLE : An Effective Defense Against Email Spam Laundering AUTHORS : Mengjun Xie, Heng Yin, Haining Wang Department of Computer Science The College of William and Mary, Williamsburg, VA 23187 {mjxie, hyin, hnw}@cs.wm.edu

In this paper, one salient characteristic of proxy-based spamming activities, namely packet symmetry is revealed by analyzing protocol semantics and timing causality. Based on the packet symmetry exhibited in spam laundering, a simple and effective technique, DBSpam is proposed to on-line detect and break spam laundering activities inside a customer network. Monitoring the bi-directional traffic passing through a network gateway, DBSpam utilizes a simple statistical method, Sequential Probability Ratio Test, to detect the occurrence of spam laundering in a timely manner. DBSpam is designed to be placed at a network vantage point such as the edge router or gateway that connects a customer network to the Internet

At present, proxies such as off-the-shelf SOCKS and HTTP proxies play an important role in the spam epidemic. Spammers launder email spam through these proxies to conceal their real identities and reduce spamming cost. The number of available spam proxies and bots in August 2006 is more than 3,200,000. Such numerous spam proxies facilitate the formation of email spam laundering, by which a spammer has great flexibility to change spam paths and bypass anti spam barriers. Due to the protocol semantics of SMTP and timing causality, the behavior of proxy-based spamming demonstrates the unique characteristics of connection correlation and packet symmetry. Utilizing this distinctive spam laundering behavior, the suspicious TCP connections involved in spam laundering can be identified.

ANTI-SPAM TECHNIQUES -Recipient-oriented Techniques Content-based Techniques : The techniques in this sub-category detect and filter spam by analyzing the content of received messages, including both message header and message body. The various techniques are : Email address filters, Heuristic filters, Machine learning based filters. Non-content-based Techniques : The techniques in this sub-category use non-content spam characteristics, such as source IP address, message sending rate, and violation of SMTP standards, to detect email spam. The various techniques include DNSBLs , MARID (MTA Authorization Records In DNS), Challenge-Response (C-R), Tempfailing ,Sender Behavior Analysis. Sender-oriented Techniques This group of techniques include : Usage Regulation , Cost-based approaches.

HoneySpam is a specialized honeypot framework to deter email address harvesters, poison spam address databases, and intercept or block spam traffic that goes through the open relay/proxy decoys set by HoneySpam. With the network virtualization offered by honeyd, HoneySpam can set up multiple fake web servers, open proxies, and open relays

PROXY BASED SPAMMING There is a customer network N, in which spam proxies reside. Both spammer S and receiving MTA M are connected to customer network N via edge router R. S may be the original spam source or just another spam proxy (but it must be closer to the real spam source). M is the outside MTA. The loosely managed sub-networks in the de-militarised zone becomes the monitored network N and the router/gateway connecting the subnetworks becomes the vantage point R. Thus, the assumption of exterior MTA M is valid even when the MTA is under the same administration domain as network N. The connection between spammer S and first hop proxy is the upstream connection, and the connection between last-hop proxy z and MTA M is the downstream connection. There is a one-to-one mapping between the upstream and downstream connections along the spam laundry path. While this kind of connection mapping is common for proxy-based spamming, it is very unusual for normal email transmission.

In normal email delivery, there is only one connection, i.e. the connection between sender and receiving MTA. For each inbound SMTP reply message on the downstream connection, later on there will be one corresponding outbound SOCKS-encapsulated reply message carried by TCP on the upstream connection. We term this communication pattern as message symmetry. This message symmetry leads to the packet symmetry at the network layer. The packet symmetry between the inbound and outbound reply packets holds most of time. According to the one-to-one mapping of packet symmetry, each SMTP reply packet observed on the downstream SMTP connection should cause one and only one TCP packet appeared on the upstream connection. If one connection among X, Y, and Z is the suspicious upstream connection, one and only one outbound TCP packet must be observed from that connection in every reply round. DBSpam consists of two major components: spam detection module and spam suppression module, in which the detection module is the core of DBSpam. The detection module of DBSpam is a simple and efficient connection correlation detection algorithm to identify the laundry path of spam messages (i.e., the suspicious downstream and upstream connections) and the spam source4 that drives spamming behind the proxies. Given the characteristic of successive arrival of observations, this correlation detection problem is well suited for the statistical method of Sequential Probability Ratio Test (SPRT) developed by Wald . As a simple and powerful mathematical tool, SPRT has been used in many areas such as portscan detection and wireless MAC protocol misbehavior detection. Basically, an SPRT can be viewed as an one-dimensional random walk. The walk starts from a point between two boundaries and can go either upward or downward with different probabilities. With each arrival of observation, the walk makes one step in the direction determined by the result of observation. Once the walk firstly hits or crosses either the upper boundary or the lower boundary, it terminates and the corresponding hypothesis is selected. For SPRT, its actual false positive probability and false negative probability are bounded by predefined values. It has been proved that SPRT minimizes the average number of required observations to reach a decision among all sequential and non sequential tests, which do not have larger error probabilities than SPRT. Basically, the inbound SMTP traffic is monitored , then the rule of packet symmetry is applied for detecting the spam laundry path inside the customer network.

In this paper, a simple and effective system DBSpam is presented , to detect and break proxy-based email spam laundering activities inside a customer network and to trace out the corresponding spam sources outside the network. Instead of content checking, DBSpam leverages the protocol semantics and timing causality of proxy-based spamming to identify spam proxies and real spam sources behind them. Based on connection correlation and packet symmetry principles, DBSpam monitors the bidirectional traffic passing through a network gateway, and utilizes a simple statistical

method, Sequential Probability Ratio Test, to quickly filter out innocent connections and identify the spam laundry path with high probability.

ANALYSIS OF RESEARCH PAPER 2 TITLE : Multi-Agent Interaction Based Collaborative P2P System for Fighting Spam

AUTHORS : Guoqing Mo, Wei Zhao Institute of Command Automation PLA University of Science and Technology Nanjing, Jiangsu Province 210007, China [email protected], [email protected] Haixia Cao College of Information Engineering Jiangxi University of science and Technology Ganzhou, Jiangxi Province 314000, China [email protected] Jianshe Dong School of Computer and Communication Lanzou University of Technology Lanzou, Gansu Province 730050, China [email protected] The paper proposes a novel method to utilize the interaction between multi agents in a peer-to-peer system for spam filtering. There are mainly two kinds of agents in the system: local agent for filtering spam and learning spam’s feature; social agent for searching the same or approximate mails. When a mail arrives, the local agent takes charge at first; the mail is classified into three categories: non-spam, spam, suspicious spam. Only the suspicious spam is sent to the P2P network by social agent for a collaborative judgment. The result of judgment is returned to local agent, local agent is continuously learning the judgment, so when the next time when same or approximate spam arrives, and local agent can block it. This is done without the user interaction.

MODEL Regardless of the spam’s content, they many characteristic in common, for example they are sent to too many recipients on the Internet, solution to the problem of spam is to use P2P network composed of social agents to catch “bulk mail” and use local agents to spot the “spam”. The architecture of the proposed system is shown in the Figure below.

From the figure, we can see that the system is based on two kinds of agent: social agent and local agent. Social agent is located in mail server and local agent is located in mail user’s host computer. Social agent is the agent who participates in the peer-to-peer network, which represents a cluster of mail server. The social agent is activated by two kinds of events:

1) Every new email’s arrival, 2) Query from local agent. Local agent is responsible for filtering spam using content-based filters. Local agent is activated when user connects to email server.

SOCIAL AGENT When a new email comes, the social agent takes the email digest, publishes it to the P2P network. After publishing, digests is stored at one peer of the P2P network. So when the same kind of mail arrives again we can create the digest and then find in the network the same message and based on different parameters social agents can take the decision of marking that mail as spam or no. If they are marked as spam then the local agent is informed about the same so that local agent can add the mails digest in its database of spam. If that message arrives again then the local agent can directly filter it out rather than sending it again to the social agent and can save bandwidth and time.

LOCAL AGENT When the user’s MUA (multi user agent) connects to the email server and want to download emails, the local agent is activated. The emails is first downloaded to local agent, to classify the incoming mail, into three categories 1) Spam 2) Non-Spam 3) Suspicious spam. Then the suspicious mail is sent to Social agents for their collaborative judgment.

ANALYSIS OF RESEARCH PAPER 3 TITLE : Spam Attacks: P2P to the Rescue AUTHORS : Ernesto Damiani DTI Universit`a di Milano 26013 Crema Italy [email protected]

S.De Capitani di Vimercati DTI Universit`a di Milano 26013 Crema Italy [email protected]

Stefano Paraboschi DIGI Universit`a di Bergamo 24044 Dalmine Italy [email protected]

Pierangela Samarati DTI Universit`a di Milano 26013 Crema Italy [email protected]

Andrea Tironi DTI Universit`a di Milano 26013 Crema Italy [email protected]

Luca Zaniboni DTI Universit`a di Milano 26013 Crema Italy [email protected]

The paper proposes a decentralized privacy-preserving approach to spam filtering. The solution exploits robust digests to identify messages that are a slight variation of one another and a peer-to-peer architecture between mail servers to collaboratively share knowledge about spam. Many spam filtering approaches represent a first step toward the design of a P2P collaborative spam filtering solution, but they do not take into consideration some important aspects (e.g., the confidentiality of the messages and the robustness against attacks). The model is built on the idea that a P2P enabled polling mechanism can help in determining what a community considers to be spam and getting rid of it. The proposal is aimed at achieving both flexibility and effectiveness. Firstly, the hierarchyaware P2P architecture can be deployed in a variety of organizational situations, in presence of multiple mail servers of different size and reliability. Secondly, the P2Pbased anti-spam filtering engine rigorously protects users’ privacy, avoiding to disclose the content of the messages they receive, and is robust about counter measures that spammers themselves may take to impair its effectiveness.

ANTI SPAM P2P ARCHITECTURE : The anti-spam system is based on a three-tiered architecture,with users at the lower level and a P2P network connecting mail servers above them. The P2P network comprises of two families of nodes: peers and super-peers. Each set of users together with their mailer form a cluster. Intra-cluster data communication takes place via direct links between the users and their mailer, while inter-cluster communication takes place via the P2P network. spam reports by users are communicated by the mail server without indication of the identity of the users who originated them. Each mail server knows the identity of its users (although it does not propagate it in association with reports), so it can be safely assumed that each user is identified by her mailer via a unique identifier. The mail servers are used as a distributed repository of knowledge about spam, to be used by our filtering service. Each mail server, in turn, gets to know which messages are spam simply by (transparently) polling the opinions of its users. PROTOCOL It is assumed that each mail server s is associated with a pair of keys, (public, private ) and it uses its private key to sign outgoing communications. Furthermore, each message m can be identified by a digest that is robust against typical disguising attempts, so that two messages can be identified to be the same message if they map to a similar digest, even if their text is not identical. The idea is that the super-peers in the network maintain a distributed collection of spam digests that peers have identified ; peers can query this collection to obtain information about unknown emails. For each message m, the mail server records the number of copies directed to its users that it has received; the number of users who have reported the message as spam; and the number of users who have submitted a contrary report (if the message was sent to them already tagged as spam). The mail server also maintains control information, mainly in the form of thresholds it uses to determine when to enact polling or to tag a message as spam. Each mail server acting as a super-peer in the P2P network maintains track of spam reports received from the mail servers referring to it. Each spam report is stored at the super-peer in the form in which it has been received, i.e., signed by the mail server that has expressed it, so that further recipients of the report will be able to assess its authenticity. At the user tier, users receive emails. Upon reception of a message m, a user can report the fact that m is spam to its own mailer. If the email received by the user has already been tagged as spam by the mail server, and if the user does not agree with it ,the user can send a contrary report to his mailer.

At the peer tier , when the number of received occurrences of a given message reaches the suspicious threshold, the server sends a query to the super-peers inquiring whether the message has been reported as spam by other mailers. In response to such a query the mail server will receive a set of signed spam reports. It then performs an aggregation of the reports, weighting them differently depending on the reputations of the mail servers involved, to determine whether m is to be considered spam. When the number of spam notifications reaches the established threshold, the server adds the message to the spam catalog and sends a message to its super-peers reporting that it considers the message to be spam. The super-peers’ additional workload consists in managing spam reports and spam inquiries coming from the mail servers that refer to them, or from other super-peers. Upon reception of a new spam report from a mail server, the super-peer adds a corresponding entry in its catalog. Upon reception of a query from a mail server, the super-peer will both broadcast the query toward other super-peers in the P2P network. As for all the communications in the protocol, the query response is signed by the super-peer. Super-peers provide a communication channel between mail servers and do not perform any intermediate aggregation of reports. Thus a solution has been presented exploiting the P2P potential to make a first step toward a spam-free email system.

DEVELOPER’S MANUAL SETUP REQUIREMENTS:  

Java 1.5 or Higher. CS Course Library. Can be http://www.cs.rit.edu/~ark/cscl.shtml#download

downloaded

from

Set the classpath to the library Example of a command to set the classpath to the current directory plus the CSCL JAR file

LINUX BASH SHELL export CLASSPATH=.:/home/manan/cscl.jar

LINUX CSH SHELL setenv CLASSPATH .:/home/manan/cscl.jar

WINDOWS Set classpath=%classpath%;c:\directory\cscl.jar;

To compile, first unjar the provided source using the following commands:

For Client Server simulator jar -xvf Demo1.jar

For P2P simulator jar –xvf Demo2.jar

Then type following command to compile the source file javac *.java

USERS’S MANUAL Running Client Server Model of Spam Filtering Start the Registry Server using the following command on the prompt. java Start edu.rit.ds.registry.RegistryServer [host] [port]

Where is the name of the host computer where the Registry Server is running, is the port number to which the Registry Server is listening. If host and port are left blank then registry will start at localhost : 9901

Start the Spam-Detection Central Server java Start SpamServer [host] [port] [name] [blacklistFileName] [localFile]

Where is the Registry Server host name, is the registry Server port number, is the name of the Spam Server, is the path of the Blacklist file, is the path of local file used for internal purpose of Spam Server.

Start as many MailServers as you want. java Start MailServer [host][port][name][SpamServerName]

Where is the Registry Server host name, [port] is the Registry Server port number, is the name of the MailServer, is the name of the Spam Detection Server.

Run Spammers java Spammer [host][port][name][MailServerName][Number_of_mail]

Where is the Registry Server host name, is the Registry Server port number, is the name of the spammer, is the name of the mail server, is the number of mails to send for testing purpose.

Running P2P Model of Spam Filtering Start the Registry Server using the following command on the prompt. java Start edu.rit.ds.registry.RegistryServer [host] [port]

Where is the name of the host computer where the Registry Server is running, is the port number to which the Registry Server is listening. If host and port are left blank then registry will start at localhost : 9901

Start as many mail severs as nodes in a P2p network java Start MailServer [host][port][name] Where is the Registry Server host name, is the Registry Server port number and is the name of the MailServer. Provide the name of Mail Server in increasing number as you start the Mail Server.

Run Spammers java Spammer [host][port][name][MailServerName][Number_of_mail]

Where is the Registry Server host name, is the Registry Server port number, is the name of the spammer, is the name of the mail server, is the number of mails to send for testing purpose.

DESIGN PART 1 : CLIENT SERVER MODEL

The components in the design for the Client Sever model are :  SERVER : The database server keeps a list of all ip addresses classified as blacklisted.The server also has a log database that has information of all the emails received ( the sender , time stamp when the message was sent).

 CLIENT : The clients are the mail servers. For every incoming mail , it checks whether the sender is blacklisted and also notifies the server about the current mail received.

Working : For every mail received by the mail server , it first queries whether the sender is present in the black list. If not , an entry is made in the log about this particular mail as to who sent it and what was the received time for the mail. Once a particular sender crosses the thresholds set for number of mails sent in specified amount of time , the sender gets black listed and any further mails from that sender is classified as spam.

PART 2 : SERVERLESS PEER-2-PEER MODEL

The components in the design of the serverless P2P model are :

The clients in the P2P network are the mail servers which maintain a distributed database within the group which keep a list of all ip addresses classified as spam in the group and also the track of every mail received in the system.

Working : We simulate the working of the P2P system CHORD for our spam classifier. For every mail received by the mail server , the hash value of the sender’s unique identity( Ip address) is found out. Then a query is sent by the mail server to the other mail servers in the P2P network whether the sender is black listed or not. According to the hash value , the query is directed to the appropriate mail server in the P2P network and the server replies back. If the sender is not black listed , a log is made by the appropriate mail server regarding the sender and the time the message was received. If the sender crosses the threshold values set for the number of mails sent in specified amount of time , the sender is black listed.

SCREEN SHOTS

FUTURE ENHANCEMENTS -

Implementation of user specified White lists.

-

Implementation of calculation of Hash value of the content of every mail received and then checking for large number of duplicate Hash values.

-

Implementation of other spam classifying techniques along with the existing one like content based filtering, ping back method, etc.

TEST CASES DESCRIPTION

CLIENT SERVER

P2P

1 SPAMMER , 1 MAIL SERVER

11.313 SEC

11.216 SEC

1 SPAMMER , 2 MAIL SERVERS

12.600 SEC

11.297 SEC

2 SPAMMERS , 1 MAIL SERVER

12.786 SEC

12.352 SEC

2 SPAMMERS, 2 MAIL SERVERS

11.461 SEC

10.677 SEC

10 SPAMMERS, 1 MAIL SERVER

25.821 SEC

20.311 SEC

5 SPAMMERS, 5 MAIL SERVERS

20.566 SEC

11.762 SEC

10 SPAMMERS,10 MAIL SERVERS

22.845 SEC

12.63 SEC

INVESTIGATIONS -

P2P system performed better than the Client – Server system even for a small number of spam mail in the system.

-

With the increasing number of nodes in the P2P network the performance of P2P is way too superior than the client server architecture.

-

As the number of clients increases the performance of the Client Server model degrades with the presence of a bottle neck while the performance of the serverless P2P system upgrades with lots of parallel computations being done.

-

The Client Server model is expensive in the sense that a dedicated server has to be present for responding to queries fired by the client mail servers.

-

The Client Server model has the drawback of a single point of failure.

-

Network bandwidth has to be compensated a little for the P2P architecture for implementation of fault tolerance.

-

A collaborative effort by the major commercial mail servers along with the use of existing DNS blacklists can help effectively fight spam.