As you can see, there is a bit of overlap among the field types. Essentially, you should ask yourself whether this field needs to be searched, whether it needs to be displayed, and whether it is too big to be stored. Those questions will help guide your selection of what field types to use. Directory The Directory object is an abstraction of the underlying storage of indexes. There are two existing implementations of the abstract Directory class: FSDirectory for file systems, and RAMDirectory for in-memory storage. In theory, you can implement your own Directory object to store your indexes in numerous underlying storage mechanisms such as databases or document management systems. Unfortunately, the Directory class adopts a file-based paradigm, which makes it tougher to understand how to implement the interface. Understanding the Lucene Query Syntax Lucene provides the flexibility for you to write your own query language. However, this flexibility has already provided a strong query language to use right out of the box. A good reference for this is avail- able online at http://jakarta.apache.org/lucene/docs/queryparsersyntax.html. The following sections explain the syntax. Terms Terms are generally like your conventional search engine. Each word, separated by a space, is a term unless you place them in quotes: members “vast right wing conspiracy” In this case, there are two terms: “members” and “vast right wing conspiracy.” Clearly, you do not want the terms vast, right, wing, or conspiracy by themselves. They are combined to be a meaningful term. Fields Our previous search will search against the default field that you specified when you initialized your QueryParser. However, sometimes you would like to search against another of the fields in your index: site:www.rnc.org “vast right wing conspiracy” In this case, you are specifying that you want to search for “www.rnc.org” in the site field (which you created) in your index. The “vast right wing conspiracy” term is still being run against the default field. Term Modifiers There are a number of ways that you can modify a term within your search. The following table demon- strates a list of these modifiers. 54 Chapter 2 05 469513 Ch02.qxd 1/16/04 11:04 AM Page 54 Technique Example Description Single Character to?t Matches any character in that one position. For Wildcard example, “toot” or “tort” would be valid matches. Multiple Character to*t Matches any number of characters in that position. Wildcard In this case, the word “toast” would also be valid. Fuzzy Usama~ Placing a tilde (~) at the end of the word provides fuzzy logic using the Levenshtein Distance algo- rithm. In this case, this would return “Osama” as a valid term. This is useful when you believe you may be a character off on a spelling. This technique implicitly boosts the term by 0.2 also. Boosting UML^5 tools Increases the relevance of a search term. In this case, it means that “UML” is five times more relevant than “tools.” The boost factor must be positive, but can be a decimal (for example, 0.5). Proximity “Microsoft Java”~7 This will return results where the words “Microsoft” and “Java” are within seven words of each other. This can provide a basic conceptual search capability by indicating how certain key words can be closely related. Boolean Operators, Grouping, and Escaping Lucene supports the common Boolean operators found in almost all search engines: ❑ AND indicates that two terms must be present together in a given document, but in no particu- lar order, such as a phrase term (for example, “cold war”). For another example, Homer AND Simpson will return pages that contain both terms, even if they are not next to each other. ❑ OR will return pages that contain either of the terms indicated. This is helpful when you have alternate ways of describing a particular term. “Bull Run” OR Manassas would return pages that contain either of the names used to describe the first battle of the American Civil War. ❑ + means that a term must exist on a given page. If you use +Wrox Java, it would return only pages that had “Wrox” on them. ❑ - means that a term cannot appear on a given page. If you wanted to look at all pages related to Wrox that don’t pertain to Microsoft, you could use Wrox -Microsoft. ❑ NOT behaves much like the “-” command. If you were looking for documents about the Bundy family, but you didn’t want to be bogged down by all the documents about Ted Bundy, you would use Bundy NOT Ted. You cannot use wildcards at the beginning of a search term. 55 Searching with Lucene 05 469513 Ch02.qxd 1/16/04 11:04 AM Page 55 Grouping is another powerful capability that also exists in Lucene. Usually, if you are going to use Boolean conditions, you need a mechanism to group together conditions. Consider the following example: (“order of battle” AND “casualty figures”) at the First Battle of (“Bull Run” OR Manassas) In this case, you want pages that contain the order of battle and the casualty figures for the first battle of what is known as either “Bull Run” or Manassas. This shows a perfect example of using grouping and Boolean operators to make sophisticated queries. Of course, to support this expansive query syntax, Lucene uses a number of special characters, listed here: + - && || ! ( ) { } [ ] ^ “ ~ * ? : \ Therefore, for you to search for a TV show called “Showdown: Iraq,” you would need to escape the colon in the query as follows: “Showdown\: Iraq”. Notice how this is just like the escape sequence in Java, so it should be easy to remember. While you have seen the power of the Lucene Query Syntax, and how useful it can be in creating sophis- ticated searches, it is very important to consider the sophistication of the users of your system. While most developers, and particularly open source developers, are strong Web researchers, most users of your system will not have a strong understanding of Lucene’s searching capabilities. Therefore, it becomes very important to provide a good user interface to enable users to maximize the benefits of searching with Lucene. Figure 2.9 provides an example of an Advanced Search page meant to leverage these advanced capabilities. Figure 2.9 56 Chapter 2 05 469513 Ch02.qxd 1/16/04 11:04 AM Page 56 Optimizing Lucene’s Performance In order to understand the performance considerations of Lucene, first consider how Lucene creates its indexes. Lucene creates segments that hold a certain number of files in them. It is easy to think of a seg- ment as an index part. Lucene holds its index in memory until it reaches the allowed capacity, and then it writes it to a segment on the disk. Once a certain number of segments have been written, Lucene merges the segments into bigger segments. To determine how often to write and merge the indexes to disk, the IndexWriter has a member variable known as the mergeFactor. The mergeFactor specifies how many files are stored before writing a seg- ment. In addition, it controls how many segments are written before they are merged together. Raising the merge factor increases the speed of your indexing activity, because more is being kept in memory and fewer file reorganization manipulations are being conducted. However, note two obvious problems here. First, your machine is limited in the amount of memory it has (a small fraction of the disk space), and sec- ond, the operating system can often limit the number of files you can have open at one time. You also need to know that IndexWriter has a member variable called maxMergeDocs. This variable sets the limit on the number of files that can be contained in one segment. Of course, the more files you have, and the less merging you do, the slower your searching will be. However, anticipating this problem, IndexWriter has a method known as optimize that will combine the segments on the disk (and reduce the number of files). Note that optimization can slow down index- ing tremendously, so a strong consideration would be to limit the use of optimize in indexing-intensive applications, and use it extensively in searching-intensive applications. Summary Lucene is a powerful search engine API. It is written in a very modular fashion, which allows you, as a developer, a tremendous amount of freedom in how you decide to use it to solve your problems. Because it is an API, it could be very effectively used to index your e-mail Inbox, a database, or a set of news feeds. The applications are limited only by how you choose to use them. This chapter covered the basics of search engines. Then we showed the techniques that search engines use to analyze text. From there, we described the internals of the Lucene API, providing some examples of how to do the major tasks required of an application developer who implements a solution with this API. We described the query syntax as a means of helping developers understand the toolset available for searching, and to encourage them to develop more sophisticated GUIs to leverage it. Part II of this book describes how to build your own portal. It provides practical examples of how you can add Lucene to your enterprise portal solution. 57 Searching with Lucene 05 469513 Ch02.qxd 1/16/04 11:04 AM Page 57 05 469513 Ch02.qxd 1/16/04 11:04 AM Page 58 Messaging with Apache James The Java Apache Mail Enterprise Server (James) is an open-source Java mail server that is part of the Apache Software Foundation’s Jakarta project. It is a 100 percent pure Java mail server that was designed to be a powerful, portable, and robust enterprise solution for e-mail and e-mail-related ser- vices. Part of its strength comes from the fact that it is based on current and open protocols. James is comprised of several different components and can be configured in different ways to offer a fully flexible and customizable framework. It is currently built on top of the Apache Avalon application framework (http://avalon.apache.org), which is also part of the Jakarta project. This framework encompasses good development practices and provides a solid foundation to host the James mail server. This chapter explores various concepts of the James server. It explains how to obtain, install, and configure the James server, and describes the various components that provide for the total e-mail solution. The chapter concludes with an introduction to the JavaMail Application Programming Interface (API), a small example application for sending and receiving e-mail using JavaMail and the James e-mail server, and an example of how James can be used as part of a portal application. This chapter covers many aspects of the James mail server, but for a more in-depth discussion and explanation of all of the components that comprise the James framework, visit its primary Web site at http://james.apache.org. Introducing James James was designed to be a complete enterprise mail solution. It can serve as a core component in an overall portal solution. The James server has many design objectives that are implemented in a number of features, including the following: 06 469513 Ch03.qxd 1/16/04 11:04 AM Page 59 ❑ Server Portability — Apache James is a 100 percent pure Java application that is based on the Java 2 platform and the JavaMail API. ❑ Complete Solution — The James mail server can handle the transport and storage of mail mes- sages on a single server. It does not require any other server or another associated application. ❑ Protocol Abstraction — James views the various mail protocols as simply communication lan- guages that tie the mail client to the mail server. It does not depend on any particular protocol, but rather follows an abstracted server design. ❑ Mailet Support — A mailet is a discrete piece of mail processing logic that is incorporated into the processing of a mail-compliant mail server. Apache James is such a server and supports the Apache Mailet API. Mailets are easy to write and enable developers to build highly customized and powerful mail applications. ❑ Resource Abstraction — Apache James abstracts its resources and accesses them through defined interfaces, much like the e-mail protocols are used. These resources include features such as JavaMail, used for mail transport, the Mailet API, and Java DataBase Connectivity (JDBC), for message and user data storage. James is highly modular and packages its compo- nents in a very flexible manner. ❑ Secure and Multi-Threaded Design — Apache James has a careful, security-oriented, fully multi-threaded design, allowing enhanced performance, scalability, and mission-critical use. This approach is based on the technology developed for the Apache JServ servlet engine. James also introduces several concepts that are at the core of how it manages to operate as a mail server, from both a production and administrative point of view. We will first describe them in a little more detail so that you get a better idea of how they work. How to configure these items in the James server is described in the section “Configuring James.” Working with Mailets and Matchers As mentioned earlier, a mailet is a discrete piece of mail processing logic that is incorporated into the processing of a mail server. James operates as a mailet-compliant mail server, which means that it under- stands how to process the Java code that uses the Mailet API. A mailet can do several things when pro- cessing a mail message. It can generate an automatic reply, build a message archive, update a database, or any other thing a developer would like to do with a mail message. James uses matchers to help deter- mine whether a mailet should process a given e-mail message that just arrived. If a match is found, James invokes that particular mailet. The Mailet API is a simple API used to build mail processing instructions for the James server. Because James is a mailet container, administrators of the mail server can deploy mailets. These mailets can either be prepackaged or custom built. In the default mode, James uses several mailets to carry out a variety of server tasks. Other mailets can be created to serve other purposes. The current Mailet API defines inter- faces for both matchers and mailets. Because the API is public, developers using the James mail server can write their own custom matchers and mailets. Writing mailets and matchers is a relatively simple process. For mailets, you typically implement the Mailet interface through the org.apache.mailet.GenericMailet class. This class has several meth- ods, but in order to write a generic mailet, you only have to override the service method: abstract void service(Mail mail) 60 Chapter 3 06 469513 Ch03.qxd 1/16/04 11:04 AM Page 60 Writing a matcher is just as simple. Simply use the org.apache.mailet.GenericMatcher class and override the match method: abstract Collection match(Mail mail) Matchers, as identified earlier, are used to match mail messages against a set of conditions. If a match is met, it returns a set of the recipients of that message. Matchers do not modify any part of the message during this evaluation. Mailets, on the other hand, are responsible for processing the message and can alter the content of the message or pass it on to some other component. James comes bundled with sev- eral mailets and matchers in its distribution. The following sections describe the various mailets and matchers that are bundled with the James server. Bundled Matchers The matchers that are bundled with James were identified by members of the user and developer com- munities because they were found useful in their own configurations. Following is a list of the specific matchers. More information on these matchers, including configuration information, can be found at http://james.apache.org/provided_matchers_2_1.html. ❑ All — A generic matcher that matches all mail messages being processed. ❑ CommandForListserv — This matcher is used as a simple filter to recognize mail messages that are list server commands. It matches messages that are addressed to the list server host as well as any message that is addressed to a user named <prefix>-on or <prefix>-off on any host. ❑ FetchedFrom — This matcher is used with the James FetchPOP server. FetchPOP is a compo- nent in James that allows an administrator to retrieve mail messages from multiple POP3 servers and deliver them to the local spool. This process is useful for consolidating mail residing in accounts on different machines to a single account. The FetchedFrom matcher is used to match a custom header set by the FetchPOP server. ❑ HasAttachment — Matches mail messages with the MIME type multipart/mixed. ❑ HasHabeasWarrantMark — Matches all mail messages that have the Habeas Warrant. A Habeas mark indicates that the message is not a spam message even though it may look like spam to the e-mail server. Information on these messages can be found at www.habeas.com. ❑ HasHeader — Matches mail messages with the specified message header. ❑ Hostls — Matches mail messages that are sent to a recipient on a host listed in a James configu- ration list. ❑ HostlsLocal — Matches mail messages sent to addresses on local hosts. ❑ InSpammerBlacklist — Checks whether the mail message is from a listed IP address tracked on mail-abuse.org. ❑ IsSingleRecipient — Matches mail messages that are sent to a single recipient. ❑ NESSpamCheck — This is a matcher derived from a spam filter on a Netscape mail server. It detects headers that indicate if it is a spam message. ❑ Recipients — Matches mail messages that are sent to a recipient listed in a specified list. ❑ RecipientslsLocal — Matches mail messages that are sent to recipients on local hosts with users that have local accounts. 61 Messaging with Apache James 06 469513 Ch03.qxd 1/16/04 11:04 AM Page 61 ❑ RelayLimit — This matcher counts the number of headers in a mail message to see if the num- ber equals or exceeds a specified limit. ❑ RemoteAddrInNetwork — Checks the remote address from the e-mail message against a con- figuration list of IP addresses and domain names. The matcher will consider it a match if the address appears in the list. ❑ RemoteAddrNotInNetwork — Checks the remote address from the e-mail message against a configuration list of IP addresses and domain names. The matcher will consider it a match if the address is not in the list. ❑ SenderInFakeDomain — Matches mail messages in which the host name in the address of the sender cannot be resolved. ❑ Senderls — Matches mail messages that are sent by a user who is part of a specific list. ❑ SizeGreaterThan — Matches mail messages that have a total size greater than a specified amount. ❑ Subjectls — Matches mail messages with a specified subject. ❑ SubjectStartsWith — Matches mail messages with a subject that begins with a specified value. ❑ Userls — Matches mail messages that are sent to addresses that have user IDs listed in a con- figuration list. Bundled Mailets The bundled mailets, like the matchers, are commonly used by members of the user and development community. More information on the following mailets, including configuration information, can be found at http://james.apache.org/provided_mailets_2_1.html: ❑ AddFooter — Adds a text footer to the mail message. ❑ AddHabeasWarrantMark — Adds a Habeas warrant mark to the mail message. ❑ AddHeader — Adds a text header to the mail message. ❑ AvalonListserv — Provides functionality for a basic list server. It implements some basic fil- tering for mail messages sent to the list. ❑ AvalonListservManager — Processes list management commands of the form <list-name>- on @ <host> and <list-name>-off @ <host>, where <list-name> and <host> are arbitrary. ❑ Forward — Forwards the mail message to the recipient(s). ❑ JDBCAlias — Performs alias translations for e-mail addresses stored in a database table. ❑ JDBCVirtualUserTable — Performs more complex translations than the JDBCAlias mailet. ❑ LocalDelivery — Delivers mail messages to local mailboxes. ❑ NotifyPostmaster — Forwards the mail message to the James postmaster as an attachment. ❑ NotifySender — Forwards the mail message to the original sender as an attachment. ❑ Null — Completes the processing of a mail message. 62 Chapter 3 06 469513 Ch03.qxd 1/16/04 11:04 AM Page 62 ❑ PostmasterAlias — Intercepts all mail messages that are addressed to postmaster@<domain>, where <domain> is one of the domains managed by the James server. It then substitutes the con- figured James postmaster address for the original one. ❑ Redirect — Provides configurable redirection services. ❑ RemoteDelivery — Manages the delivery of mail messages to recipients located on remote SMTP hosts. ❑ ServerTime — Sends a message to the sender of the original mail message with a timestamp. ❑ ToProcessor — Redirects processing of the mail message to the specified processor. ❑ ToRepository — Places a copy of the mail message in the specified directory. ❑ UseHeaderRecipients — Inserts a new message into the queue with recipients from the MimeMessage header. It ignores recipients associated with the JavaMail interface. Understanding SpoolManager As a mail server, James uses POP3 and SMTP services to receive and send e-mail messages. What James does with a message once it receives it, however, is up to the SpoolManager. James separates the ser- vices that are used to deliver the mail messages from the service that it uses to process a piece of mail once it is received. The SpoolManager is a mailet and is the service component that James uses for its mail processing engine. As previously described, it is a combination of matchers and mailets that actu- ally carry out the mail processing. The SpoolManager continues to check the spool repository for any new mail messages. Mail can be placed in the spool repository from any number of sources. These include the POP3 or SMTP services. The SpoolManager contains a series of processors. Each one will indicate what state the mail message is in as it is processed in the SpoolManager. When a piece of mail is found in the repository, it is first sent to the root, or first, processor. Besides holding newly arrived mail messages, the spool repository also holds messages as they transit from one processor to another. Mail messages continue through the vari- ous processors until they are finally marked as completed by a mailet. The SpoolManager can be configured to address many needs that an administrator may have. Processes to perform operations such as filtering and sorting can easily be created through custom matchers and mailets that are used by the SpoolManager component. A large part of the James mail server’s flexibility lies in the power of the SpoolManager. Understanding Repositories James uses repositories to store mail and user information. There are several different types of reposito- ries and each serves a different purpose. The user repository is used to store information about users of the mail server. This may include user names, passwords, and aliases. The mail repository is used to store mail messages that have been delivered. Spool repositories will in turn store messages that are currently being processed. Last is the news repository, which is used to store news messages. Aside from having different types of repositories, James can also use different types of storage for these reposi- tories. These storage types include File, Database, and DBFile. Each of these is briefly described next. 63 Messaging with Apache James 06 469513 Ch03.qxd 1/16/04 11:04 AM Page 63 [...]... specifies the POP3 port number as 111 80 Messaging with Apache James If the POP3 port is something other than the default port or 110, it must be stated when the connection to the store is made (see line 36 ) 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 0 43 044 045 046 047 048 049 050 051 052 0 53 054 055 056 057 058 059 060 061 062 0 63 064 065 066 067 068 069 070 071 072 0 73 074 075 076... align=”left”> To: 77 Chapter 3 114 115 116 117 118 119 120 121 122 1 23 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 1 43 144 145 146 147 148 149 150 151 152 1 53 154 155 156 157 158 159 160 161 162 1 63 164 165 166 167 168 78 . to use JavaMail, you will need to install a few items first: ❑ Install the Java 1.2 (or later) Java Development Kit (JDK). ❑ Obtain the latest JavaMail API. This can be found at http:/ /java. sun.com/products/javamail/ index.html follow Step 3 to add any other user. To see other commands that can be run in the RemoteManager, type in the word help. Introducing JavaMail API The JavaMail API is a package of Java classes. your enterprise portal solution. 57 Searching with Lucene 05 4695 13 Ch02.qxd 1/16/04 11:04 AM Page 57 05 4695 13 Ch02.qxd 1/16/04 11:04 AM Page 58 Messaging with Apache James The Java Apache Mail