Profiling Engine SDK
the Needs of Routing and Categorizing Services
in the routing and categorization industries have either used
full-featured indexing engines or else have written their own
simple scanners. Simple scanners, while appealing in terms of
cost, lack the power and flexibility needed to do most types
of analysis. The alternative of using traditional indexing engines
must deal with the fact that those indexers were primarily directed
towards uses different than those of the categorization industry.
Our goal with the Lextek
Profiling Engine was to develop a version of our Onix indexing
engine that was optimized to meet the needs and aims of the categorization
and routing markets. To deal with these very different needs
we designed the Lextek Profiling Engine to create memory-based
indexes. Most routers, rather than indexing hundred or thousands
of documents at a time, index only a few documents. Our Profiler
includes in the indexing process only those features that are
needed, improving speed and reducing overhead. Because we are
not creating indexes that will be reused, we can eliminate compression,
encryption, and the speed problems associated with disk access.
Much of the time taken
in querying a traditional index is taken up in searching through
the index on the disk. That takes time both to wait for the disk
to transfer the information but also to compute where the desired
data is. In general accessing a disk is several orders of magnitude
more lengthy than going directly to memory. By keeping the index
in memory we can access it with extremely dramatic speed increases.
In internal tests these changes have resulted in speedups of
up to 5 times those of comparable traditional indexers.
The Lextek Profiling Engine
SDK also has an extremely advanced query language. This language
eases the creation of the large types of queries often found
in the categorization industry. It is a full programming language
allowing code reuse. It also has numerous ranking methods, deferred
evaluation of queries, named sub-queries, and features that allow
the Profiler to dramatically optimize your queries for speed.
Key Benefits of
the Lextek Profiling Engine
fast memory based indexer
the overhead associated with large static indexes
Advanced Ranking Methods
Language optimized for large complex queries with hundreds of
Language designed to aid code reuse and debugging the complex
queries found in categorizers
is easy to integrate into your existing applications and projects.
It uses an API very similar to the Onix Indexing Toolkit. If
you already are using Onix you can move to the Profiling toolkit
in a matter of minutes. If you are using a competing toolkit
you can still convert over your projects with minimal work. Usually
converting over takes only a day.
integrated the Lextek Profiling Engine you are ready to take
advantage of the new features optimized for analyzing texts.
You will find dramatic speed increases over competing products.
You will also find analysis tools that can significantly improve
the accuracy and usefulness of your work.
For information and evaluation
copies please contact us at
Provo, UT 84604