A hastily cobbled together log propagation thingamabob claimed that the backend answered snappy enough, network inspection showed a very high amount of requests to storage (a database that contains the records backing the search) and timeouts & session pool overflows were all over the place. The DidYouMean service was appointed scapegoat and killed with extreme prejudice.

Things got exponentially worse! Uptime was about 2 minutes, with last minute performance quickly falling off to unusable. Phones started ringing, emails ticked in and an angry mob with pitchforks lay siege to the office. Inspection revealed that killing DidYouMean meant that the service layer unsuccessfully tried to get the result for 20 seconds (yes, that timeout was far too high) before giving up, quickly filling Apache session pools. DidYouMean was resurrected, services started up again and all was well. Or at least back to where it was before the wrong service was unfairly executed.

Mads stands up and codes. A sure sign of high alert

Waiting for the next hammer to drop, code was reviewed (again), pool sizes were tweaked and logs were watched intensely. At 12:09:47 and 952 milliseconds, the impact riveter started again and storage staggered. But lo and behold: The maintenance guy had changed log levels to DEBUG (for a limited amount of time). An hour and 20,000 observed requests for the exact same ID later, the magical incantation i++; was inserted in a while loop. Testing, deployment, re-deployment, tomcat restart and another tomcat restart followed quickly.

It turned out that certain rare records triggered the endless loop. The progressively worse performance stemmed from more and more of these edge cases piling up, each looping forever. As the overwhelmed storage was on the same server as the searcher, the shared request pool was flooded with storage requests, only occasionally allowing search requests.

With the roaring fire quelled, business returned to normal. By pure coincidence, the assignment for the next days is vastly improved propagation, logging and visualisation of timing information throughout the services.

We use a backing storage for documents in our home brewed search system Summa. It was supposed to be a trivial key-value store with document-IDs resolving to documents. Then an evil librarian pointed out that books are related to each other, so we had to add some sort of relational mapping. For some years we have used relational databases for this, going from PostgreSQL through Derby and landing on H2, which has served us fairly well. Documents with relations were a bit slow to resolve but there were less than 5% of those in the full corpus, so for all practical purposes they presented no problem.

Fast forward to a week ago, where we added a new target to our integrated search. One million fresh documents or a 10% increase in total document count. Unfortunately most of these documents were related to each other, increasing our total relation count 2000%. Our full indexing time soared from “It’ll be done before noon” to “I hope it finishes before tomorrow”. Ouch!

A long discussion about database design and indexes followed. The conclusion was that we really did not use H2 for what it was good at and that maybe we should look at a graph-oriented database.

Implementing storage with neo4j

Neo4j is an open source graph database. It is written in Java and can be used as an embedded application. This is important for us as we like our packages to be self contained. Friday is work-on-whatever-project-you-think-can-benefit-Statsbiblioteket-day, so I dedicated it to trying out neo. Keep in mind that I only heard about Neo this morning and that I hadn’t looked at a single page about the product before I started the project.

• 10:36 Created test project
• 10:40 Selected Neo4j version 1.3 stable Enterprise, added to Maven POM, downloaded files
• 10:50 Created skeleton class for Summa NeoStorage
• 10:55 Added basic properties, created skeleton Unit test
• 11:15 Added code for flushing Summa Record to Neo
• 11:36 Added code for retrieving a previously flushed record + unit test. Hello world completed
• 11:40 Break finished, started on ModificationTime retrieval
• 12:35 Proof of concept retrieval using modification time (slow development due to human error)
• 13:20 Finished bulk ingest and record-by-record extraction using modification time order, including unit test
• 13:50 Finished mapping of Summa Record child-relations to Neo, both for ingest and extraction

Including mistakes, distractions from colleagues and reading documentation, it took under 4 hours to integrate Neo as the new backend for our storage. There’s still a lot of minor things to add and special cases to cover, but the result is complete enough to be used for most workflows in Summa.

Implementation was a breeze, the API was very clean and the examples and guides at the website were to the point and well thought out. Kudos to the developers for great work!

Performance peek

Since the Neo Storage isn’t finished, it would be unfair to compare it to H2 with regard to ingest. However, the extraction part is complete enough to test.

With the old H2-backed storage, our extraction time fell to below 5 records/sec for the new documents with 2-4 relations each (extraction time for records without relations is 2-3000 records/sec).

Creating a test-storage using Neo with 100000 documents, each with 5 children, changed the extraction speed to 2500 expanded records/second or 15,000 raw records/second if we count the children.

Granted, the only fair test is with production data, but so far Neo4j looks like a clear winner for our purposes!

SOLR-2412 is a fairly thin wrapper around LUCENE-2369. LUCENE-2369 was designed with the clear trade-offs

* Slow startup
* Low memory overhead
* Fast response

with the archetypal usage scenario being a large index containing one or more rich hierarchies that is batch-updated every night (see Hierarchical faceting – working code for more details). With fear of misrepresenting, SOLR-64 and SOLR-792 were created from a feature-standpoint with performance characteristics being secondary.

Feature wise, SOLR-2412 (let’s call it Exposed faceting from now on) differs markedly from pivot faceting (SOLR-792) at this time, as neither of the two solutions can do what the other one does. However, I feel confident that Exposed faceting can be tweaked to do pivot faceting later on. The main reason to use Exposed over SOLR-792 would be to change trade-offs.

Compared to SOLR-64, Exposed faceting’s features differs primarily by supporting multiple paths per document: A product belonging to multiple categories, multiple locations for a bus route and so on.

The next step is to create a test bed for doing performance measurements on Exposed vs. Solr’s different faceting implementations. Naturally the hoped-for outcome is that Exposed is markedly better under the defined trade-offs.

Many different places have tried building their own data well (sometimes a large national data well) but most have given up. Why? Primarily because of the unwillingness of publishers to hand over data to every single data well but also because the very concept of a very large local data well has been made (at least somewhat) obsolete by the new “Web Scale Discovery” tools – such as Summon, Primo Central and EBSCOHost.

Some libraries are fine with using the standard discovery interfaces that these services provide and some would rather use their APIs and build their own interface on top, perhaps adding tight integration with their library system and other locally developed systems.

However this area is fast moving and just as it looked as if pretty much all metadata would eventually be available in all these systems EBSCO has decided not to allow their metadata to be indexed outside of their own system. They will however allow for “just-in-time” searches to be used. It appears as if the market is fragmenting back into federated search – and the problems with federated search are well-known and are what made us all pursue integrated search to begin with.

But can we do better this time around? Fortunately the answer is yes – if we can get a little help from our friends.

There are two main problems with federated search:

• Response times: The entire federated system is only as fast as the slowest search node.
• Merging: There is no meaningful way to merge different result sets as they can have vastly different sorting criteria.

We can’t do a lot about the problems with response times but fortunately the new systems are _a lot_ faster then the old ones, so hopefully it wont be that big an issue.

Merging has however gotten easier, strangely enough as a side effect of making the sorting of individual results more complicated. The magic here lies in relevancy ranking and the fact that pretty much all new systems are based on the same principles and code base (ie. Lucene/Solr).

So how does this work? The relevancy ranking of a given document in a query is based on different things but a major contributor is the term frequency–inverse document frequency.

We have chosen to call this concept Virtual Integrated Search as the end result is (potentially) virtually indistinguishable from from having a large local data well coupled with a true integrated search. For the Well11 and code4lib 2011 conferences we have prepared a first stab at an implementation and integration with our existing Search and Summa system. This is not much more than an internal beta version where a primary focus of the frontend has been to make is possible to tweak the way merging is performed.

What will it look like? The main purpose for us is to make it look to the user as if all the data is retrieved from a single source. Currently we present results from Summon in a box of its own but the plan is to simply go back and present all results in a single list.

This is something we will be working on and experimenting with in the coming months and we are quite excited about the possibilities.

code4lib 2009 was the best library-oriented conference we’ve been to, so our hopes for 2011 are high. The program certainly looks interesting and one common theme – merging of search results from different sources – is perfectly timed to our current project on merging of Summa and Summon search results. Hopefully we’ll have enough experience by then to do a lightning talk about it.

We will also be attending the pre-conference on Solr and since hierarchical-like faceting seems to be a fairly hot topic this year, we plan to hack together a Solr-based proof of concept of our take on the problem before the conference.

One of the reasons why we’ve been rolling our own instead of taking the obvious Solr-road is historical. Solr didn’t meet our needs when we started. It still doesn’t in its current form, but it’s getting close. Another reason is that the scope of Summa is different from Solr’s: We’ve invested a lot of energy in handling data-flow with transformations in a multi-stage system with a persistent storage service that handles parent-child relations and keeps track of updates.

The interesting thing here is that Summa and Solr are not necessarily in competition. Solr has a lot of great analyzers and has gained a lot of momentum during the last years. We would therefore like to use Solr at the core of Summa instead of the custom Lucene searcher we have currently. At Statsbiblioteket we have done a fair amount of work with low-memory index-lookup, sorting and hierarchical faceting which we’d like to take with us, so this calls for participation in Solr development.

Relevance Correlation between systems

Statsbiblioteket has recently begun collaboration with the Summon team from Serial Solutions. Due to Danish legislation we are not allowed to let Summon index our own data, so the upcoming search setup needs to merge results from Summon with our own. As both systems sort the results by relevance ranking, proper merging is quite hard. This is due to the fact that relevance scores are not comparable between different systems. One solution is to perform a local comparison of the returned fields in the results, but as this completely ignores the whole term statistics based ranking of Lucene/Solr, we would like something better.

Up until now we have been spoiled by using relevance ranked integrated search across different sources where we controlled the full indexing process. SOLR-1632 seems to be the right way to provide similar functionality for Solr, but it is not mature yet and is quite a large thing to put into production. Better to walk before running. As part of our collaborative agreement Serial Solutions has agreed to deliver statistics about their metadata to us on an experimental basis. With this we can hopefully do better merging and issue boosted queries to get the hits that are relevant to us, thus approximating the results we would have gotten if all metadata had been in a single index.

To recap, we want hierarchical faceting. Searching for “monkeys” should give us something like

Classification
  - Primates (9)
    - Atelidae (2)
      - Atelinae (2)
        - Ateles (2)
    - Cebidae (5)
      - Cebus (2)
        - C. apella (1)
        - C. capucinus (1)
      - Saimiri (3)
        - S. sciureus (3)
    - Cercopithecidae (1)
      - Papio (1)
        - P. anubis (1)

Wishes

Let’s be a little more specific in our requirements. Unless otherwise noted, the current implementation fulfill the following:

• n-level: There should be no hard limit on the depth of the hierarchy. The current implementation works well to at least level 30 and there are some obvious optimizations that could be done to get better performance for deeper levels.
• Diverse levels: There should be no requirements for the documents to specify tags to a common level: Having document 1 state the tag “A/B” and document 2 state “F/G/H/I/J” should work.
• Multi-value: Each document should be capable of stating zero or more hierarchical tags.
• Minimum index-impact: The index should not grow significantly, neither should the tags require extended pre-processing to work. The current implementation requirement is that tags are stated in a field with a delimiter between levels, such as “mytag:foo/bar/zoo”.
• Summing: The reference counts for sub-tags under a given tag should be summed recursively and provided in the faceting result.
• Sorting: It should be possible to control the sorting on the different levels, e.g. sorting alphanumerically on the first level, then by reference count on the second and again alphanumerically of all subsequent levels. The current implementation allows for custom sorting in the inner workings but the interface only provides index-order, locale-based and count-order out of the box.
• Output control: It should be possible to limit the output in different ways as hierarchies tend to grow big. The current implementation allows limiting on the depth (levels) as well as the width (sub-tags under a given tag).
• Filtering: If should be possible to specify thresholds for which tags are relevant. This should be done on a level-basis, so the requirement for level 1 could be a total count of at least 100, while the requirement for level 2 could be 10.
• Starting path: To facilitate a dynamic interface, it should be possible to request a hierarchical tag structure starting at a given point in the hierarchy. This has not been implemented yet.
• Low memory impact: As always, we want the impact on memory to be minimized. This is a fuzzy requirement. The current implementation keeps true to the original premises and adds 2 * log2(maxdepth+1) bits for each tag on top of the standard faceting overhead. At extraction time, the temporary memory requirements grow linearly with the number of unique tags to return on all levels.
• Fast: Another fuzzy requirement. The current implementation takes n time for pre-processing, where n is the number of tags: All it does is split each tag into sub-parts, count the parts and make a comparison to the previous tag. For constructing the result, it has a worst-case requirement of n * maxdepth + n * log(n). Some clever caching should be possible, but is is not trivial.
• What else? It would be surprising if this list were complete. Suggestions are welcome.

Experimental setup

An index builder were created that takes 3 parameters: The number of documents, the maximum number of tags at any given level for any given document and the maximum number of levels. Each tag is a random character from A-Z. Asking for 3 documents with a maximum of 2 tags and maximum level 3 might give

Doc 1: A, A/F, A/T, A/T/R, A/T/Y, F/G/I
Doc 1: Z
Doc 2: F/H, F/H/I, B, B/B

For each test, a search was performed that hits all documents and the faceting system extracts a result to XML. This was all on the local machine with direct Java-calls (no web services or similar overhead). The hardware was a Dell M6500 laptop with Intel i7 CPU, PC1333 RAM and SSD.

Some measurements

Some documents, small hierarchy

1,000,000 documents, max tags/level 4, max levels 3.
Request a maximum of 5 tags/level down to level 5 in reference count order.

Hierarchical facet startup time = 1642 ms
18,278 unique tags, 2,383,248 references to tags
Mem usage: preFacet=4 MB, postFacet=9 MB
Collection and extraction #0 for 1000000 documents in 171 ms
Generated XML with 13594 characters

Few documents, wide hierarchy

10,000 documents, max tags/level 10, max levels 4.
Request a maximum of 5 tags/level down to level 5 in reference count order.

Hierarchical facet startup time = 0:06 minutes
438,746 unique tags, 1,478,224 references to tags
Mem usage: preFacet=3 MB, postFacet=9 MB
Collection and extraction #0 for 10000 documents in 169 ms
Generated XML with 70046 characters

Many documents, wide hierarchy, extreme references

10,000,000 documents, max tags/level 10, max levels 4
Request a maximum of 5 tags/level down to level 5 in reference count order.

Hierarchical facet startup time = 4:50 minutes
475,254 unique tags,  1,430,592,254 references to tags.
Mem usage: preFacet=26 MB, postFacet=3319 MB
Collection and extraction #0 for 10000000 documents in 0:21 minutes

Few documents, deep hierarchy

10,000 documents, max tags/level 3, max levels 15.
Request a maximum of 5 tags/level down to level 5 in reference count order.

Hierarchical facet startup time = 6:57 minutes
50,538,133 unique tags, 52,237,029 references to tags
Mem usage: preFacet=61 MB, postFacet=603 MB
Collection and extraction #0 for 10000 documents in 0:10 minutes
Generated XML with 196739 characters

Shallow, single-tag, uniform

Special index to compare to Solr hierarchical faceting: 260,000 documents, each with a single unique two-level tag: A/1, A/2 … A/10000, B/1, B/2 … Z/10000.
Request unlimited tags down to level 5 (effectively unlimited with this corpus).

Hierarchical facet startup time = 2196 ms
260,000 unique tags, 260,000 references to tags
Mem usage: preFacet=2 MB, postFacet=37 MB
Collection and extraction #0 for 260000 documents in 1048 ms
Collection and extraction #1 for 259630 documents in 884 ms
Collection and extraction #2 for 258829 documents in 995 ms
Generated XML with 15316147 characters in 320 ms

More realistically, setting the retrieval limit to the top 10 tags we get

Collection and extraction #0 for 260000 documents in 167 ms
Collection and extraction #1 for 259912 documents in 76 ms
Collection and extraction #2 for 258982 documents in 60 ms
Generated XML with 27347 characters in 7 ms

Sample output

<facetresponse xmlns="http://lucene.apache.org/exposed/facet/response/1.0"
  query="even:true" hits="9984" countms="5" countcached="false" totalms="34">
  <facet name="hierarchical" fields="deep" order="count" maxtags="20"
  mincount="0" offset="0" hierarchical="true" potentialtags="89534"
  count="7589" totalCount="141989" totaltags="26">
      <tag count="320" totalcount="6304" term="Y">
        <subtags potentialtags="3987" count="674" totalCount="5984" totaltags="26">
          <tag count="17" totalcount="301" term="K">
            <subtags potentialtags="222" count="72" totalCount="284" totaltags="26">
              <tag count="4" totalcount="22" term="G">
                <subtags potentialtags="16" count="18" totalCount="18" totaltags="15">
                  <tag count="2" totalcount="2" term="A"></tag>
                  <tag count="2" totalcount="2" term="J"></tag>
                  <tag count="2" totalcount="2" term="U"></tag>
                  <tag count="1" totalcount="1" term="B"></tag>
                  <tag count="1" totalcount="1" term="C"></tag>
                </subtags>
              </tag>
              <tag count="3" totalcount="22" term="Q">
                <subtags potentialtags="19" count="19" totalCount="19" totaltags="18">
                  <tag count="2" totalcount="2" term="Q"></tag>
                  <tag count="1" totalcount="1" term="A"></tag>
                  <tag count="1" totalcount="1" term="B"></tag>
                  <tag count="1" totalcount="1" term="C"></tag>
                  <tag count="1" totalcount="1" term="D"></tag>
                </subtags>
...

Download

Important: This is not usable for production, partly because Lucene 4 is still in development, partly because it was hacked together in a week.

Do a SVN checkout of Lucene trunk and patch it with LUCENE-2369, then fix the errors as Lucene trunk is a moving target (I should really make a compiled version with a command line test tool). The test-code for the hierarchical faceting is located in the aptly named class TestHierarchicalFacets.

Use only for good.
Toke Eskildsen
statsbiblioteket.dk

Classification
  - Primates (9)
    - Atelidae (2)
      - Atelinae (2)
        - Ateles (2)
    - Cebidae (5)
      - Cebus (2)
        - C. apella (1)
        - C. capucinus (1)
      - Saimiri (3)
        - S. sciureus (3)
    - Cercopithecidae (1)
      - Papio (1)
        - P. anubis (1)

Example data and goal

We consider three documents, each with some multi-level faceting information

Doc 1: A/B/C, D/E/F
Doc 2: A/B/C, A/B/J
Doc 3: A,     D/E,   G/H/I

The multi-level tags are thus

A
A/B/C
A/B/J
D/E
D/E/F
G/H/I

Visually we want all levels to be expanded and counted so that we see

A      (4)
 - B   (3)
   - C (2)
   - J (1)
D      (2)
 - E   (2)
   - F (1)
G      (1)
  - H  (1)
   - I (1)

for a search that matches all 3 documents.

The easy, non-fast and non-light solution

If the documents were expanded, either at index time or facet structure build time, to have the tags

Doc 1: A,     A/B,   A/B/C, D,     D/E,   D/E/F
Doc 2: A,     A/B,   A/B/C, A,     A/B,   A/B/J
Doc 3: A,     D,     D/E,   G,     G/H,   G/H/I

then a standard single-level faceting system would produce the desired output (a small lie as we also need to limit the number of tags returned in a level-aware way). Unfortunately this expansion does not come cheap. In this example we go from 7 tag references to 18 and from 6 unique tags to 10. This is not too bad, but when we have deeper levels and more diverse tags, the overhead increases: Faceting on house location with country/state/city/street/number would result in 5 times the references of non-hierarchical faceting. More references means more processing time and more memory overhead.

The hard, fast, light and purple solution

We do not create new tags from the existing tags

Doc 1: A/B/C, D/E/F
Doc 2: A/B/C, A/B/J
Doc 3: A,     D/E,   G/H/I

Instead we augment each tag in the original list with two pieces of information: The level L in the hierarchy (starting at level 1) and the previous level P that the tag T matches. C is the count from a given search. Our list is thus enriched to the following, where the text in the parentheses explain how the previous level was derived:

L P T      C
1 0 A     (1)   (no previous tag)
3 1 A/B/C (2)   (The previous tag "A" matches only the first level of "A/B/C")
3 2 A/B/J (1)   (The previous tag "A/B/C" matches first and second level of "A/B/J")
2 0 D/E   (1)   (no previous match)
3 2 D/E/F (1)   (The previous tag "D/E" matches both the first and second level of "D/E/F")
3 0 G/H/I (1)   (no previous tag)

As both the level L and the previous level P are at most max-depth, we can represent the numbers in a packed structure, where the extra information for each tag takes up 2 * log2(maxdepth+1) bits. For 10 million tags of a maximum depth of 7, this is 10M * 2 * log(8) bits = 10M * 2 * 3 / 8 bytes = 7 MB. 50 million tags of max depth 30 would take 59 MB of extra overhead.

The tag counts are updated just as for standard single-level faceting. The tags at any given level is those that satisfies L >= level, P < level. The counts for any given level is a sum of the counts from the starting point until the next tag matching the equation. Let’s walk through the example:

At level 1, the equation is L >= 1, P < 1. For tag A this is 1 >= 1, 0 < 1 which clearly matches. Next match is tag D/E and the last is G/H/I (remember that counts are summed between matches). As we know that we’re extracting at level 1, we can split the terms for D/E and G/H/I to get D and G and thus have

A (4)
D (2)
G (1)

At level 2, we use the starting points for the tags from level 1 and the equation is L >= 2, P < 2. Tag A no longer matches, but A/B/C does with 3 >= 2, 1 < 2. Tag A/B/J does not, so its count is added to A/B/C. D/E matches with 2 >= 2, 0 < 2, but D/E/F does not with 3 >= 2, 2 < 2 so it only adds to the count for D/E. Last fit is G/H/I with 3 >= 2, 0 < 2. Extracting the proper level 2 terms the result is expanded to

A   (4)
A/B (3)
D   (2)
D/E (2)
G   (1)
G/H (1)

At level 3, we use the staring points from the tags from level 2 and the equation L >= 3, P < 3. Tag A/B/C still matches with 3 >= 3, 1 < 3 and A/B/J matches too. Tag D/E does not with 2 >= 3, 0 < 2. Next match if D/E/F with 3 >= 3, 2 < 3 and last is G/H/I with 3 >= 3, 0 < 3. Extracting at level 3, the tags are expanded to

A     (4)
A/B   (3)
A/B/C (2)
A/B/J (1)
D     (2)
D/E   (2)
D/E/F (1)
G     (1)
G/H   (1)
G/H/I (1)

For brevity, some devils has been skipped here.

• To fit sub-tags under the correct tags, a recursive descend in a sub-set of the tags should be used.
• To avoid re-calculating sums, there should be a counting pass at first.
• To provide mixed count and custom sorting of tags, an expanded facet query syntax must be provided which works with a custom sorter.

The time spend on calculating the facets is dictated by the query options. Worst case is 100% orthogonal tags, tag sorting by counting and a complete extraction of all results. In that case the tag-list must be iterated as many times as there are levels. For the location-example above, this is 5 times, which coincidentally is the same factor as the tags-explosion for the easy implementation. Back-of-the-envelope, the worst case execution time is thus on par with the easy implementation, while memory overhead is a lot less.

As soon at the tags begin to overlap at some levels, sum count caching kicks in. Coupled with a limit on the number of extracted tags at any given level and the fact that we only need to re-visit the sub-tags for weeded super-tags, the number of passes falls drastically. This brings the expected execution time down to less than the easy method.

Fast (cpu-cycles), light (1 byte/unique tag on top of traditional faceting as a rule of thumb) and … Okay, the purple part was a lie. Now it just needs to be implemented.

That’s all, folks!

Laying the ground

First of all, let us describe three common ways of looking at an index.

• Sorting: Sorting on title, author and similar fields should be done with respect to the locale of the user. This is often done by comparing the relevant strings from a search result to each other, which scales horribly when the number of hits rises as locale-aware comparison of Strings is expensive. Another way that is making the rounds is to index collator keys, which are not usable for human reading, but very fast for comparison.
• Faceting: Fairly standard by now, faceting does present some architectural challenges. Creating facets for subjects with a low number of possible values is simple enough, but scaling to faceting on title, author or similar fields with many unique value takes its toll on memory if all the values are kept there. Solr provides an UnInvertedField which offloads most of the terms to storage.
• Index lookup: Also known as “author browse” and probably some other names, as it is not that well-known. The idea is to provide the user with an authoritative list of possible values for a given field. When the user starts to type, valid values with the typed prefix is shown together with a few of the previous values as well as the following values. The difference from the well-known suggestion concept is that index lookup is complete and that the result is in locale-aware sorted order. It can be seen as a dictionary. If memory is plentiful, a simple implementation method is to hold a sorted list of terms in memory and do a standard locale aware binary search for the entry point in the list.

One common element for these three ideas is the terms. Terms kept in memory. With lower end hardware or higher end index size, this is a problem. Lucene (and thereby Solr) is switching to something called BytesRef, which can represent characters in a more compact way than Java Strings. They really do help a lot, but for some that’s not enough.

Now, a Lucene index naturally contains all these terms: They can be looked up, they can be sequentially accessed or (dramatic pause) they can be accessed by ordinal. An ordinal is just an integer and takes up very little memory to represent. Requesting a term by ordinal is a bit costly though, as it must be fetched from storage, which requires some seeking.

Another way

Going back to the three ideas, another common element is that only a relative small amount of the terms is actually shown to the user. For sorting, it might be the top 20 titles. For faceting, it might be 200 tags shared among 10 facets. For index lookup it might be 10 author names. Fast resolving of terms is not required!

Armed with this knowledge, let’s define four building blocks.

• Ordinal to term lookup: Given an ordinal, the term (aka BytesRef) is returned by querying the index. Trivial for single segments, fairly simple for multiple segments as they are just appended. This is just a logical mapping and requires practically no memory.
• Indirects: The ordinals are sorted, typically with respect to a locale, and the sorted lists is called the indirects list. If an index in the indirect is lower than another, it means that its corresponding term comes before the other indirect entry’s term with respect to sorting. We always need to sort in order to have indirects, even if the terms in the segments are already in order. This is because an index is often composed of several segments that most probably contains duplicate terms. In that case, the sorting can also be seen as de-duplication on ordered lists. This works very well with the collator key idea from above.
• Document to single indirect For each document id (still just an integer), an indirect index is kept. Resolving the indirect returns a term corresponding to the document. Memory wise this requires a list of integers as long as the number of documents.
• Document to indirects For each document id, a list of corresponding indirects is kept. By following the indirects through the ordinals, the corresponding terms can be resolved. Memory wise this requires a list of integers as long as the number of documents plus a list of integers as long as the total number of indirects for all documents.

The common element here is arrays of integers. Java’s int[] is not that memory hungry to start with, but we can reduce the impact even further by using the more compact representation PackedInts. Revisiting the three common ways of looking at an index with the new structures, we see how it all comes together.

• Sorting: The document to single indirect is used for this as the order of two documents can be determined by comparing their respective entries in the integer array. This is very fast.
• Faceting: The document to indirects is used for faceting. A counting array of length #indirects is made and the documents ids are iterated. For each id, the indirects are looked up and the corresponding counter entries are incremented. As it is just a matter of iteration and array lookups and updates, this is very fast. After that, the most popular entries in the counter lists can be extracted and the terms resolved. As the indirects can be ordered, it is also possible to extract by lexicographical order or similar.
• Index lookup: The indirects are immediately usable for this as a binary search can be performed. For each iteration of the binary search, one actual term needs to be looked up in storage and compared to the given prefix, but even with 10 million terms, this requires only 24 lookups. Most of these lookups will normally be disk cached.
• Enhanced index lookup: By using faceting for index lookup, we gain the advantage of showing how many documents will match the displayed terms for a given field. Furthermore this can be coupled with previous selections or a query, limiting the displayed terms to those that will result in at least one hit for a search. This might sound like heavy processing, but luckily the counters from the faceting can be easily cached. While an initial faceting of 5 million tags takes about 1 second on a laptop anno 2010, subsequent index lookups within that structure takes only 1-3 milliseconds.

An example

2 test indexes were created with 1 and 10 million documents respectively. Each document had an id, a title (10 random characters), an author (10 random characters) and 0-5 sample-tags which were a single term A-Y. For each index, a search was performed that hit half of the available documents and sorting (Lucene term order on author), faceting (locale “da” for the title, count for the title (yes, again) and count for the sample-tags) and index lookup (on title with locale “da”) was performed. After that a search that hit 1/10 of the index (random distribution) was performed with the same sorting, faceting and index lookups as before.

1 million documents, collator sort

Index = /home/te/projects/index1M (1000000 documents)
Used heap after loading index and performing a simple search: 5 MB
Maximum possible memory (Runtime.getRuntime().maxMemory()): 88 MB

First natural order sorted search for "even:true" with 500000 hits: 2473 ms
Subsequent 5 sorted searches average response time: 23 ms

Facet pool acquisition for for "even:true" with structure groups(
  group(name=sorted, order=locale, locale=da, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 1:20 minutes
First faceting for even:true: 128 ms
Subsequent 4 faceting calls (count caching disabled) response times: 75 ms

Initial lookup pool request (might result in structure building): 0:33 minutes
First index lookup for "even:true": 56 ms
Subsequent 91 index lookups average response times: 0 ms

First natural order sorted search for "multi:A" with 95222 hits: 6 ms
Subsequent 5 sorted searches average response time: 5 ms

Facet pool acquisition for for "multi:A" with structure groups(
  group(name=sorted, order=locale, locale=da, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 0 ms
First faceting for multi:A: 23 ms
Subsequent 4 faceting calls (count caching disabled) response times: 23 ms

Initial lookup pool request (might result in structure building): 0 ms
First index lookup for "multi:A": 42 ms
Subsequent 91 index lookups average response times: 0 ms

Free memory with sort, facet and index lookup structures intact: 68 MB
Total test time: 1:57 minutes

10 million documents, collator sort

Index = /home/te/projects/index10M (10000000 documents)
used heap after loading index and performing a simple search: 25 MB
Maximum possible memory (Runtime.getRuntime().maxMemory()): 910 MB

First natural order sorted search for "even:true" with 5000000 hits: 0:21 minutes
Subsequent 5 sorted searches average response time: 227 ms

Facet pool acquisition for for "even:true" with structure groups(
  group(name=sorted, order=locale, locale=da, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 10:08 minutes
First faceting for even:true: 934 ms
Subsequent 4 faceting calls (count caching disabled) response times: 897 ms

Initial lookup pool request (might result in structure building): 3:21 minutes
First index lookup for "even:true": 348 ms

First natural order sorted search for "multi:A" with 947218 hits: 99 ms
Subsequent 5 sorted searches average response time: 45 ms

Facet pool acquisition for for "multi:A" with structure groups(
  group(name=sorted, order=locale, locale=da, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 0 ms
First faceting for multi:A: 267 ms
Subsequent 4 faceting calls (count caching disabled) response times: 233 ms

Initial lookup pool request (might result in structure building): 0 ms
First index lookup for "multi:A": 99 ms
Subsequent 91 index lookups average response times: 2 ms

Used memory with sort, facet and index lookup structures intact: 685 MB
Total test time: 14:00 minutes

As can be seen, the initialization time is very high with many documents. Using a hybrid approach with indexed collator keys helps a lot but is – for now – incompatible with lexicographically sorted faceting and index lookup as they require that human readable terms are returned. This can be solved without extra memory overhead by having parallel structures in the index for the terms and the collator keys for the terms, but the code has not been written yet. Simulating the parallel structures idea by sorting in term natural order, we get the following measurements.

1 million documents, term natural order sort

Index = /home/te/projects/index1M (1000000 documents)
used heap after loading index and performing a simple search: 6 MB
Maximum possible memory (Runtime.getRuntime().maxMemory()): 88 MB

First natural order sorted search for "even:true" with 500000 hits: 2586 ms
Subsequent 5 sorted searches average response time: 23 ms

Facet pool acquisition for for "even:true" with structure groups(
  group(name=sorted, order=index, locale=null, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 0:44 minutes
First faceting for even:true: 120 ms
Subsequent 4 faceting calls (count caching disabled) response times: 66 ms

Initial lookup pool request (might result in structure building): 0:15 minutes
First index lookup for "even:true": 26 ms
Subsequent 91 index lookups average response times: 0 ms

First natural order sorted search for "multi:A" with 95222 hits: 6 ms
Subsequent 5 sorted searches average response time: 4 ms

Facet pool acquisition for for "multi:A" with structure groups(
  group(name=sorted, order=index, locale=null, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 0 ms
First faceting for multi:A: 22 ms
Subsequent 4 faceting calls (count caching disabled) response times: 20 ms

Initial lookup pool request (might result in structure building): 0 ms
First index lookup for "multi:A": 7 ms
Subsequent 91 index lookups average response times: 0 ms

Used memory with sort, facet and index lookup structures intact: 66 MB
Total test time: 1:03 minutes

10 million documents, term natural order sort

Index = /home/te/projects/index10M (10000000 documents)
used heap after loading index and performing a simple search: 25 MB
Maximum possible memory (Runtime.getRuntime().maxMemory()): 910 MB

First natural order sorted search for "even:true" with 5000000 hits: 0:22 minutes
Subsequent 5 sorted searches average response time: 229 ms

Facet pool acquisition for for "even:true" with structure groups(
  group(name=sorted, order=index, locale=null, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 4:46 minutes
First faceting for even:true: 865 ms
Subsequent 4 faceting calls (count caching disabled) response times: 788 ms

Initial lookup pool request (might result in structure building): 1:41 minutes
First index lookup for "even:true": 318 ms
Subsequent 91 index lookups average response times: 2 ms

First natural order sorted search for "multi:A" with 947218 hits: 101 ms
Subsequent 5 sorted searches average response time: 51 ms

Facet pool acquisition for for "multi:A" with structure groups(
  group(name=sorted, order=index, locale=null, fields(a)),
  group(name=count, order=count, locale=null, fields(a)),
  group(name=multi, order=count, locale=null, fields(facet))): 0 ms
First faceting for multi:A: 220 ms
Subsequent 4 faceting calls (count caching disabled) response times: 220 ms

Initial lookup pool request (might result in structure building): 0 ms
First index lookup for "multi:A": 92 ms
Subsequent 91 index lookups average response times: 2 ms

Used memory with sort, facet and index lookup structures intact: 648 MB
Total test time: 6:58 minutes

Observations

It seems obvious that using indexed collator keys is the way to go. It is significantly faster. The drawbacks are slightly increased indexing time and increased index size.

The current proof of concept plays nice with reopening of the index. This means that an update to the index only requires a partial refresh of the overall structure. In order to do so, some intermediate structures needs to be held in memory. Disabling this caching makes reopening just as expensive as the initial open, but frees some memory. The amount of freed memory has not been measured yet, but I would guess about 1/4-1/3 of total memory usage, depending on the index.

In perspective

The sorting part in itself required about 1/3 of the size needed by standard Lucene natural order sort. It is more difficult to compare the faceting and index lookup as Lucene does not provide either. Solr handles faceting, so I hope to find the time to make a comparison with that product.

To boil it down, the proof of concept makes it possible to provide faceting on authors, sorting on title and a bit on the side for an index with 20 million documents on a 2 GB machine. Scaling up is theoretically n log(n) in time but less than that in the tests above, probably due to caching and JITting.

Try it for yourself

Remember, this is a proof of concept. Check out Lucene trunk and apply the patch at LUCENE-2369. This contains an interactive proof of concept for the sorting part and a non-interactive unit test that builds an index and performs faceting and index lookup. Look in the “exposed”-folders in src and test.

Digital Resources Dep. People report from ECDL2010

Have been meaning to write about our fascinating conference experience for days! This is it.

The first keynote was by Susan Dumais from Microsoft Research on a study in retrieval of dynamic web information. It was interesting to hear about users reusing the same queries and revisiting the same pages but in different re-visitation patterns (fast, hybrid, medium, slow). Esben has some very nice drawings in his notes Also 43% of search activity is repeat clicks or repeat queries. Also have handwritten notes with nice table. The DiffIE plug-in highlights the changes since the user last visited a page, and the study showed that users noticed more changes using DiffIE and revisited pages more. The DiffIE plug-in highlights the changes since the user last visited a page, and the study showed that users noticed more changes using DiffIE and revisited pages more. This could also be used to see new results on the same query in a search interface.

After the keynote, Esben went to the session on metadata and Bolette went to a steering committee meeting.

Metadata Session

User contributed descriptive metadata for libraries and cultural institutes:
Easing user interaction by using known “portals” for allowing users to contribute knowledge.
Exposing hidden collections, with verry little or no meta data to the public, in order to use crowd sourcing to enrich the data.
report
four categories
1.Personal & historical
2.Link out
3.corrections & translations
4.link in

Crowd cataloguing NOT professional, will be entered into meta data with flickr.com as source… (is flickr.com known in 100 years)

Steering Committee Meeting

The SC meeting raised some interesting discussions. The TPDL charter discussion was mostly on organisational details. The discussion on how to get good quality papers in an interdisciplinary conference was more interesting. The challenge in cross-field conferences is that we try to write papers, that can be read by everyone in the community. This means that we leave out the important technical parts specific to our own field and the paper ends up more superficial and somehow less scientific… We also talked about introducing multiple themes in next years conference: TPDL2011. And about a number of other things…

Next there was a panel session.

Panel session

Developing services to support research data management and sharing

Mandatory data management plan in research grant applications, are more and more common. There is a need for a “creative commons” approach to the curation and preservation of research data.
There are trust issues in who has the data that is curated, and preserved. Both in relation to confidentiality and the data being preserved.

Transparency in data records.
Peer review hardly ever touches on the gathering of data and it’s relation to the results…
What is being done to alleviate the sharing of person-sensitive data? not much at present and it is beginning to become an issue, recommendations are to consider this early on in the grant application process.
Measures mus be taken in order to insure consent for data sharing and archiving! At present ethic advisors recommend destroying the datasets upon the end of person-sensitive research.
up front prepare for sharing!

Problems exist in the preservation and curation of datasets, how will the “user guide” to the data be preserved in complex datasets.

This area needs to write and talk about the successes of data sharing.

Evening Program

After the panel it was time for an extended coffee break, meeting an old friend and relaxing before the evening program. The welcome reception at Glasgow City Chambers was amazing! Well, the reception was nice as receptions are. Talked to nice people, had a bit of wine, listened to an official city representative welcoming the conference to his city and talked to more nice people, but the building was absolutely amazing! And the nice Indian student helper gave us directions and reserved a table at a nearby Indian restaurant for us, and we had a great dinner!

Wednesday

The second keynote by Liina Munari from the European Commission was on EU-supported DL projects and included a lot of slides with a lot of bullet points…

Next Bolette went to the Digital Preservation session and Esben went to the Web 2.0 session.

Digital Preservation Session

The first presentation in the digital preservation session was about Hoppla, which is a system designed to help small institutions with logical preservation:

Then there was a presentation on estimating digitization costs and one on a vocabulary for preserved new media art. New media art with both digital and physical elements or maybe interactive present interesting preservation challenges, and maybe with this media rather than preservation we should talk about facilitating recreation or re-performance.

Web 2.0 session

Privacy aware folksonomies

The problem regarding privacy in folksonomies are, to a large extent, related to the fact that the provider has all the knowledge about all the users and all their tags. This issue can become a real privacy nightmare when tags are analysed in order to associate users in groups, it is likely that all users who tagged images with “bobs_wedding" also attended said wedding, imagination and varying degrees of evil seems to be the limiting factor in the possibilities of data mining.

A possible solution combining cryptography and data split over several databases was shown, this allowed a granularity of access, between users, friends and providers.

SEAmless WEb EDiting for curated content SEAWEED

A project to release the user from the chains of the edit->review->publish process, it’s basically just a new take on web editing, but looks promising, especially in conjunction annotation.
Details are available here(WP plugin) and here.

Poster Session

Before lunch the somewhat nervous Esben did a very good poster spotlight presentation ending with “And this was the…” Look at watch… “50 second presentation of my poster.” And he did get the audience to laugh After lunch was the poster session and quite a few people came to talk to Esben about his work. Some people also came to talk about his fascinating fabric poster, especially the people who were presenting posters themselves and had been travelling with big paper posters rolled up. Bolette thinks Esben got a number of good ideas for further work but unfortunately has too little time… Bolette also went to see some of the other posters and talk to some people.

Once again the late afternoon session lost to the need for relaxing. We went shopping on the way back to the hotel. Whisky, book, dvd. Slept for 45 minutes and walked via the botanical garden to the conference dinner, which was at an old church converted to restaurant complete with whisky bar in the cellar. It was very impressive and we talked about whether it would be a good idea for some Danish churches, since we don’t seem to be using them all that much for their intended purpose anymore. Oh, and the bagpipe was fun and the food was good.

Thursday

By Thursday everyone was tired and the Theatre 1 didn’t seem quite as full as the previous days. We went to the session on query log analysis and the one ontologies and TPDL2011 was announced. And then we used 11 hours travelling home!

Query log analysis session.

Determining time of queries for search of re-ranking

This talked was focused about how the time of search and the implicit information in the search terms can be analysed and used in retrieving the correct data entries, the talk focused on how the terminology changes over time. The term “Hillary Cllinton 1997-2002” should return many of the same results as “New York Senator” and “First Lady“.
The team had found that of searches 1,5% eg. “Presidential election 2008” uses explicit time queries, and 7% use implicit time queries eg. “German world cup“
As many events occur at a specific time, or at least over short periods, the researchers introduced Time granularity which returned relevant search results for the period of granularity [01/01/2004;01/01/2005].
Questions were asked regarding problems with events at or near the borders of time granular, this was an issue they where aware of, a possible solution could be double overlapping time granulars [01/01/2004;01/01/2005] and [01/06/2004;01/06/2005] allowing for inclusion of search results otherwise omitted.

Query transformation in cultural metadata.

User contributed descriptive metadata for libraries and cultural institutions.