Retrieval
=========

After we ingest our higher level objects to ElasticSearch, we can use all of the associated tools that come with
ElasticSearch to retrieve objects.

More recently, two stage retrieval systems, which deploy a deep learning reranking model on top of retrieved document
results, have shown to be better performing for both basic information retrieval and also for downstream tasks such
as question answering.

We also deploy a two stage reranking model. Our reranker uses BERT-Large as its base architecture, and by default is
trained on MS-Marco. On our roadmap is easily training this model on user feedback.

Our setting is slightly different from the traditional document retrieval setting, and also different from settings
such as question answering. Like question answering, we are retrieving relatively short contexts, but the contexts are
composed of not pure text sequences. Also unlike question answering, we are seeking to return interesting information,
not necessarily the specific answer to a user's query.

With this last point in mind, we prioritize diversity in returned PDFs. To do this, we use ElasticSearch to retrieve
a set of N documents, given all the text content in that document. We find all objects of the
type defined by the query on these returned documents. We then run reranking on all these objects.

Instead of returning this reranked list, we choose to filter the list such that only the top ranking object for each
of the initial N documents remains. In this way, we end up with a ranked list of the initial N documents based on
how informative the "most" informative object in that document is.

For pagination, we paginate at the document level. If you retrieve the first 25 documents, you will get a ranked list
of those 25 documents according to ElasticSearch + reranking. If you ask for more results, the next page of objects
will come from the next 25 documents, and thus will have no overlap with the first page of results.

In this way, you can scroll through hundreds of documents, finding fresh relevant objects to explore.




Elasticsearch Index - Fields
============================


+-------------------+---------+----------------------------------------------------------------------------------------+
| Field             | Type    | Description                                                                            |
+===================+=========+========================================================================================+
| Object fields                                                                                                        |
+-------------------+---------+----------------------------------------------------------------------------------------+
| area              | integer | size (in sq. pixels) of identified area                                                |
+-------------------+---------+----------------------------------------------------------------------------------------+
| cls               | text    | detected object class (figure, table, body text, section header, etc)                  |
+-------------------+---------+----------------------------------------------------------------------------------------+
| content           | text    | Text content within the detected object                                                |
+-------------------+---------+----------------------------------------------------------------------------------------+
| context_from_text | text    | Context surrounding mentions of identified object within the body text. (Experimental) |
+-------------------+---------+----------------------------------------------------------------------------------------+
| dataset_id        | text    | (COSMOS internal) - Identifier for dataset.                                            |
+-------------------+---------+----------------------------------------------------------------------------------------+
| detect_score      | float   | Score (strength) of detected classification prediciton                                 |
+-------------------+---------+----------------------------------------------------------------------------------------+
| full_content      | text    | Combined content of the object and its associated parent/header                        |
+-------------------+---------+----------------------------------------------------------------------------------------+
| header_content    | text    | Text content of the associated parent/header object (caption, section header)          |
+-------------------+---------+----------------------------------------------------------------------------------------+
| img_pth           | text    | Path to image file on disk                                                             |
+-------------------+---------+----------------------------------------------------------------------------------------+
| pdf_name          | text    | Name of source PDF                                                                     |
+-------------------+---------+----------------------------------------------------------------------------------------+
| postprocess_score | float   | Confidence of postprocess detection correction process                                 |
+-------------------+---------+----------------------------------------------------------------------------------------+
| Entity fields                                                                                                        |
+-------------------+---------+----------------------------------------------------------------------------------------+
| aliases           | text    | Aliases for known entities                                                             |
+-------------------+---------+----------------------------------------------------------------------------------------+
| canonical_id      | text    | Canonical ID for known entities (e.g. UMLS id)                                         |
+-------------------+---------+----------------------------------------------------------------------------------------+
| description       | text    | Description of known entities                                                          |
+-------------------+---------+----------------------------------------------------------------------------------------+
| name              | text    | Name of known entity                                                                   |
+-------------------+---------+----------------------------------------------------------------------------------------+
| types             | keyword | Type of entity                                                                         |
+-------------------+---------+----------------------------------------------------------------------------------------+