KItinerary::ExtractorEngine

Search for usage in LXR

KItinerary::ExtractorEngine Class Reference

#include <extractorengine.h>

Public Member Functions

 ExtractorEngine (ExtractorEngine &&) noexcept
 
 ExtractorEngine (const ExtractorEngine &)=delete
 
const BarcodeDecoderbarcodeDecoder () const
 
void clear ()
 
const ExtractorDocumentNodeFactorydocumentNodeFactory () const
 
QJsonArray extract ()
 
void setAdditionalExtractors (std::vector< const AbstractExtractor * > &&extractors)
 
void setContent (const QVariant &data, QStringView mimeType)
 
void setContext (const QVariant &data, QStringView mimeType)
 
void setContextDate (const QDateTime &dt)
 
void setData (const QByteArray &data, QStringView fileName={}, QStringView mimeType={})
 
void setUseSeparateProcess (bool separateProcess)
 
QString usedCustomExtractor () const
 

Detailed Description

Semantic data extraction engine.

This will attempt to find travel itinerary data in the given input data (plain text, HTML text, PDF documents, etc), and return the extracted JSON-LD data.

Creating Extractors

Extractor API

For adding custom extractors, two parts are needed:

  • JSON meta-data describing the extractor and when to apply it, as described in the Extractor documentation.
  • An extractor JavaScript file, compatible with QJSEngine.

The extractor script will have access to API defined in the JsApi namespace:

  • JsApi::Context: information about the input data being processed.
  • JsApi::JsonLd: functions for generating JSON-LD data.
  • JsApi::Barcode: barcode decoding functions.

The entry point to the script is specified in the meta-data, its argument depends on the extractor type:

  • Plain text extractors are passed a string. If input is HTML or PDF, the string will be the text of the document stripped of all formatting etc.
  • HTML extractors are passed a HtmlDocument instance allowing DOM-like access to the document structure.
  • PDF extractors are passed a PdfDocument instance allowing access to textual and image content.
  • Apple Wallet pass extractors are passed a KPkPass::BoardingPass instance.
  • iCalendar event extractors are passed KCalendarCore::Event instances.

These functions should return an object or an array of objects following the JSON-LD format defined on schema.org. JsApi::JsonLd provides helper functions to build such objects. If null or an empty array is returned, the next applicable extractor is run.

Returned objects are then passed through ExtractorPostprocessor which will normalize, augment and validate the data. This can greatly simplify the extraction, as for example the expansion of an IATA BCBP ticket token already fills most key properties of a flight reservation automatically.

Development Tools

For interactive testing during development of new extractors, it is recommended to link (or copy) the JSON meta data and JavaScript code files to the search path for Extractor meta data.

Additionally, there's an interactive testing and inspection tool called kitinerary-workbench (see https://invent.kde.org/pim/kitinerary-workbench).

Automated Testing

There are a few unit tests for extractors in the kitinerary repository (see autotests/extractordata), however the majority of real-world test data cannot be shared this way, due to privacy and copyright issues (e.g. PDFs containing copyrighted vendor logos and user credit card details). Therefore there is also support for testing against external data (see extractortest.cpp).

External test data is assumed to be in a folder named kitinerary-tests next to the kitinerary source folder. The test program searches this folder recursively for folders with the following content and attempts to extract data from each test file in there.

  • context.eml: MIME message header data specifying the context in which the test data was received. This typically only needs a From: and Date: line, but can even be entirely empty (or non-existing) for structured data that does not need a custom extractor. This context information is applied to all tests in this folder.
  • <testname>.[txt|html|pdf|pkpass|ics|eml|mbox]: The input test data.
  • <testname.extension>.json: The expected JSON-LD output. If this file doesn't exists it is created by the test program.
  • <testname.extension>.skip: If this file is present the corresponding test is skipped.

Definition at line 104 of file extractorengine.h.

Member Function Documentation

const BarcodeDecoder * ExtractorEngine::barcodeDecoder ( ) const

Barcode decoder for use by KItinerary::ExtractorDocumentProcessor.

Use this rather than your own instance as it caches repeated attempts to decode the same image.

Definition at line 167 of file extractorengine.cpp.

void ExtractorEngine::clear ( )

Resets the internal state, call before processing new input data.

Definition at line 114 of file extractorengine.cpp.

const ExtractorDocumentNodeFactory * ExtractorEngine::documentNodeFactory ( ) const

Factory for creating new document nodes.

This is only for use by KItinerary::ExtractorDocumentProcessor instances.

Definition at line 162 of file extractorengine.cpp.

QJsonArray ExtractorEngine::extract ( )

Perform the actual extraction, and return the JSON-LD data that has been found.

Definition at line 140 of file extractorengine.cpp.

void ExtractorEngine::setAdditionalExtractors ( std::vector< const AbstractExtractor * > &&  extractors)

Sets additional extractors to run on the given data.

Extractors are usually automatically selected, this is therefore most likely not needed to be called manually. This mainly exists for the external extractor process.

Definition at line 152 of file extractorengine.cpp.

void ExtractorEngine::setContent ( const QVariant data,
QStringView  mimeType 
)

Already decoded data to extract from.

Parameters
dataHas to contain a object of a supported data type matching mimeType.

Definition at line 125 of file extractorengine.cpp.

void ExtractorEngine::setContext ( const QVariant data,
QStringView  mimeType 
)

Provide a document part that is only used to determine which extractor to use, but not for extraction itself.

This can for example be the MIME message part wrapping a document to extract. Using this is not necessary when this document part is already included in what is passed to setContent() already anyway.

Definition at line 130 of file extractorengine.cpp.

void ExtractorEngine::setContextDate ( const QDateTime dt)

Set the date the extracted document has been issued at.

This does not need to be perfectly accurate and is used to complete incomplete date information in the document (typically a missing year). This method does not need to be called when setContext is used.

Definition at line 135 of file extractorengine.cpp.

void ExtractorEngine::setData ( const QByteArray data,
QStringView  fileName = {},
QStringView  mimeType = {} 
)

Set raw data to extract from.

Parameters
dataRaw data to extract from.
fileNameUsed as a hint to determine the type, optional and used for MIME type auto-detection if needed.
mimeTypeMIME type of data, auto-detected if empty.

Definition at line 120 of file extractorengine.cpp.

void ExtractorEngine::setUseSeparateProcess ( bool  separateProcess)

Perform extraction of "risky" content such as PDF files in a separate process.

This is safer as it isolates the using application from crashes/hangs due to corrupt files. It is however slower, and not available on all platforms. This is off by default.

Definition at line 147 of file extractorengine.cpp.

QString ExtractorEngine::usedCustomExtractor ( ) const

Returns the extractor id used to obtain the result.

Can be empty if generic extractors have been used. Not supposed to be used for normal operations, this is only needed for tooling.

Definition at line 157 of file extractorengine.cpp.


The documentation for this class was generated from the following files:
This file is part of the KDE documentation.
Documentation copyright © 1996-2021 The KDE developers.
Generated on Mon Dec 6 2021 23:05:24 by doxygen 1.8.11 written by Dimitri van Heesch, © 1997-2006

KDE's Doxygen guidelines are available online.