KItinerary::ExtractorEngine

Search for usage in LXR

KItinerary::ExtractorEngine Class Reference

#include <extractorengine.h>

Public Types

enum  Hint { NoHint = 0 , ExtractFullPageRasterImages = 1 , ExtractGenericIcalEvents = 2 }
 
typedef QFlags< HintHints
 

Public Member Functions

 ExtractorEngine (const ExtractorEngine &)=delete
 
 ExtractorEngine (ExtractorEngine &&) noexcept
 
const BarcodeDecoderbarcodeDecoder () const
 
void clear ()
 
const ExtractorDocumentNodeFactorydocumentNodeFactory () const
 
QJsonArray extract ()
 
Hints hints () const
 
void setAdditionalExtractors (std::vector< const AbstractExtractor * > &&extractors)
 
void setContent (const QVariant &data, QStringView mimeType)
 
void setContext (const QVariant &data, QStringView mimeType)
 
void setContextDate (const QDateTime &dt)
 
void setData (const QByteArray &data, QStringView fileName={}, QStringView mimeType={})
 
void setHints (Hints hints)
 
void setUseSeparateProcess (bool separateProcess)
 
QString usedCustomExtractor () const
 

Detailed Description

Semantic data extraction engine.

This will attempt to find travel itinerary data in the given input data (plain text, HTML text, PDF documents, etc), and return the extracted JSON-LD data.

Creating Extractors

Extractor API

For adding custom extractors, two parts are needed:

  • JSON meta-data describing the extractor and when to apply it, as described in the Extractor documentation.
  • An extractor JavaScript file, compatible with QJSEngine.

The extractor script will have access to API defined in the JsApi namespace:

The entry point to the script is specified in the meta-data, its argument depends on the extractor type:

  • Plain text extractors are passed a string. If input is HTML or PDF, the string will be the text of the document stripped of all formatting etc.
  • HTML extractors are passed a HtmlDocument instance allowing DOM-like access to the document structure.
  • PDF extractors are passed a PdfDocument instance allowing access to textual and image content.
  • Apple Wallet pass extractors are passed a KPkPass::BoardingPass instance.
  • iCalendar event extractors are passed KCalendarCore::Event instances.
  • UIC/ERA/VDV/IATA standardized ticket codes are passed as their respective types.
  • Binary data is passed as ArrayBuffer.

These functions should return an object or an array of objects following the JSON-LD format defined on schema.org. JsApi::JsonLd provides helper functions to build such objects. If null or an empty array is returned, the next applicable extractor is run.

Returned objects are then passed through ExtractorPostprocessor which will normalize, augment and validate the data. This can greatly simplify the extraction, as for example the expansion of an IATA BCBP ticket token already fills most key properties of a flight reservation automatically.

Development Tools

For interactive testing during development of new extractors, it is recommended to link (or copy) the JSON meta data and JavaScript code files to the search path for Extractor meta data.

Additionally, there's an interactive testing and inspection tool called kitinerary-workbench (see https://invent.kde.org/pim/kitinerary-workbench).

Automated Testing

There are a few unit tests for extractors in the kitinerary repository (see autotests/extractordata), however the majority of real-world test data cannot be shared this way, due to privacy and copyright issues (e.g. PDFs containing copyrighted vendor logos and user credit card details). Therefore there is also support for testing against external data (see extractortest.cpp).

External test data is assumed to be in a folder named kitinerary-tests next to the kitinerary source folder. The test program searches this folder recursively for folders with the following content and attempts to extract data from each test file in there.

  • context.eml: MIME message header data specifying the context in which the test data was received. This typically only needs a From: and Date: line, but can even be entirely empty (or non-existing) for structured data that does not need a custom extractor. This context information is applied to all tests in this folder.
  • <testname>.[txt|html|pdf|pkpass|ics|eml|mbox]: The input test data.
  • <testname.extension>.json: The expected JSON-LD output. If this file doesn't exists it is created by the test program.
  • <testname.extension>.skip: If this file is present the corresponding test is skipped.

Definition at line 107 of file engine/extractorengine.h.

Member Typedef Documentation

◆ Hints

Member Enumeration Documentation

◆ Hint

Hints about the document to extract based on application knowledge that can help the extractor.

Enumerator
ExtractFullPageRasterImages 

perform expensive image processing on (PDF) documents containing full page raster images

ExtractGenericIcalEvents 

generate Event objects for generic ical events.

Definition at line 162 of file engine/extractorengine.h.

Constructor & Destructor Documentation

◆ ExtractorEngine()

ExtractorEngine::ExtractorEngine ( )

Definition at line 103 of file engine/extractorengine.cpp.

◆ ~ExtractorEngine()

ExtractorEngine::~ExtractorEngine ( )

Definition at line 111 of file engine/extractorengine.cpp.

Member Function Documentation

◆ barcodeDecoder()

const BarcodeDecoder * ExtractorEngine::barcodeDecoder ( ) const

Barcode decoder for use by KItinerary::ExtractorDocumentProcessor.

Use this rather than your own instance as it caches repeated attempts to decode the same image.

Definition at line 180 of file engine/extractorengine.cpp.

◆ clear()

void ExtractorEngine::clear ( )

Resets the internal state, call before processing new input data.

Definition at line 117 of file engine/extractorengine.cpp.

◆ documentNodeFactory()

const ExtractorDocumentNodeFactory * ExtractorEngine::documentNodeFactory ( ) const

Factory for creating new document nodes.

This is only for use by KItinerary::ExtractorDocumentProcessor instances.

Definition at line 175 of file engine/extractorengine.cpp.

◆ extract()

QJsonArray ExtractorEngine::extract ( )

Perform the actual extraction, and return the JSON-LD data that has been found.

Definition at line 153 of file engine/extractorengine.cpp.

◆ hints()

ExtractorEngine::Hints ExtractorEngine::hints ( ) const

The currently set extraction hints.

Definition at line 143 of file engine/extractorengine.cpp.

◆ setAdditionalExtractors()

void ExtractorEngine::setAdditionalExtractors ( std::vector< const AbstractExtractor * > && extractors)

Sets additional extractors to run on the given data.

Extractors are usually automatically selected, this is therefore most likely not needed to be called manually. This mainly exists for the external extractor process.

Definition at line 165 of file engine/extractorengine.cpp.

◆ setContent()

void ExtractorEngine::setContent ( const QVariant & data,
QStringView mimeType )

Already decoded data to extract from.

Parameters
dataHas to contain a object of a supported data type matching mimeType.

Definition at line 128 of file engine/extractorengine.cpp.

◆ setContext()

void ExtractorEngine::setContext ( const QVariant & data,
QStringView mimeType )

Provide a document part that is only used to determine which extractor to use, but not for extraction itself.

This can for example be the MIME message part wrapping a document to extract. Using this is not necessary when this document part is already included in what is passed to setContent() already anyway.

Definition at line 133 of file engine/extractorengine.cpp.

◆ setContextDate()

void ExtractorEngine::setContextDate ( const QDateTime & dt)

Set the date the extracted document has been issued at.

This does not need to be perfectly accurate and is used to complete incomplete date information in the document (typically a missing year). This method does not need to be called when setContext is used.

Definition at line 138 of file engine/extractorengine.cpp.

◆ setData()

void ExtractorEngine::setData ( const QByteArray & data,
QStringView fileName = {},
QStringView mimeType = {} )

Set raw data to extract from.

Parameters
dataRaw data to extract from.
fileNameUsed as a hint to determine the type, optional and used for MIME type auto-detection if needed.
mimeTypeMIME type of data, auto-detected if empty.

Definition at line 123 of file engine/extractorengine.cpp.

◆ setHints()

void ExtractorEngine::setHints ( ExtractorEngine::Hints hints)

Set extraction hints.

Definition at line 148 of file engine/extractorengine.cpp.

◆ setUseSeparateProcess()

void ExtractorEngine::setUseSeparateProcess ( bool separateProcess)

Perform extraction of "risky" content such as PDF files in a separate process.

This is safer as it isolates the using application from crashes/hangs due to corrupt files. It is however slower, and not available on all platforms. This is off by default.

Definition at line 160 of file engine/extractorengine.cpp.

◆ usedCustomExtractor()

QString ExtractorEngine::usedCustomExtractor ( ) const

Returns the extractor id used to obtain the result.

Can be empty if generic extractors have been used. Not supposed to be used for normal operations, this is only needed for tooling.

Definition at line 170 of file engine/extractorengine.cpp.


The documentation for this class was generated from the following files:
This file is part of the KDE documentation.
Documentation copyright © 1996-2024 The KDE developers.
Generated on Sun Feb 25 2024 18:40:32 by doxygen 1.10.0 written by Dimitri van Heesch, © 1997-2006

KDE's Doxygen guidelines are available online.