KItinerary::ExtractorEngine
#include <extractorengine.h>
Public Types | |
enum | Hint { NoHint = 0, ExtractFullPageRasterImages = 1, ExtractGenericIcalEvents = 2 } |
typedef QFlags< Hint > | Hints |
Public Member Functions | |
ExtractorEngine (const ExtractorEngine &)=delete | |
ExtractorEngine (ExtractorEngine &&) noexcept | |
const BarcodeDecoder * | barcodeDecoder () const |
void | clear () |
const ExtractorDocumentNodeFactory * | documentNodeFactory () const |
QJsonArray | extract () |
Hints | hints () const |
void | setAdditionalExtractors (std::vector< const AbstractExtractor * > &&extractors) |
void | setContent (const QVariant &data, QStringView mimeType) |
void | setContext (const QVariant &data, QStringView mimeType) |
void | setContextDate (const QDateTime &dt) |
void | setData (const QByteArray &data, QStringView fileName={}, QStringView mimeType={}) |
void | setHints (Hints hints) |
void | setUseSeparateProcess (bool separateProcess) |
QString | usedCustomExtractor () const |
Detailed Description
Semantic data extraction engine.
This will attempt to find travel itinerary data in the given input data (plain text, HTML text, PDF documents, etc), and return the extracted JSON-LD data.
Creating Extractors
Extractor API
For adding custom extractors, two parts are needed:
- JSON meta-data describing the extractor and when to apply it, as described in the Extractor documentation.
- An extractor JavaScript file, compatible with QJSEngine.
The extractor script will have access to API defined in the JsApi namespace:
- JsApi::Context: information about the input data being processed.
- JsApi::JsonLd: functions for generating JSON-LD data.
- JsApi::Barcode: barcode decoding functions.
The entry point to the script is specified in the meta-data, its argument depends on the extractor type:
- Plain text extractors are passed a string. If input is HTML or PDF, the string will be the text of the document stripped of all formatting etc.
- HTML extractors are passed a HtmlDocument instance allowing DOM-like access to the document structure.
- PDF extractors are passed a PdfDocument instance allowing access to textual and image content.
- Apple Wallet pass extractors are passed a KPkPass::BoardingPass instance.
- iCalendar event extractors are passed KCalendarCore::Event instances.
These functions should return an object or an array of objects following the JSON-LD format defined on schema.org. JsApi::JsonLd provides helper functions to build such objects. If null
or an empty array is returned, the next applicable extractor is run.
Returned objects are then passed through ExtractorPostprocessor which will normalize, augment and validate the data. This can greatly simplify the extraction, as for example the expansion of an IATA BCBP ticket token already fills most key properties of a flight reservation automatically.
Development Tools
For interactive testing during development of new extractors, it is recommended to link (or copy) the JSON meta data and JavaScript code files to the search path for Extractor meta data.
Additionally, there's an interactive testing and inspection tool called kitinerary-workbench
(see https://invent.kde.org/pim/kitinerary-workbench).
Automated Testing
There are a few unit tests for extractors in the kitinerary repository (see autotests/extractordata), however the majority of real-world test data cannot be shared this way, due to privacy and copyright issues (e.g. PDFs containing copyrighted vendor logos and user credit card details). Therefore there is also support for testing against external data (see extractortest.cpp).
External test data is assumed to be in a folder named kitinerary-tests
next to the kitinerary
source folder. The test program searches this folder recursively for folders with the following content and attempts to extract data from each test file in there.
context.eml:
MIME message header data specifying the context in which the test data was received. This typically only needs aFrom:
andDate:
line, but can even be entirely empty (or non-existing) for structured data that does not need a custom extractor. This context information is applied to all tests in this folder.<testname>
.[txt|html|pdf|pkpass|ics|eml|mbox]: The input test data.<testname.extension>
.json: The expected JSON-LD output. If this file doesn't exists it is created by the test program.<testname.extension>
.skip: If this file is present the corresponding test is skipped.
Definition at line 104 of file extractorengine.h.
Member Enumeration Documentation
◆ Hint
Hints about the document to extract based on application knowledge that can help the extractor.
Enumerator | |
---|---|
ExtractFullPageRasterImages | perform expensive image processing on (PDF) documents containing full page raster images |
ExtractGenericIcalEvents | generate Event objects for generic ical events. |
Definition at line 159 of file extractorengine.h.
Member Function Documentation
◆ barcodeDecoder()
const BarcodeDecoder * ExtractorEngine::barcodeDecoder | ( | ) | const |
Barcode decoder for use by KItinerary::ExtractorDocumentProcessor.
Use this rather than your own instance as it caches repeated attempts to decode the same image.
Definition at line 178 of file extractorengine.cpp.
◆ clear()
void ExtractorEngine::clear | ( | ) |
Resets the internal state, call before processing new input data.
Definition at line 115 of file extractorengine.cpp.
◆ documentNodeFactory()
const ExtractorDocumentNodeFactory * ExtractorEngine::documentNodeFactory | ( | ) | const |
Factory for creating new document nodes.
This is only for use by KItinerary::ExtractorDocumentProcessor instances.
Definition at line 173 of file extractorengine.cpp.
◆ extract()
QJsonArray ExtractorEngine::extract | ( | ) |
Perform the actual extraction, and return the JSON-LD data that has been found.
Definition at line 151 of file extractorengine.cpp.
◆ hints()
ExtractorEngine::Hints ExtractorEngine::hints | ( | ) | const |
The currently set extraction hints.
Definition at line 141 of file extractorengine.cpp.
◆ setAdditionalExtractors()
void ExtractorEngine::setAdditionalExtractors | ( | std::vector< const AbstractExtractor * > && | extractors | ) |
Sets additional extractors to run on the given data.
Extractors are usually automatically selected, this is therefore most likely not needed to be called manually. This mainly exists for the external extractor process.
Definition at line 163 of file extractorengine.cpp.
◆ setContent()
void ExtractorEngine::setContent | ( | const QVariant & | data, |
QStringView | mimeType | ||
) |
Already decoded data to extract from.
- Parameters
-
data Has to contain a object of a supported data type matching mimeType
.
Definition at line 126 of file extractorengine.cpp.
◆ setContext()
void ExtractorEngine::setContext | ( | const QVariant & | data, |
QStringView | mimeType | ||
) |
Provide a document part that is only used to determine which extractor to use, but not for extraction itself.
This can for example be the MIME message part wrapping a document to extract. Using this is not necessary when this document part is already included in what is passed to setContent() already anyway.
Definition at line 131 of file extractorengine.cpp.
◆ setContextDate()
void ExtractorEngine::setContextDate | ( | const QDateTime & | dt | ) |
Set the date the extracted document has been issued at.
This does not need to be perfectly accurate and is used to complete incomplete date information in the document (typically a missing year). This method does not need to be called when setContext is used.
Definition at line 136 of file extractorengine.cpp.
◆ setData()
void ExtractorEngine::setData | ( | const QByteArray & | data, |
QStringView | fileName = {} , |
||
QStringView | mimeType = {} |
||
) |
Set raw data to extract from.
- Parameters
-
data Raw data to extract from. fileName Used as a hint to determine the type, optional and used for MIME type auto-detection if needed. mimeType MIME type of data
, auto-detected if empty.
Definition at line 121 of file extractorengine.cpp.
◆ setHints()
void ExtractorEngine::setHints | ( | ExtractorEngine::Hints | hints | ) |
Set extraction hints.
Definition at line 146 of file extractorengine.cpp.
◆ setUseSeparateProcess()
void ExtractorEngine::setUseSeparateProcess | ( | bool | separateProcess | ) |
Perform extraction of "risky" content such as PDF files in a separate process.
This is safer as it isolates the using application from crashes/hangs due to corrupt files. It is however slower, and not available on all platforms. This is off by default.
Definition at line 158 of file extractorengine.cpp.
◆ usedCustomExtractor()
QString ExtractorEngine::usedCustomExtractor | ( | ) | const |
Returns the extractor id used to obtain the result.
Can be empty if generic extractors have been used. Not supposed to be used for normal operations, this is only needed for tooling.
Definition at line 168 of file extractorengine.cpp.
The documentation for this class was generated from the following files:
Documentation copyright © 1996-2023 The KDE developers.
Generated on Sun Jun 4 2023 04:01:32 by doxygen 1.8.17 written by Dimitri van Heesch, © 1997-2006
KDE's Doxygen guidelines are available online.