Flux can import any type of file as-is, with the contents of the file becoming a new document in MarkLogic. The term “generic files” is used in this context to refer to files that do not require any special processing other than potentially decompressing the files.

Table of contents

Usage

The import-files command imports a set of files into MarkLogic, with each file being written as a separate document. You must specify at least one --path option along with connection information for the MarkLogic database you wish to write to. For example:

  • ./bin/flux import-files \
    --path /path/to/files \
    --connection-string "flux-example-user:password@localhost:8004" \
    --permissions flux-example-role,read,flux-example-role,update

  • bin\flux import-files ^
    --path path\to\files ^
    --connection-string "flux-example-user:password@localhost:8004" ^
    --permissions flux-example-role,read,flux-example-role,update

Controlling document URIs

Each document will have an initial URI based on the absolute path of the associated file. See common import features for details on adjusting this URI. In particular, the --uri-replace option is often useful for removing most of the absolute path to produce a concise, self-describing URI.

Specifying a document type

The type of each document written to MarkLogic is determined by the file extension found in the URI along with the set of MIME types configured in MarkLogic. For unrecognized file extensions, or URIs that do not have a file extension, you can force a document type via the --document-type option. The value of this option must be one of JSON, XML, or TEXT.

Specifying an encoding

MarkLogic stores all content in the UTF-8 encoding. If your files use a different encoding, you must specify that via the --encoding option so that the content can be correctly translated to UTF-8 when written to MarkLogic:

  • ./bin/flux import-files \
    --path source \
    --encoding ISO-8859-1 \
    --connection-string "flux-example-user:password@localhost:8004" \
    --permissions flux-example-role,read,flux-example-role,update

  • bin\flux import-files ^
    --path source ^
    --encoding ISO-8859-1 ^
    --connection-string "flux-example-user:password@localhost:8004" ^
    --permissions flux-example-role,read,flux-example-role,update

Importing large binary files

Flux can leverage MarkLogic’s support for large binary documents by importing binary files of any size. To ensure that binary files of any size can be loaded, consider using the --streaming option introduced in Flux 1.1.0. When this option is set, Flux will stream the contents of each file from its source directly into MarkLogic, thereby avoiding reading the contents of a file into memory.

As streaming a file requires Flux to only send one document at a time to MarkLogic, you should not use this option when importing smaller files that easily fit into the memory available to Flux.

When using --streaming, the following options will have no effect due to Flux not reading the file contents into memory and always sending one file per request to MarkLogic:

  • --batch-size
  • --encoding
  • --failed-documents-path
  • --uri-template

You typically will also not want to use the --transform option as applying a REST transform in MarkLogic to a very large binary document may exhaust the amount of memory available to MarkLogic.

In addition, when streaming documents to MarkLogic, URIs will be encoded. For example, a file named my file.json will result in a URI of /my%20file.json. This is due to an issue in the MarkLogic REST API endpoint that will be resolved in a future server release.

Importing gzip files

To import gzip files with each file being decompressed before written to MarkLogic, include the --compression option with a value of GZIP. You can also import gzip files as-is - i.e. without decompressing them - by not including the --compression option. The --streaming option introduced in Flux 1.1.0 can also be used for very large gzip files that may not fit into the memory available to Flux or to MarkLogic.

Importing ZIP files

To import each entry in a ZIP file as a separate document, include the --compression option with a value of ZIP. Each document will have an initial URI based on both the absolute path of the ZIP file and the name of the ZIP entry. You can also use the --document-type option as described above to force a document type for any entry that has a file extension not recognized by MarkLogic. The --streaming option introduced in Flux 1.1.0 can also be used for ZIP files containing very large binary files that may not fit into the memory available to Flux or to MarkLogic.

Extracting text

As of Flux 1.3.0, text can be extracted from files via Apache Tika and written as separate documents in MarkLogic. This is typically useful when importing binary content such as PDF and Word files, where both the binary file and extracted text can be stored in MarkLogic.

Behavior change in Flux 2.1.2: Microsoft Office file extraction

Flux 2.1.2 upgrades Apache Tika from 3.3.1 to 3.3.2. Tika 3.3.2 changes the default parser for Microsoft Office OOXML files (.docx, .pptx, .xlsx, .vsdx) from a DOM-based extractor to a SAX-based extractor. The SAX parser is faster and more memory-efficient, but may produce slightly different whitespace in extracted text compared to previous Flux versions — for example, paragraph separators may differ.

If your application depends on the exact text output from Office files and you need the previous DOM-based behavior, you can restore it by providing a Tika configuration file that sets useSAXDocxExtractor and/or useSAXPptxExtractor to false on the OfficeParserConfig.

Text extraction is enabled by including the following option when executing the import-files command:

--extract-text

When included, Flux will extract text from each file and create a separate document based on the following default behavior:

  1. The document will be JSON with a URI equalling that of the binary document URI plus -extracted-text.json.
  2. The document will inherit any permissions assigned to the binary document.
  3. The document will not inherit any collections assigned to the binary document.
  4. The document will have keys of source-uri, content, and extracted-metadata.

The content in the document contains the extracted text. The extracted-metadata contains each metadata key and value produced by Apache Tika.

If you do not want Flux to write the file to a separate document but only want the extracted text document written, include the following option:

--extracted-text-drop-source

Writing extracted text XML documents

Extracted text documents can be written as XML instead of JSON by including the following option:

--extracted-text-document-type XML

When writing Tika metadata to the extracted text document, Flux will attempt to determine a well-known namespace based on the name of the metadata key. For example, a Tika metadata key that begins with pdf: will result in an element assigned to the http://ns.adobe.com/pdf/1.3/ namespace.

Assigning metadata to extracted text documents

Collections can be assigned to extracted text documents via the following option, which accepts a comma-delimited list of collection names:

--extracted-text-collections collection1,collection2

Permissions can be assigned to extracted text documents via the following options, which accepts a comma-delimited sequence of MarkLogic role names and capabilities:

--extracted-text-permissions role1,read,role2,update

Zip bomb protection

When importing ZIP files from untrusted sources, you can protect against zip bombs — archives crafted to expand to an enormous amount of data and potentially exhaust the memory available to Flux. Both options below are disabled by default.

To limit the maximum number of uncompressed bytes read from any single ZIP entry:

--zip-max-uncompressed-entry-bytes 268435456

A value of 268435456 (256 MB) is a reasonable starting point for most use cases. If a single entry exceeds this limit, an error is thrown. Set to 0 or any value less than 1 to disable.

To limit the maximum number of entries processed from a single ZIP file:

--zip-max-entry-count 100000

A value of 100000 is a reasonable starting point for most use cases. Set to 0 or any value less than 1 to disable.

This site uses Just the Docs, a documentation theme for Jekyll.