RecordLoader

This class will be used to create new content objects, which implement com.marklogic.recordloader.ContentInterface. One alternative implementation is provided, as com.marklogic.recordloader.xcc.XccModuleContentFactory, which creates objects in the class com.marklogic.recordloader.xcc.XccModuleContent.

When XccModuleContentFactory is used, new documents must fit in memory, and will be posted to the XQuery main module designated by the CONTENT_MODULE_URI property. If the SKIP_EXISTING or ERROR_EXISTING features are desired, the module must implement each itself (see below).

When RecordLoader invokes this module, it will set external variables:

• $URI
• $XML-STRING
• $NAMESPACE  (using DEFAULT_NAMESPACE)
• $LANGUAGE  (using LANGUAGE)
• $ROLES-EXECUTE  (comma-separated values, using ROLES_EXECUTE)
• $ROLES-INSERT  (comma-separated values, using ROLES_INSERT)
• $ROLES-READ  (comma-separated values, using ROLES_READ)
• $ROLES-UPDATE  (comma-separated values, using ROLES_UPDATE)
• $COLLECTIONS  (comma-separated values, using any base collections plus OUTPUT_COLLECTIONS)
• $SKIP-EXISTING  (using SKIP_EXISTING)
• $ERROR-EXISTING  (using ERROR_EXISTING)
• $FORESTS  (comma-separated forest ids, via OUTPUT_FORESTS)

The following XQuery implements an example ContentModule, which implements a simple transform to lower-case all element names. Note that the module implements its own versions of the SKIP_EXISTING and ERROR_EXISTING checks.

xquery version "1.0-ml";

declare variable $URI as xs:string external;
declare variable $XML-STRING as xs:string external;
declare variable $NAMESPACE as xs:string external;
declare variable $LANGUAGE as xs:string external;
declare variable $ROLES-EXECUTE as xs:string external;
declare variable $ROLES-INSERT as xs:string external;
declare variable $ROLES-READ as xs:string external;
declare variable $ROLES-UPDATE as xs:string external;
declare variable $COLLECTIONS as xs:string external;
declare variable $SKIP-EXISTING as xs:boolean external;
declare variable $ERROR-EXISTING as xs:boolean external;
declare variable $FORESTS as xs:string external;

declare function local:do($list as node()*)
 as node()*
{
  for $n in $list return typeswitch($n)
  (: lower-case element localnames :)
  case element() return element {
    QName(namespace-uri($n), lower-case(local-name($n)))
  } {
    $n/@*, local:do($n/node())
  }
  case document-node() return document { local:do($n/node()) }
  default return $n
};

if ($SKIP-EXISTING and doc($URI)) then ()
else if ($ERROR-EXISTING and doc($URI)) then error((), 'DUPLICATE-URI', $URI)
else xdmp:document-insert(
  $URI,
  local:do(xdmp:unquote(
    $XML-STRING,
    $NAMESPACE,
    if ($LANGUAGE) then concat('default-language=', $LANGUAGE) else ()
  )),
  (
    for $r in tokenize($ROLES-EXECUTE, ',')[. ne '']
    return xdmp:permission($r, 'execute'),
    for $r in tokenize($ROLES-INSERT, ',')[. ne '']
    return xdmp:permission($r, 'insert'),
    for $r in tokenize($ROLES-READ, ',')[. ne '']
    return xdmp:permission($r, 'read'),
    for $r in tokenize($ROLES-UPDATE, ',')[. ne '']
    return xdmp:permission($r, 'update')
  ),
  tokenize($COLLECTIONS, ',')[. ne ''],
  0,
  for $id in tokenize($FORESTS, ',')[. ne '']
  return xs:unsignedLong($id)
)

Another included factory is com.marklogic.recordloader.http.HttpContentFactory. This is similar to the XCC version above, except that the implementation must use xdmp:get-request-field() rather than declaring external to populate the module variables. Note that some variables take xs:string* rather than xs:string: these are COLLECTIONS, FORESTS, ROLES-EXECUTE. ROLES-INSERT. ROLES-READ. and ROLES-UPDATE. The xs:boolean variables are xs:string instead. Also, FORESTS will be passed in as forest names, so the module must be prepared to translate these into forest ids.

declare variable $URI as xs:string := xdmp:get-request-field(
  'URI');
declare variable $XML-STRING as xs:string := xdmp:get-request-field(
  'XML-STRING');
declare variable $NAMESPACE as xs:string := xdmp:get-request-field(
  'NAMESPACE');
declare variable $LANGUAGE as xs:string := xdmp:get-request-field(
  'LANGUAGE');
declare variable $ROLES-EXECUTE as xs:string* := xdmp:get-request-field(
  'ROLES-EXECUTE);
declare variable $ROLES-INSERT as xs:string* := xdmp:get-request-field(
  'ROLES-INSERT);
declare variable $ROLES-READ as xs:string* := xdmp:get-request-field(
  'ROLES-READ');
declare variable $ROLES-UPDATE as xs:string* := xdmp:get-request-field(
  'ROLES-UPDATE);
declare variable $COLLECTIONS as xs:string* := xdmp:get-request-field(
  'COLLECTIONS');
declare variable $SKIP-EXISTING as xs:boolean := xs:boolean(
  xdmp:get-request-field('SKIP-EXISTING'));
declare variable $ERROR-EXISTING as xs:boolean := xs:boolean(
  xdmp:get-request-field('ERROR-EXISTING'));
declare variable $FORESTS as xs:string* := xdmp:get-request-field(
  'FORESTS');

if ($SKIP-EXISTING and doc($URI)) then ()
else if ($ERROR-EXISTING and doc($URI)) then error((), 'DUPLICATE-URI', $URI)
else xdmp:document-insert(
  $URI,
  xdmp:unquote(
    $XML-STRING,
    $NAMESPACE,
    if ($LANGUAGE) then concat('default-language=', $LANGUAGE) else ()
  ),
  (
    for $r in tokenize($ROLES-EXECUTE, ',')[. ne '']
    return xdmp:permission($r, 'execute'),
    for $r in tokenize($ROLES-INSERT, ',')[. ne '']
    return xdmp:permission($r, 'insert'),
    for $r in tokenize($ROLES-READ, ',')[. ne '']
    return xdmp:permission($r, 'read'),
    for $r in tokenize($ROLES-UPDATE, ',')[. ne '']
    return xdmp:permission($r, 'update')
  ),
  $COLLECTIONS,
  0,
  for $fn in $FORESTS return xdmp:forest($fn)
)

Not everyone wants to include their password in the properties file. A "little" bit of security was added by adding the ENCRYPTED_PASSWORD Flag. When this flag is enabled the recordloader process will look for the "ciphertext" and "keyfile" files that contain the encrypted password to be used. To create the two encrypted password files, execute:

java -cp recordloader.jar com.marklogic.ps.PasswordEncrypter PASSWORD

Not everyone wants to include their password in the properties file. A "little" bit of security was added by adding the ENCRYPTED_PASSWORD Flag. When this flag is enabled the recordloader process will look for the "ciphertext" and "keyfile" files that contain the encrypted password to be used. To create the two encrypted password files, execute:

java -cp recordloader.jar com.marklogic.ps.PasswordEncrypter PASSWORD

Note that this option requires the server to perform a separate check for each document uri. This can reduce performance.

Note that if using CONTENT_FACTORY_CLASSNAME=com.marklogic.recordloader.xcc.XccModuleContentFactory, this option requires the module to implement its mechanism (see above).

Note that namespace is ignored: only the local-name is used. The named node must have a simple text value: it may not be empty, and it must not contain any non-text children.

The special value ID_NAME=#AUTO will cause RecordLoader to automatically generate ids, in sequence, for each input record. Since RecordLoader automatically includes the base filename in each output URI, this is safe.

Note that when the input is standard input, the default value is #AUTO - not #FILENAME.

The special value ID_NAME=#FILENAME will cause RecordLoader to automatically load each input file into a single document per input file, using the file's basename to compose the new document uri. This is the default behavior.

Examples: ID_NAME=MedlineID, ID_NAME=@id

• REPORT: throws a MalformedInputException
• REPLACE: replaces invalid sequence with a '?' or similar.
• IGNORE: skips over the invalid sequence.

• com.marklogic.recordloader.svn.SvnInputHandler treats INPUT_PATH as a subversion repository url (EXPERIMENTAL).

If null, the default database language will be used.

Element name in which each document is found. These may not nest. If no RECORD_NAME is set, the first child element of the first root element will be used for the entire RecordLoader run.

If ID_NAME is set to an element or attribute name, or set to #AUTO (including when RecordLoader reads from standard input), then the special value RECORD_NAME=#DOCUMENT will cause RecordLoader to treat every document root element as a record. This mode is slower than ID_NAME=#FILENAME, but useful when the filenames are not appropriate as document URIs.

If true, existing document uris will be skipped. This allows RecordLoader to resume after being interrupted. This option may be combined with START_ID, in case the known value for START_ID already exists.

Note that one read I/O is required per skip, so SKIP_EXISTING is slower than using START_ID (below).

Note that if using CONTENT_FACTORY_CLASSNAME=com.marklogic.recordloader.xcc.XccModuleContentFactory, this option requires the module to implement its mechanism (see above).

Number of RecordLoader threads.

Note that when using standard input, this value is ignored.

Note that RecordLoader uses at most 1 thread per input file or zip entry.

• NONE: throw an exception (see also: FATAL_ERRORS).
• FULL: do everything reasonable to ingest the document.