marklogic-community / corb2

What is CoRB?

CoRB is a Java tool designed for bulk content-reprocessing of documents stored in MarkLogic. CoRB stands for Content Reprocessing in Bulk and is a multi-threaded workhorse tool at your disposal. In a nutshell, CoRB works off a list of documents in a database and performs operations against those documents. CoRB operations can include generating a report across all documents, manipulating the individual documents, or a combination thereof.

User Guide

This document and the wiki provide a comprehensive overview of CoRB and the options available to customize the execution of a CoRB job, as well as the ModuleExecutor Tool, which can be used to execute a single (XQuery or JavaScript) module in MarkLogic.

For additional information, please refer to the CoRB Wiki.

Downloads

Download the latest release directly from https://github.com/marklogic-community/corb2/releases or resolve dependencies through Maven Central or JCenter.

• CoRB v2.4.0 (or later) requires Java 8 (or later) to run.
• CoRB v2.3.2 is the last release compatable with Java 7 and 6.
• CoRB v2.2.0 (or later) requires marklogic-xcc 8.0.* (or later) to run.

Note: marklogic-xcc 8 is backwards compatible to MarkLogic 5 and runs on Java 1.6 or later.

Getting Help

To get help with CoRB

• Post a question to Stack Overflow with the markogic and corb tags.
• Submit issues or feature requests at https://github.com/marklogic-community/corb2/issues

Running CoRB

The entry point is the main method in the com.marklogic.developer.corb.Manager class. CoRB requires the MarkLogic XCC JAR in the classpath, preferably the version that corresponds to the MarkLogic server version, which can be downloaded from https://developer.marklogic.com/products/xcc. Use Java 1.8 or later.

CoRB needs options specified through one or more of the following mechanisms:

1. command-line parameters
2. Java system properties ex: -DXCC-CONNECTION-URI=xcc://user:password@localhost:8202
3. As properties file in the class path specified using -DOPTIONS-FILE=myjob.properties. Relative and full file system paths are also supported.

If specified in more than one place, a command line parameter takes precedence over a Java system property, which take precedence over a property from the OPTIONS-FILE properties file.

Note: Any or all of the properties can be specified as Java system properties or key value pairs in properties file.

Note: CoRB exit codes 0 - successful, 0 - nothing to process (ref: EXIT-CODE-NO-URIS), 1 - initialization or connection error and 2 - execution error

Note: CoRB now supports Logging Job Metrics back to the MarkLogic database log and/or as document in the database.

Options

Additional options

URIS_BATCH_REF

If a module, including those specified by PRE-BATCH-MODULE, PROCESS-MODULE or POST-BATCH-MODULE have an external or global variable named URIS_BATCH_REF, the variable will be set to the first non-numeric item in the sequence returned by URIS-MODULE. This means that, when used, the URIS-MODULE must return a sequence with the special string value first, then the URI count, then the sequence of URIs to process.

As an example, a batch ref can be a link/id of a document that manages the status of the batch job, where pre-batch module updates the status to start and post-batch module can set it to complete. This example can be used to manage status and errors in automated batch jobs.

ExportBatchToFileTask, PreBatchUpdateFileTask and PostBatchUpdateFileTask use URIS_BATCH_REF as the file name if EXPORT-FILE-NAME is not specified. This is useful for automated jobs where name of the output file name can be determined only by the URIS-MODULE.

URIS_TOTAL_COUNT

Total count of uris is set as an external variable to PRE-BATCH-MODULE and POST-BATCH-MODULE (since 2.4.5)

Custom Inputs to XQuery or JavaScript Modules

Any property specified with prefix (with '.') INIT-MODULE, URIS-MODULE, PRE-BATCH-MODULE, PROCESS-MODULE, POST-BATCH-MODULE will be set as an external variable in the corresponding XQuery module (if that variable is defined as an external string variable in XQuery module). For JavaScript modules the variables need be defined as global variables.

Custom Input Examples:

• URIS-MODULE.maxLimit=1000 Expects an external string variable maxLimit in URIS-MODULE XQuery or global variable for JavaScript.
• PROCESS-MODULE.startDate=2015-01-01 Expects an external string variable startDate in PROCESS-MODULE XQuery or global variable for JavaScript.

Alternatively, URIS-MODULE can pass custom inputs to PRE-BATCH-MODULE, PROCESS-MODULE, POST-BATCH-MODULE by returning one or more of the property values in above format before the count the of URIs. If the URIS-MODULE needs URIS_BATCH_REF (above) as well, it needs to be just before the URIs count.

Custom Input From URIS-MODULE Example

let $uris := cts:uris()
return ("PROCESS-MODULE.foo=bar", "POST-BATCH-MODULE.alpha=10", fn:count($uris), $uris)

Adhoc Modules

Appending |ADHOC to the name or path of a XQuery module (with .xqy extension) or JavaScript (with .sjs or .js extension) module will cause the module to be read from the file system and executed in MarkLogic without being uploaded to Modules database. This simplifies running CoRB jobs by not requiring deployment of any code to MarkLogic, and makes the set of CoRB files and configuration more self contained.

INIT-MODULE, URIS-MODULE, PROCESS-MODULE, PRE-BATCH-MODULE and POST-BATCH-MODULE can be specified adhoc by adding the suffix |ADHOC for XQuery or JavaScript (with .sjs or .js extension) at the end. Adhoc XQuery or JavaScript remains local to the CoRB and is not deployed to MarkLogic. The XQuery or JavaScript module should be in its named file and that file should be available on the file system, including being on the java classpath for CoRB.

Adhoc Examples:

• PRE-BATCH-MODULE=adhoc-pre-batch.xqy|ADHOC adhoc-pre-batch.xqy must be on the classpath or in the current directory.
• PROCESS-MODULE=/path/to/file/adhoc-transform-module.xqy|ADHOC XQuery module file with full path in the file system.
• URIS-MODULE=adhoc-uris.sjs|ADHOC Adhoc JavaScript module in the classpath or current directory.

Inline Adhoc Modules

It is also possible to set a module option with inline code blocks, rather than a file path. This can be done by prepending either INLINE-XQUERY| or INLINE-JAVASCRIPT| to the option value, followed by the XQuery or JavaScript code to execute. Inline code blocks are executed as "adhoc" modules and are not uploaded to the Modules database. The |ADHOC suffix is optional for inline code blocks.

Inline Adhoc Example

URIS-MODULE=INLINE-XQUERY|xquery version '1.0-ml'; let $uris := cts:uris('', 'document', cts:collection-query('foo')) return (count($uris), $uris)

JavaScript Modules

JavaScript modules are supported and can be used in place of an XQuery module. However, if returning multiple values (ex: URIS-MODULE), values must be returned as a Sequence. MarkLogic JavaScript API has helper functions to convert Arrays into Sequence (Sequence.from()) and inserting values into another Sequence (fn.insertBefore()).

JavaScript module must have an .sjs file extension when deployed to Modules database. However, adhoc JavaScript modules support both .sjs and .js file extensions.

For example, a simple URIS-MODULE may look like this:

var uris = cts.uris();
fn.insertBefore(uris, 0, fn.count(uris));

To return URIS_BATCH_REF, we can do the following:

fn.insertBefore(fn.insertBefore(uris, 0, fn.count(uris)), 0, "batch-ref")

Note: Do not use single quotes within (adhoc) JavaScript modules. If you must use a single quote, escape it with a quote (ex: ''text'')

Encryption

It is often required to protect the database connection string or password from unauthorized access. So, CoRB optionally supports encryption of the entire XCC URL or any parts of the XCC URL (if individually specified), such as XCC-PASSWORD.

com.marklogic.developer.corb.PrivateKeyDecrypter

PrivateKeyDecrypter automatically detects if the text is encrypted. Unencrypted text or clear text is returned as-is. Although not required, encrypted text can be optionally enclosed with "ENC" ex: ENC(xxxxxx) to clearly indicate that it is encrypted.

Generate keys and encrypt XCC URL or password using one of the options below.

Java Crypt

• Use the PrivateKeyDecrypter class inside the CoRB JAR with the gen-keys option to generate a key.
  java -cp /path/to/lib/* com.marklogic.developer.corb.PrivateKeyDecrypter gen-keys /path/to/private.key /path/to/public.key RSA 1024

Note: if not specified, default algorithm: RSA, default key-length: 1024

• Use the PrivateKeyDecrypter class inside the CoRB JAR with the encrypt option to encrypt the clear text such as an xcc URL or password.
  java -cp /path/to/lib/* com.marklogic.developer.corb.PrivateKeyDecrypter encrypt /path/to/public.key clearText RSA

Note: if not specified, default algorithm: RSA

RSA keys

• openssl genrsa -out private.pem 1024 Generate a private key in PEM format
• openssl pkcs8 -topk8 -nocrypt -in private.pem -out private.pkcs8.key Create a PRIVATE-KEY-FILE in PKCS8 standard for java
• openssl rsa -in private.pem -pubout > public.key Extract public key
• echo "uri or password" | openssl rsautl -encrypt -pubin -inkey public.key | base64 Encrypt URI or password. Optionally, the encrypted text can be enclosed with "ENC" ex: ENC(xxxxxx)

ssh-keygen

• ssh-keygen ex:key as id_rsa after selecting a passphrase
• openssl pkcs8 -topk8 -nocrypt -in id_rsa -out id_rsa.pkcs8.key (asks for passphrase)
• openssl rsa -in id_rsa -pubout > public.key (asks for passphrase)
• echo "password or uri" | openssl rsautl -encrypt -pubin -inkey public.key | base64

com.marklogic.developer.corb.JasyptDecrypter

JasyptDecrypter automatically detects if the text is encrypted. Unencrypted text or clear text is returned as-is. Though, not required, encrypted text can be optionally enclosed with "ENC" ex: ENC(xxxxxx) to clearly indicate that it is encrypted.

Encrypt the URI or password as below. It is assumed that the jasypt distribution is available on your machine.

jasypt-1.9.2/bin/encrypt.sh input="uri or password" password="passphrase" algorithm="algorithm" (ex: PBEWithMD5AndTripleDES or PBEWithMD5AndDES)

jasypt.properties file

jasypt.algorithm=PBEWithMD5AndTripleDES #(If not specified, default is PBEWithMD5AndTripleDES)
jasypt.password=passphrase

com.marklogic.developer.corb.HostKeyDecrypter

HostKeyDecrypter uses internal server identifiers to generate a private key unique to the host server. It then uses that private key as input to AES-258 encryption algorithm. Due to the use of AES-258, it requires JCE Unlimited Strength Jurisdiction Policy Files.

Note: certain server identifiers used may change in cases of driver installation or if underlying hardware changes. In such cases, passwords will need to be regenerated. Encrypted passwords will be always be unique to the server they are generated on.

Encrypt the password as follows:
java -cp /path/to/lib/* com.marklogic.developer.corb.HostKeyDecrypter encrypt clearText

To test if server is properly configured to use the HostKeyDecrypter:
java -cp /path/to/lib/* com.marklogic.developer.corb.HostKeyDecrypter test

SSL Support

CoRB provides support for SSL over XCC. As a prerequisite to enabling CoRB SSL support, the XDBC server must be configured to use SSL. It is necessary to specify XCC-CONNECTION-URI property with a protocol of 'xccs'. To configure a particular type of SSL configuration use the following property:

com.marklogic.developer.corb.TrustAnyoneSSLConfig

TrustAnyoneSSLConfig is the default implementation of the SSLContext. It will accept any certificate presented by the MarkLogic server.

com.marklogic.developer.corb.TwoWaySSLConfig

TwoWaySSLConfig is more complete and configurable implementation of the SSLContext. It supports SSL with mutual authentication. It is configurable via the following properties:

Load Balancing and Failover with Multiple Hosts

CoRB 2.4+ supports load balancing and failover using com.marklogic.developer.corb.ContentSourcePool. This is automatically enabled when multiple comma separated values (supports encryption) are specified for for XCC-CONNECTION-URI or XCC-HOSTNAME.

XCC-CONNECTION-URI=xcc://hostname1:8000/dbname,xcc://hostname2:8000/dbname,..

OR

XCC-HOST-NAME=hostname1,hostname2,..

The default implementation for com.marklogic.developer.corb.ContentSourcePool is com.marklogic.developer.corb.DefaultContentSourcePool. It uses below options for CONNECTION-POLICY for allocating connections to callers.

• ROUND-ROBIN - (Default) Connections are allocated using round-robin algorithm.
• RANDOM - Connections are randomly allocated.
• LOAD - Host with least number of active connections is allocated to caller.

Query and Connection Retries

CoRB automatically retries the requests a given URI when it encounters com.marklogic.xcc.exceptions.ServerConnectionException from MarkLogic. If necessary, the number of retry attempts can be configured using XCC-CONNECTION-RETRY-LIMIT. If multiple hosts are specified, we can optionally configure retries per each host using XCC-CONNECTION-HOST-RETRY-LIMIT. CoRB waits at least XCC-CONNECTION-RETRY-INTERVAL seconds before a connection is retried on a failed host.

CoRB also supports retries of requests failed due to query errors. This feature is only intended for sporadic query errors which are not specific to a particular URI. A good example may include occasional time out exceptions from MarkLogic when the ML is too busy and request time limit is low. We can configure which queries can be retried using QUERY-RETRY-ERROR-CODES or QUERY-RETRY-ERROR-MESSAGE (when error codes are not available). If necessary, the number of query retry attempts can be configured using QUERY-RETRY-LIMIT. CoRB waits at least QUERY-RETRY-INTERVAL seconds before retrying a query.

QUERY-RETRY-ERROR-CODES=XDMP-EXTIME,SVC-EXTIME
QUERY-RETRY-ERROR-MESSAGE=ErrorMsg1,ErrorMsg2

Usage Examples

Refer to the wiki for examples of how to execute a CoRB job and various ways of configuring the job options.

ModuleExecutor Tool

Sometimes, a two or more staged CoRB job with both a selector and transform isn't necessary to get the job done. Sometimes, only a single query needs to be executed and the output captured to file. Maybe even to execute only a single query with no output captured. In these cases, the ModuleExecutor Tool can be used to quickly and efficiently execute your XQuery or JavaScript files.