All Projects → xenon-middleware → xenon

xenon-middleware / xenon

Licence: Apache-2.0 License
A middleware abstraction library that provides a simple programming interface to various compute and storage resources.

Programming Languages

52797 projects - #9 most used programming language

Projects that are alternatives of or similar to xenon

An event-centric publisher/subscribe model for objects inspired by the Qt framework
Stars: ✭ 37 (+32.14%)
Mutual labels:  middleware
Redux tools to speed up development.
Stars: ✭ 16 (-42.86%)
Mutual labels:  middleware
PHP client library for Akamai {OPEN} EdgeGrid Authentication scheme (based on Guzzle)
Stars: ✭ 38 (+35.71%)
Mutual labels:  middleware
PSR-7 Zend ACL implementation - Permission Library [ slim, psr7, acl, permissions, zend ]
Stars: ✭ 33 (+17.86%)
Mutual labels:  middleware
A set of tools to facilitate react-redux development and decouple logic from compontents
Stars: ✭ 37 (+32.14%)
Mutual labels:  middleware
🚀 Node.js Web Framework
Stars: ✭ 32 (+14.29%)
Mutual labels:  middleware
Node.js middleware for handling resumable uploads
Stars: ✭ 17 (-39.29%)
Mutual labels:  middleware
Connect middleware for BrowserSync
Stars: ✭ 16 (-42.86%)
Mutual labels:  middleware
DevOps code to deploy eScience services
Stars: ✭ 19 (-32.14%)
Mutual labels:  middleware
A Redux middleware for global loader
Stars: ✭ 13 (-53.57%)
Mutual labels:  middleware
PSR-15 middleware to implement Basic and Digest Http authentication
Stars: ✭ 29 (+3.57%)
Mutual labels:  middleware
Error trace for ThinkJS 3.x
Stars: ✭ 12 (-57.14%)
Mutual labels:  middleware
Simple PSR-15 middleware dispatcher
Stars: ✭ 31 (+10.71%)
Mutual labels:  middleware
PSR-15 middleware to generate access logs
Stars: ✭ 21 (-25%)
Mutual labels:  middleware
.NET Cross platform and highly composable middleware for building web request handlers in F#
Stars: ✭ 178 (+535.71%)
Mutual labels:  middleware
Angular in ASP.NET Core with JWT solution by systemjs
Stars: ✭ 48 (+71.43%)
Mutual labels:  middleware
ReSwift+redeux dev tools
Stars: ✭ 13 (-53.57%)
Mutual labels:  middleware
Middleware for JWT in HORSE
Stars: ✭ 39 (+39.29%)
Mutual labels:  middleware
NiFi Bundle for FIX Protocol
Stars: ✭ 14 (-50%)
Mutual labels:  middleware
Throttling Middleware for Martini
Stars: ✭ 63 (+125%)
Mutual labels:  middleware


Build Status Build status Windows codecov SonarCloud GitHub license Download DOI Research Software Directory CII Best Practices

Copyright 2013-2021 The Netherlands eScience Center

What problem does Xenon solve?

Many applications use remote storage and compute resources. To do so, they need to include code to interact with the scheduling systems and file transfer protocols used on those remote machines.

Unfortunately, many different scheduler systems and file transfer protocols exist, often with completely different programming interfaces. This makes it hard for applications to switch to a different system or support multiple remote systems simultaneously.

Xenon solves this problem by providing a single programming interface to many different types of remote resources, allowing applications to switch without changing a single line of code.

Xenon abstraction

How does Xenon work?

Xenon is an abstraction layer that sits between your application and the (usually remote) resource it uses. Xenon is written in Java, but is also accessible from other languages (e.g. Python) through its gRPC interface and via the command line.

Xenon API

Overview of the Xenon ecosystem of tools

component repository
Xenon library
Xenon cloud adaptors like s3
Xenon grid adaptors like gridftp
Xenon hadoop adaptors like hdfs
gRPC extension for Xenon
command line interface to Xenon
Python API for Xenon
Docker images

Supported middleware

Xenon currently supports the following file access mechanisms:

  • file (local file manipulation)
  • ftp
  • sftp
  • webdav
  • s3
  • hdfs

Xenon currently supports the following job submission mechanisms:

  • local
  • ssh
  • at
  • gridengine
  • slurm
  • torque

See the roadmap for the planned extensions.

Adding Xenon as a dependency to your project

Follow the instructions from to include Xenon as a dependency for Gradle, Maven, SBT, or Leiningen projects, e.g. Gradle:

	allprojects {
		repositories {


	dependencies {
	        compile group: 'nl.esciencecenter.xenon', name: 'xenon', version: '3.1.0'

This will give the core adaptors to get cloud, grid and hadoop adaptors add the following dependencies:

    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-cloud', version: '3.0.2'
    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-grid', version: '3.0.0'
    compile group: 'nl.esciencecenter.xenon.adaptors', name: 'xenon-adaptors-hadoop', version: '3.0.0'

Simple examples

Here are some examples of basic operations you can perform with Xenon:

Copying a file from a local filesystem to a remote filesystem

import nl.esciencecenter.xenon.XenonException;
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class CopyFileLocalToSftpAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // Use the file system adaptors to create file system representations; the remote file system
        // requires credentials, so we need to create those too.
        // Assume the remote system is actually just a Docker container (e.g.
        //, accessible via
        // port 10022 on localhost
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem localFileSystem = FileSystem.create("file");
        FileSystem remoteFileSystem = FileSystem.create("sftp", location, credential);

        // create Paths for the source and destination files, using absolute paths
        Path sourceFile = new Path("/etc/passwd");
        Path destFile = new Path("/tmp/password");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;
        boolean recursive = false;

        // perform the copy and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = localFileSystem.copy(sourceFile, remoteFileSystem, destFile, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = localFileSystem.waitUntilDone(copyId, timeoutMilliSecs);

        // throw any exceptions
        XenonException copyException = copyStatus.getException();
        if (copyException != null) {
          throw copyException;

Submitting a job

The following code performs a wordcount of a file residing on a remote machine:

import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.JobStatus;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmSubmitWordCountJob {

    public static void main(String[] args) throws Exception {

        // Assume the remote system is actually just a Docker container (e.g.
        //, accessible to user 'xenon' via
        // port 10022 on localhost, using password 'javagat'
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler representation
        Scheduler scheduler = Scheduler.create("slurm", location, credential);

        JobDescription description = new JobDescription();
        description.setArguments("-l", "/etc/passwd");

        // submit the job
        String jobId = scheduler.submitBatchJob(description);

        long WAIT_INDEFINITELY = 0;
        JobStatus jobStatus = scheduler.waitUntilDone(jobId, WAIT_INDEFINITELY);

        // print any exceptions
        Exception jobException = jobStatus.getException();
        if (jobException != null)  {
          throw jobException;


The output of the job will be written to /tmp/wc.stdout.txt file in the xenonmiddleware/slurm Docker container.

For more examples, see the tutorial at Read The Docs.


Xenon's JavaDoc is available online at

Documentation for maintainers

For developers of Xenon itself


The Xenon library is copyrighted by the Netherlands eScience Center and released under the Apache License, Version 2.0. A copy of the license may be obtained from

Xenon uses several third-party libraries that have their own (permissive, open source) licenses. See the file legal/ for an overview.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected]