All Projects → mohd-akram → jify

mohd-akram / jify

Licence: MIT License
JSON indexed file database/querying library/tool

Programming Languages

typescript
32286 projects
javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to jify

Inquiry Deprecated
[DEPRECATED]: Prefer Room by Google, or SQLDelight by Square.
Stars: ✭ 264 (+1452.94%)
Mutual labels:  query, nosql
Lucenenet
Apache Lucene.NET
Stars: ✭ 1,704 (+9923.53%)
Mutual labels:  query, index
Octosql
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
Stars: ✭ 2,579 (+15070.59%)
Mutual labels:  query, nosql
WindowsMonitor
WMI namespaces and classes
Stars: ✭ 15 (-11.76%)
Mutual labels:  query
Cosmos.Identity
A Cosmos storage provider for ASP.NET Core Identity.
Stars: ✭ 26 (+52.94%)
Mutual labels:  nosql
mistql
A miniature lisp-like language for querying JSON-like structures. Tuned for clientside ML feature extraction.
Stars: ✭ 260 (+1429.41%)
Mutual labels:  query
kubesql
A tool based on presto using sql to query the resources of kubernetes, such as pods, nodes and so on.
Stars: ✭ 56 (+229.41%)
Mutual labels:  query
kvs
Highly available distributed strong eventual consistent and sequentially consistent storage with search
Stars: ✭ 15 (-11.76%)
Mutual labels:  nosql
SlideBar
SlideBar for Android 一个很好用的联系人字母快速索引
Stars: ✭ 47 (+176.47%)
Mutual labels:  index
ci4-album
🔥 CodeIgniter 4 example Album module uses Domain Driven Design Architecture with Tactical Pattern
Stars: ✭ 67 (+294.12%)
Mutual labels:  query
manon
🧪 Play with SpringBoot 2, JWT, Querydsl, GraphQL, Docker, ELK, PostgreSQL, MariaDB, Redis, MongoDB, Flyway, Maven, Gradle, TestNG, JUnit5, JaCoCo, GreenMail, CI, Quality Gates, Prometheus, Gatling, etc.
Stars: ✭ 26 (+52.94%)
Mutual labels:  nosql
nosqlilab
A lab for playing with NoSQL Injection
Stars: ✭ 90 (+429.41%)
Mutual labels:  nosql
cloud-enablement-aws
Enabling MarkLogic in the cloud (AWS)
Stars: ✭ 18 (+5.88%)
Mutual labels:  nosql
Mega-index-heroku
Mega nz heroku index, Serves mega.nz to http via heroku web. It Alters downloading speed and stability
Stars: ✭ 165 (+870.59%)
Mutual labels:  index
typeql
TypeQL: the query language of TypeDB - a strongly-typed database
Stars: ✭ 157 (+823.53%)
Mutual labels:  query
react-query-builder
Simple, configurable react query builder
Stars: ✭ 37 (+117.65%)
Mutual labels:  query
VBA-Arrays
😎 Array functions that are similar JavaScript functions. Example: Push, Pop, Shift, Unshift, Sort, length, toString.
Stars: ✭ 48 (+182.35%)
Mutual labels:  query
sparseIndexTracking
Design of Portfolio of Stocks to Track an Index
Stars: ✭ 27 (+58.82%)
Mutual labels:  index
hexo-generator-index2
Filtered index generator for Hexo
Stars: ✭ 40 (+135.29%)
Mutual labels:  index
sql-concat
A MySQL query builder
Stars: ✭ 14 (-17.65%)
Mutual labels:  query

jify

jify is an experimental library/tool for querying large (GBs) JSON files. It does this by first indexing the required fields. It can also be used as an append-only database.

When a JSON file is indexed (eg. data.json) an index file is created in the same directory with a .index.json extension (eg. data.index.json).

Install

npm install jify

Usage

const { Database, predicate: p } = require('jify');

async function main() {
  const db = new Database('books.json');

  // Create
  await db.create();

  // Insert - Single
  await db.insert({
    title: 'Robinson Crusoe',
    year: 1719,
    author: { name: 'Daniel Defoe' }
  });

  // Insert - Batch
  await db.insert([
    {
      title: 'Great Expectations',
      year: 1861,
      author: { name: 'Charles Dickens' }
    },
    {
      title: 'Oliver Twist',
      year: 1838,
      author: { name: 'Charles Dickens' }
    },
    {
      title: 'Pride and Prejudice',
      year: 1813,
      author: { name: 'Jane Austen' }
    },
    {
      title: 'Nineteen Eighty-Four',
      year: 1949,
      author: { name: 'George Orwell' }
    }
  ]);

  // Index - creates books.index.json file
  await db.index('title', 'year', 'author.name');

  // Query
  console.log('author.name = Charles Dickens, year > 1840');
  const query = { 'author.name': 'Charles Dickens', year: p`> ${1840}` };
  for await (const record of db.find(query))
    console.log(record);

  let records;

  // Range query
  console.log('1800 <= year < 1900');
  records = await db.find({ year: p`>= ${1800} < ${1900}` }).toArray();
  console.log(records);

  // Multiple queries
  console.log('year < 1800 or year > 1900');
  records = await db.find(
    { year: p`< ${1800}` }, { year: p`> ${1900}` }
  ).toArray();
  console.log(records);
}

main();

CLI

$ jify index --field title --field author.name --field year books.json
$ jify find --query "author.name=Charles Dickens,year>1840" books.json
$ jify find --query "year>=1800<1900" books.json
$ jify find --query "year<1800" --query "year>1900" books.json

Implementation

The index is implemented as a JSON array of skip list entries. The entries are encoded as strings and all numbers embedded in the string are encoded using Z85. This implementation was chosen for its simplicity and to allow for using a single JSON file as an index. Better performance might be achieved by using a different data structure, a binary format, or multiple index files.

Performance

jify is reasonably fast. It can index about 1M records (~700 MB) per minute and supports parallel indexing of fields. Inserting (with indexes) has similar performance. Query time is < 5ms for the first result + (0.1ms find + 0.1ms fetch) per subsequent result. All tests on a MBP 2016 base model.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].