Directory

Storing a collection in multiple files within a directory

Introduction

This page describes how to set up persistent file storage for a collection using Nabu, where each document is stored within a file under a specific directory on the file system.

Directory storage requires the Nabu storage engine

Usage

Statically created

To configure

collections:
  myCollection:
    storageEngine:
      directory:
        path: ./content/myCollection
        extension: .yaml
        format: yaml

Dynamically created

Tashmet.connect(store.proxy()).then(async tashmet => {
  const collection = await tashmet.db('myDb').createCollection('myCollection', {
    storageEngine: {
      directory: {
        path: 'content/myCollection',
        extension: '.yaml',
        format: 'yaml'
      }
    }
  });
});

The collection stored in 'content/myCollection' where each document will reside in a yaml file that derives its name from the _id of the document.

---

Reuse across database

By defining a custom I/O rule for the storage engine, we can reuse the same configuration across multiple collections.

const store = Nabu
  .configure({})
  .use(mingo())
  .io('yaml', (ns, options) => ({
    directory: {
      path: `${ns.db}/${ns.collection}`,
      extension: '.yaml',
      format: 'yaml'
    }
  }))
  .bootstrap();

Tashmet.connect(store.proxy()).then(async tashmet => {
  const collection = await tashmet.db('myDb').createCollection('myCollection', {
    storageEngine: 'yaml'
  });
});

Alternatively we can set the default I/O rule and skip the storageEngine directive entirely.

const store = Nabu
  .configure({
    defaultIO: 'yaml'
  })
  // ...

Tashmet.connect(store.proxy()).then(async tashmet => {
  const collection = await tashmet.db('myDb').createCollection('myCollection');
});

---

Parameters

Path

path: string

Path to the directory where files are stored

Extension

extension: string

File extension (including the dot)

{
  // ...
  extension: '.yaml'
}

Format

format: string | Document

File format. The current valid file formats include:

• format: 'json'

• format: 'yaml'

If we want to access YAML front matter, format also accepts the following configuration.

// YAML with front matter
{
  // ...
  format: {
    yaml: {
      frontMatter: true,
      contentKey: 'content' 
    }
  }
}

Merge Stat

mergeStat?: Document

Include file information for each document when it's loaded from the file system. Expressions in merge are computed against the underlying file information (lstat). Consider the following example for how to include the path of the file.

// Include file path
{
  // ...
  mergeStat: {
    path: '$path'
  }
}

The following fields are available for merging:

path, dev, mode, nlink, uid, gid, rdev, blksize, ino, size, blocks, atimeMs, mtimeMs, ctimeMs, birthtimeMs, atime, mtime, ctime, birthtime

Any fields defined in the merge will be pruned before writing them to disk

Construct

construct?: Document

Add additional fields to the document. This stage is performed after merge is completed. Expressions in construct are computed against the actual, loaded document.

Consider the following example where we read markdown files with YAML front matter. The markdown content will be stored under a field named markdown and we use construct to add an additional field html that contains the converted output.

{
  path: '/content',
  extension: '.md',
  format: {
    yaml: {
      frontMatter: true,
      contentKey: 'markdown'
    }
  },
  construct: {
    html: {
      $markdownToHtml: '$markdown'
    }
  }
}

Any fields defined in the construct stage will be pruned before writing them to disk

Default

default?: Document

Set default values for fields that are not present in the stored content.

{
  path: '/content/posts',
  extension: '.md',
  format: {
    yaml: {
      frontMatter: true,
      contentKey: 'markdown'
    }
  },
  default: {
    slug: '$_id'
  }
}