Requirements
Python 3
Install
sudo apt-get install fuse libfuse2
pip3 install -r requirements.txt
If you will be using kerberos, install libkrb5-dev:
sudo apt-get install libkrb5-dev
Configuration
cp example.config.yaml config.yaml
$EDITOR config.yaml
Running
If you are using kerberos, run a kinit:
kinit -kt $USER $USER@REALM
In all cases you then will have to create a new empty directory that with be the mount point:
mkdir /mnt/dest_mount
And finaly you can run py-hdfs-fuse:
python3 hdfs_mount.py [--loglevel LEVEL] config.yaml
Have fun!
Note: if anything goes wrong and you have to kill py-hdfs-mount, you will probably have to run this command on the mounted folder to unlock it:
fusermount -u /mnt/dest_mount
umount -l /mnt/dest_mount
Tested with
- Vim (open file, edit randomly, save and close)
- cp/mv
Functionnalities
- Cached writes (HDFS is an immutable FS (so writes=delete+insert))
- Random writes (slow - because of the immutability of HDFS - but working!)
- Very fast ls (cached directory metadata)
- directory stored as a zip file in HDFS (to solve small files problem)
- directory stored as a avro file in HDFS (to solve small files problem)
- CRC32 checksum
- Load options from configuration file
Implemented FUSE methods
Basic
- access
- chmod
- chown
- getattr
- readdir
- readlink
- mknod
- rmdir
- mkdir
- statfs
- unlink
- symlink
- rename
- link
- utimens
File methods
- open
- create
- read
- write (caching is done in memory)
- truncate
- flush (writes the in memory written chunks to a temporary file in the local FS in the right order and calls fsync)
- fsync (send the temporary file to HDFS)
- release