fast_io

fast_io is a new C++20 library for extremely fast input/output and aims to replace iostream and cstdio. It is header-only (module only in the future) for easy inclusion in your project. It requires a capable C++20 compiler supporting concepts.

QQ group

1076846339

Discord URL.

You can directly ask me questions in Discord.

https://discord.gg/wYfh8kk

What does "fast" mean in fast_io?

It does not necessary mean it will be faster than everything else. (Or it would be named as fastest_io. Of course those fmt trolls would claim that.) It means it is significantly faster than stdio.h and iostream for all cases. fast means faster than stdio.h and iostream.

Hello World

#include<fast_io.h>

int main()
{
	print("Hello World!\n");
}

Examples

https://github.com/expnkx/fast_io_examples

Deprecated examples are in https://bitbucket.org/ejsvifq_mabmip/fast_io_deprecated

However, they might not work current fast_io since fast_io keeps changing.

This i18n repo stores the i18n source files. https://bitbucket.org/ejsvifq_mabmip/fast_io_i18n

Online docs

https://ewindy.gitee.io/fast_io_rst/index.html

Compiler Support

• GCC 11. You can download the latest GCC compiler for windows here. https://bitbucket.org/ejsvifq_mabmip/mingw-gcc/src/master/ For Linux, you can watch this video to help you install the latest GCC easily. https://www.youtube.com/watch?v=qFToZjo2x24
• VS 19.28.29617
• Clang 12 with GNU libstdc++, including windows. No Clang with libc++ yet since libc++ has not supported concepts and ranges headers.
• Clang 12 with msvc stl.
• Todo: Intel C++ Compiler

Platform Support

• Microsoft Windows
• Linux
• BSD platforms (Including FreeBSD, NetBSD, OpenBSD)
• Apple Darwin (MacOS)
• Microsoft DOS
• Barebone (No any operating system.)
• Cygwin/MSYS2
• Android
• Todo: Webassembly

Supported libc FILE* hacks platforms

• glibc
• MSVCRT
• Universal CRT
• BSD libc/Apple Darwin
• MUSL libc
• DJGPP
• Newlib
• Bionic(Android)

Supported C++ standard library std::streambuf/std::filebuf hacks platforms

• GCC libstdc++
• LLVM libc++
• MSVC STL

Special Support

• LLVM Address Sanitizer

Design Goal

A general purpose I/O library to replace stdio.h and iostream

• As close to system call as possible.
• No default locale. It is optional.
• Correctly deal with EBCDIC exec-charset (where no libc correctly deals it).

Safe

• No easily misused stuff like std::endl
• No internal iomanip states (since it creates security issues)
• Providing RAII for C FILE*&POSIX fd&win32 HANDLE
• No dual error reporting mechanism. Exception as the ONLY error reporting mechanism.
• No std::error_code and std::system_error
• No useless bounds checking
• Addresss Sanitizer special code for protecting memory safety issues.

Easy to use

• Unicode/UTF-8 support
• Support POSIX iconv. You can use fast_io for your coding conversion.
• Compatible with C stdio and C++ iostream
• Binary serialization for trivially copyable types and C++ standard library containers
• All fast_io devices can be natively put in C++ containers. std::vector<fast_io::obuf_file> is valid
• Basic/Lua/Python/etc format (print, scan). No support to C and C++ since they are security hazards.
• Static I/O manipulator
• Provide APIs to expose the internal implementation of C FILE* and C++ stream.

Customizability

• Exception Safe & Exception neutral
• Native Handle Interface
• Extremely easy to support custom devices
• std::mutex mutex stream
• Compilation time open mode parse. Supports C style open mode and C++ style open mode.
• No traits_type and EOF
• Dynamic Type Support
• Multi Process
• Memory map
• debugging IO
• GUI debugging IO
• Freestanding mode
• Round-trip floating point algorithm
• Network
• Intrinsic SHA-1, Intrinsic HMAC-SHA1, Intrinsic SHA-256, Intrinsic HMAC-SHA256, SHA-512, HMAC-SHA512
• zlib compression/decompression support
• Non crypto hash algorithms support. Jenkins Hash
• Filesystem
• OpenSSL BIO, Qt QFile, MFC CFile support

Codecvt for File I/O support, including correctly dealing with execution charset of them

• UTF-8
• UTF-16LE
• UTF-16BE
• UTF-32LE
• UTF-32BE
• GB18030
• UTF-EBCDIC
• posix iconv is also supported

Post C++20 Plan

1. Module support
2. Coroutine support for async IO
3. Improve and refactor code once Zero-overhead deterministic exceptions are added to the standard

Possible Improvements

2. Interfaces for cloud computing algorithms like MapReduce

I'm aiming for this to be in the C++ standard library in the future :)

Getting Started

Please see examples in the examples folder.

compile option:

g++ -o example example.cc -Ofast -std=c++20 -s

Documentation

See Wiki Page: https://github.com/expnkx/fast_io/wiki

Benchmarks

1. I/O 10M integers

Goal: Print out ten million integers from 0 to 10M to file. Then reopen that file to scan back.

All benchmarks are in benchmarks/0000.10m_size_t/unit.

Notice: I modified libstdc++'s BUFSIZ 1048576 due to BUFSIZE is too small (512 bytes) for MinGW-W64, or it performs horribly.

Rust language is 10x slower than fast_io. + binary bloat and itoa library is still extremely slow and usable for me. It is at least 3x slower than fast_io.

Run the same test on MSVC 19.26.28805.

Run the same test on GCC 11. glibc + libstdc++

You can see fast_io can also boost the performance of existing facilities for 10x! Yes, it can even improve FILE* and fstream's performance for 10x depending on platforms since I use concepts to abstract them all. fmtlib actually slows down I/O performance.

2. Output 10M double in round-trip mode with Ryu algorithm

We only perform this test for MSVC since only msvc's charconv implements it. Yes. fast_io defeats msvc's charconv for over 20% for running the same algorithm.

All benchmarks are in benchmarks/0001.10m_double/charconv.

Run the same test on MSVC 19.26.28805.

3. Raw I/O Performance

All benchmarks are in benchmarks/0014.file_io/file_io.

Output 100000000x "Hello World\n"

Notice: I modified libstdc++'s std::filebuf's BUFSIZ to 1048576 due to BUFSIZE is too small (512 bytes) for MinGW-W64 or it performs horribly.

4. Binary Size Just use the benchmark in benchmarks/0014.file_io/file_io. That would introduce Ryu floating table for printing time, and I did not compile the code with -DFAST_IO_OPTIMIZE_SIZE. Or it would be nothing. Dude, you should avoid stream as plague tbh. It is not healthy.

5. Code Convert

Generate 100000000 🚄 emojis by using the program in benchmarks/0020.utf/fill_nc.cc

Benchmarks are in examples/0043.iconv Universal iconv. (UTF-8 to GB18030 as an example) iconv test: | Platform | Windows |MinGW-W64 GCC 11.0.0 | MSVCRT + libstdc++ | |--------------------------------|-------------------------|-----------------------|------------------------------------------------------| | |

UTF8->UTF16LE

Benchmarks are in examples/0022.utf

iconv test: | Platform | Windows |MinGW-W64 GCC 11.0.0 | MSVCRT + libstdc++ | |--------------------------------|-------------------------|-----------------------|------------------------------------------------------| | |

UTF8->UTF32LE

Benchmarks are in examples/0022.utf

iconv test: | Platform | Windows |MinGW-W64 GCC 11.0.0 | MSVCRT + libstdc++ | |--------------------------------|-------------------------|-----------------------|------------------------------------------------------| | |

Credit

This project is made possible by referencing from other open-source projects. (I won't take their code directly. They are reimplemented by myself to fit the purpose of this library or it might have integration issues.)