makuto / Cakelisp

#+TITLE:Cakelisp

[[file:images/CakeLisp_gradient_128.png]]

This is a programming language where I [[https://en.wikipedia.org/wiki/You_can%27t_have_your_cake_and_eat_it][can have my cake and eat it too]]. I wanted to do this after my [[https://macoy.me/code/macoy/LanguageTests][LanguageTests]] experiment revealed just how wacky Common Lisp implementations are in regards to performance. I was inspired by Naughty Dog's use of GOAL, GOOL, and Racket/Scheme (on their modern titles). I've also taken several ideas from Jonathan Blow's talks on Jai.

The goal is a metaprogrammable, hot-reloadable, non-garbage-collected language ideal for high performance, iteratively-developed programs (especially games).

It is a transpiler which generates C/C++ from an S-expression syntax. Cakelisp takes some inspiration from Lisp, but is not compatible and does not aspire to become "a Lisp".

You can see the [[https://macoy.me/blog/programming/CakelispIntro][introduction to Cakelisp]] and check out the [[https://news.ycombinator.com/item?id=25491568][Hacker News announcement thread]].

• Features

• The metaprogramming capabilities of Lisp: True full-power macro support. Macros can use compile-time code execution to conditionally change what is output based on the context of the invocation
• The performance of C: No heavyweight runtime, boxing/unboxing overhead, etc.
• "Real" types: Types are identical to C types, e.g. int is 32 bits with no sign bit or anything like other Lisp implementations do
• No garbage collection: I primarily work on games, which make garbage collection pauses unacceptable. I also think garbage collectors add /more/ complexity than manual management
• Hot reloading: It should be possible to make modifications to functions /and structures/ at runtime to quickly iterate
• Truly seamless C and C++ interoperability: No bindings, no wrappers: C/C++ types and functions are as easy to declare and call as they are in C/C++. In order to support this, I've decided to ignore type deduction when possible and instead rely on the C compiler/linker to relay typing errors. Cakelisp will blindly generate what look like C/C++ function calls without knowing if that function actually exists, because the C/C++ compiler will tell us what the answer is
• Compile-time code modification: After all macros are expanded, the programmer can specify compile-time functions which can do arbitrary modification of the expanded code. This makes it possible to validate functions, automatically insert profiling instrumentation (similar to this [[https://www.youtube.com/watch?v=59lKAlb6cRg][Jai demonstration]]), and other tasks which would be cumbersome or impossible to do with macros alone
• Output human-readable C/C++ source and header files: This makes it possible to use Cakelisp in a subset of your project. It also means Cakelisp will work on any platform C/C++ works on. Generated code closely resembles the source Cakelisp code whenever possible
• Build system: Simple projects will automatically be built and linked into an executable. Complex projects can use compile-time code execution to override stages of the build process. The code essentially knows how to build itself!

For more advantages, see [[file:doc/NeatUses.org][doc/NeatUses.org]].

Some of these features come naturally from using C as the backend. Eventually it would be cool to not have to generate C (e.g. generate LLVM bytecode instead), but that can a project for another time.

• Terms Cakelisp is copyright (C) 2020 Macoy Madson [email protected].

Licensed under GPL-3.0-or-later, with added [[https://www.gnu.org/licenses/gpl-faq.en.html#LinkingOverControlledInterface][Linking Over Controlled Interface]] exception. runtime/ is licensed under MIT License.

Please see [[file:doc/Legal.org][doc/Legal.org]] for a detailed explanation.

Contact [email protected] if you would like to negotiate an exception for your use-case.

• Building Cakelisp itself ** Linux Run Build.sh in cakelisp/: #+BEGIN_SRC sh cd cakelisp ./Build.sh #+END_SRC

This script first builds cakelisp explicitly, then uses the cakelisp_bootstrap executable to build Cakelisp using Cakelisp. Subsequent executions of Build.sh will also build using Cakelisp, which means all the caching features will be in effect (making builds much faster).

You can then run ./bin/cakelisp. Have fun! A simple test: #+BEGIN_SRC sh ./bin/cakelisp --execute test/Hello.cake #+END_SRC

It shouldn't be hard to build Cakelisp using your favorite build system. Simply build all the .cpp files in src (with UNIX defined), then link them into an executable which also links -ldl. Leave out Main.cpp and you can embed Cakelisp in a static or dynamic library! ** Windows Windows does not come with a compiler installed by default. In order to provide a solid "native" experience, Cakelisp supports MSVC. It should be possible to compile on Windows using MinGW as well (in which case, you should refer to the Linux instructions, or open an Issue requesting Windows-native MinGW bootstrap building).

Once you do have a compiler, I aspire to make Cakelisp the /easiest/ way to make C/C++-based projects on Windows. You shouldn't need to touch Visual Studio project files ever again!

The following instructions will assume you are using MSVC.

1. Install Visual Studio. You can download the Community edition for free [[https://visualstudio.microsoft.com/downloads/][here]].

   Ensure that you pick the C++ track, and make sure the C++ Developer Tools box is checked (if you clicked C++, you shouldn't need to make any other changes).

   Note that you need to upgrade depending on what kinds of projects you use, e.g. if you are in a team making proprietary software, you must pay for Professional or Enterprise

2. Download Cakelisp. If you have git or any git client installed, clone this repository using the URL provided on this page. If you don't have Git, download the .zip file from the repository home page

3. In cakelisp/, double-click Build.bat.

   This should automatically find your Visual Studio and set the proper environment variables. If it doesn't, see the [[https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160#developer_command_file_locations][MSVC documentation]] on running from the command line. You may have to update the path to vcvars64.bat set in Build.bat to whatever Visual Studio version you installed. You should be able to find the script via searching for vcvars in your C:\Program Files (x86) folder.

4. Check that the build succeeded. Cakelisp is first built manually, then bootstrap builds itself using Bootstrap_MSVC.cake. You may need to modify Build.bat or Bootstrap_MSVC.cake to change compiler settings. Subsequent executions of Build.bat will use cached versions to only build changed files

If you have cakelisp.exe in bin/, you are ready to use Cakelisp! Due to the use of environment variables to select the compiler, you can only build programs if you've set the variables. To do so, refer to Build.bat, or create a .bat file based on this template: #+BEGIN_SRC bat echo off rem Set environment variables for compiler/linker selection rem Include help message in case this file isn't present if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" ( call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" ) else ( echo This script builds using MSVC. echo You must download and install MSVC before it will work. Download it here: echo https://visualstudio.microsoft.com/downloads/ echo Select workloads for C++ projects. Ensure you install the C++ developer tools. echo If you're still seeing this, you may need to edit Build.bat to your vcvars path echo Please see the following link: echo https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160 goto fail )

rem EDIT ME! If you keep this build script in the same directory as your project, update rem "bin\cakelisp.exe" to wherever you have a built version of Cakelisp rem Add --execute before the .cake files to run your project after building "bin\cakelisp.exe" YourProgram.cake

rem Make the result clear @if %ERRORLEVEL% == 0 ( echo Success! goto success ) else ( echo Error while building goto fail )

:fail goto end

:success goto end

:end rem Give the user a chance to read the input (not required) pause #+END_SRC *** Building from Visual Studio It is also possible to create a Visual Studio project to build Cakelisp. This is especially useful if you are debugging Cakelisp itself.

You can refer to an existing project in cakelisp/VisualStudio. The steps to create a new project are as follows:

• Open Visual Studio

• File -> New -> New Project

• Visual C++ -> Empty Project

• Set the name, location, and solution as desired, then click OK

• Project -> Add Existing Item

• Browse to cakelisp/src, then select all the files (via Ctrl-A, etc.) and click Add. This makes Cakelisp itself built by Visual Studio (i.e., not bootstrap built)

• Build -> Configuration Manager

• Select x64 for Active solution platform. Cakelisp works in Win32, but you should probably be using x64 anyways, because it's the dominant architecture, and the hard-coded compile/link commands assume x64

• Project -> Properties...

• In the Debugging section, set Working Directory to wherever your project's working directory is, e.g. where your .cake files are

• Set Command Arguments to the .cake files you want to build. You can leave this empty to see the Cakelisp help page

• Set the Environment to include the necessary environment variables. This is required so that Cakelisp knows how to find MSVC when launching it as a sub-process. The easiest way to find these is to open a Command Prompt, then run a vcvars script. For example, I run the following: #+BEGIN_SRC bat "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" #+END_SRC Your path may be different based on your Visual Studio version and installation. See [[https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160][documentation]] on how to find this file.

  Next, run set in that same Command Prompt. Select all of the text output by that command and hit Enter to copy it. Finally, return to the Environment setting in Visual Studio, click the down arrow on the field, then <Edit...>. Paste into the top text field, then hit OK.

• Expand Configuration Properties -> C/C++ -> Preprocessor

• Double click or <Edit...> the Preprocessor Definitions field and add the following to the beginning: #+BEGIN_SRC sh CAKELISP_EXPORTING;_CRT_SECURE_NO_WARNINGS;WINDOWS; #+END_SRC CAKELISP_EXPORTING indicates Cakelisp should export its symbols to DLLs. The CRT definition is going to be removed eventually; it makes MSVC more lenient with some errors Cakelisp has. The WINDOWS definition ensures you build with Cakelisp's Windows-specific code enabled

• Hit F5 or go to Debug -> Start Debugging. Visual Studio will build Cakelisp, and if it succeeds, launch Cakelisp. If you have no Command Arguments set, you should see the Cakelisp help output in a command window.

This project is for building Cakelisp itself; you don't need to make any new projects for your projects written in Cakelisp (in .cake files). Change the Debugging -> Command Arguments setting to build different Cakelisp files, or change the Working Directory to build a different Cakelisp project (e.g. one in a separate repository). ** Dependencies Currently, Cakelisp has no dependencies other than:

• C++ STL and runtime: These are normally included in your toolset
• Child-process creation: On Linux, unistd.h. On Windows, windows.h
• Dynamic loading: On Linux, libdl. On Windows, windows.h
• File modification times: On Linux, sys/stat.h
• C++ compiler toolchain: Cakelisp needs a C++ compiler and linker to support compile-time code execution, which is used for macros and generators

I'm going to try to keep it very lightweight. It should make it straightforward to port Cakelisp to other platforms.

Note that your /project/ does not have to include or link any of these unless you use hot-reloading, which requires dynamic loading. This means projects using Cakelisp are just as portable as any C/C++ project - there's no runtime to port (except hot-reloading, which is optional).

• Building a project using Cakelisp Cakelisp will automatically figure out how to build simple projects into executables.

For more complex projects, many hooks and variables are provided for overriding the build process. Your code is defined in Cakelisp, and so are all build commands. This gives the code the ability to know how to build itself.

For example, you could have a .cake module which includes a 3rd party graphics library. By importing that module, the module's compile-time hooks are added to the build process, which can do things like add the 3rd party graphics library's lib files to the link stage.

The build hooks are all regular Cakelisp code, which means you could do something as advanced as cloning a repository from the internet, launching a subprocess to cmake and make that project, then let Cakelisp finish the build by linking the output libraries.

One huge advantage to defining your build process in a "real" programming language (as opposed to a domain-specific language interpreted by a build system) is that you can attach a debugger and single step through the build process when things go wrong.

• Learning Cakelisp Check out [[file:doc/Cakelisp.org][doc/Cakelisp.org]] for a detailed explanation of the Cakelisp language and build system.

Explore test/ and runtime/ for examples of Cakelisp code. [[https://macoy.me/code/macoy/gamelib][GameLib]] is a collection of modules built for making games in Cakelisp. Check both src/ and test/ in GameLib for more extensive code examples.

• Tooling support ** Emacs Open .cake files in lisp-mode: #+BEGIN_SRC lisp (add-to-list 'auto-mode-alist '("\.cake?\'" . lisp-mode)) #+END_SRC ** Build systems A build system may work with Cakelisp, because Cakelisp outputs C/C++ source/header files. Cakelisp must run before your regular build system runs, or in a stage where Cakelisp can create and add files to the build. This is because Cakelisp handles its own modules such that adding support to an existing build system would be challenging.

Ideally, you should be able to rely on Cakelisp's built-in build system. This allows Cakelisp files to know how to build themselves. ** Debugging See [[file:doc/Debugging.org][doc/Debugging.org]]. Cakelisp doesn't really have an interpreter. Cakelisp always generates C/C++ code to do meaningful work. This means the Cakelisp transpiler, macros, generators, and final code output can be debugged using a regular C/C++ debugger like GDB, LLDB, or Visual Studio Debugger.

Mapping files will make it possible to step through code in the Cakelisp language (i.e. not in the generated language). This is similar to how debuggers allow you to step through code in C files, when under the hood it's actually stepping through machine code. It will require building support into your editor in order to properly jump to the right Cakelisp file and line (among other things).

• Similar applications/languages See [[file:doc/VsOtherLanguages.org][doc/VsOtherLanguages.org]] for projects similar to Cakelisp.