godot-maintainers-docs/release-management/building-releases.md

# Building releases

Guidelines for release coordinators to build official releases.

- Target users: **Release coordinators**.
- Relevant reading for: **Advanced engine users and contributors**, to understand how official builds are made.
- Other relevant docs: [Building from source](https://docs.godotengine.org/en/latest/contributing/development/compiling/).

This document starts by presenting how to make release builds in the most succint way possible, and later sections give more details on relevant steps or components.

## Build process

We'll use `4.4-beta1` as the dev snapshot being built in this example. The process is the same for dev snapshots and stable releases.

Login to the build server via SSH. Hostname IP and port are shared in private.

Then the following commands are the exact set needed to make an official build:

```bash
screen
cd godot-build-scripts/4.4
cd git
git checkout master -b 4.4-beta1
git pull origin master
cd ..

# Start the build.
sh build.sh -v 4.4-beta1 -g 4.4-beta1
# Here it takes several hours, you can detach from the `screen` session with 'Ctrl+A' then 'D'.
# When you resume with `screen -r`, confirm that all builds succeeded with:
tail -n 1 out/logs/*

# Package the release (rename and zip files, sign, notarize, etc.).
sh build-release.sh -v 4.4-beta1 -t 4.4.beta1

# If the above step succeeds, and only if this is meant to be published officially:
sh publish-release.sh -v 4.4-beta1
```

## Context to go deeper

### Reproducible builds via containers

Our build pipeline is set up on a physical build server running Fedora. All platforms are cross-compiled from Fedora using relevant cross-compilation toolchains.

The Podman/Docker build containers (or images, we tend to use the terms interchangeably) are built using the Dockerfiles and scripts in this repository:

https://github.com/godotengine/build-containers

The build server already has the latest version of these containers prebuilt. You can review the list with `sudo podman images | less`, for example:

```bash
REPOSITORY                                       TAG                      IMAGE ID      CREATED        SIZE
localhost/godot-web                              4.3-f40                  ff11195607d6  4 months ago   2.42 GB
localhost/godot-ios                              4.3-f40                  a6d60e6ad94a  4 months ago   5.49 GB
localhost/godot-osx                              4.3-f40                  e82d7cc9a7af  4 months ago   4.78 GB
localhost/godot-android                          4.3-f40                  5b6d3f00e1b4  4 months ago   4.29 GB
localhost/godot-windows                          4.3-f40                  0e4d07925ffe  4 months ago   2.46 GB
localhost/godot-linux                            4.3-f40                  1b4e60457cee  4 months ago   2.8 GB
localhost/godot-fedora                           4.3-f40                  5244d1fe13e6  4 months ago   1.01 GB
registry.fedoraproject.org/fedora                40                       e442870eb306  4 months ago   233 MB
```

The "TAG" column is a version string defined at build time when calling the `build.sh` script, in the form `<godot_branch>-<fedora_version>`.

Each image can then be referred to as `localhost/<name>:<tag>` e.g. `localhost/godot-windows/4.3-f40`.

Once a minor version is released, we stick to the same prebuilt images during its lifetime, to avoid introducing regressions from toolchain updates.

So in the above example, the latest image is `4.3-f40` (used for Godot 4.3.x and based on Fedora 40), and there would likely be a set eventually named `4.4-f41` with Fedora 41 as base and all toolchains updated.

Check the README file in `build-containers` for more details, and a list of the toolchains currently in use.

## In-container build scripts

Once the containers are built locally, they can be used via the scripts hosted here:

https://github.com/godotengine/godot-build-scripts

On the build server, we cloned this repository multiple times for each Godot branch, in the `~/godot-build-scripts/<branch>/` folders:

```bash
$ ls ~/godot-build-scripts/
3.5  3.6  4.0  4.1  4.2  4.3  4.4
```

Each of these is checked out on the relevant branch of `godot-build-scripts`. Older branches may have slight variations to the workflow described above.

Some files in these folders are `.gitignore`d but quite important:
- `config.sh` is generated by copying `config.sh.in` and filling in the blanks. It contains all the details needed to sign, notarize and publish our builds. It's also the file which defines which image version/tag we're using for those builds (e.g. `4.3-f40`).
- Signing keys for various platforms.

### Building the binaries

The entry point to make builds is `build.sh`, which is responsible for:
- Downloading the dependencies in `deps` if missing.
  * Note: If you update dependencies in the script or at their download location, the easiest way to ensure things are up-to-date is to delete `deps` and let `build.sh` download everything again.
- Cloning the Godot Git repo in `git` (only needed the first time).
- Validating that the specified branch or commit matches the version we claim to build with `-v`.
- Creating a tarball of the Godot branch or commit we specify with `-g`.
- Starting builds for all platforms sequentially.

Builds are done by invoking `podman` (RedHat's better Docker interface) with relevant arguments and the respective scripts found in `build-windows/build.sh`, `build-linux/build.sh`, etc. Review those scripts for an overview of what SCons options are used for each platform.

> [!TIP]
> In a situation where you want to build for one specific platform or a subset of platforms and not all of them, the easiest way currently is to manually edit the main `build.sh` file (with `nano` or command line editor of choice), and comment out the `${podman_run}` lines you don't want to build (or `exit 0` early). Remember to reset those changes before making a full official build.
>
> You can also build only "classical" (no .NET) or "mono" builds with the `-b classical|mono` option.

### Packaging the binaries

The next step is to run `build-release.sh`, which basically takes care of copying, renaming, signing, notarizing, and zipping all editor and template binaries in the proper way.

> [!TIP]
> Like `build.sh`, this file can be hacked together if you ever need to do test builds only for some platforms. Just remember to reset the changes before making a full official build.

### Publishing the binaries

Finally, the `publish-release.sh` script takes care of:

- Uploading binaries to GitHub at https://github.com/godotengine/godot-builds
  * It does this using the scripts in `~/upload/godot-builds/`, and a GitHub write token saved in `~/.bashrc` for the @GodotBuilder account, which needs to be renewed occasionally.
- Uploading the Web editor build to our web server. If the Web editor build should be marked as the latest stable one, pass the `-l` argument to `publish-release.sh`.
- Publish the NuGet packages to NuGet Gallery.
- Publish the Android library to MavenCentral.

For stable builds only, it additionally uploads the release to:
- GitHub at https://github.com/godotengine/godot
  * This requires getting a GitHub write token for one of our accounts. @GodotBuilder doesn't have write access to godotengine/godot for safety.
- Steam
- EGS
- itch.io

On Steam, EGS, and itch.io, the builds need to be published manually in their respective backends (including writing a release post for Steam and itch.io). This is outside the scope of this howto.

> [!NOTE]
> Stable releases of the editor should also be uploaded to Google Play and the Horizon Store. Until we automate this, this should be synced with @m4gr3d.

## Using `screen`

`screen` is a "window manager" for the terminal, which lets you have one or multiple interactive shells which can persist even when logging off from your main session.

This is useful over SSH, as the connection might disconnect unexpectedly, or you might simply need to turn off the computer to go to bed.

If you used `screen` before starting a build, you can "detach" from the `screen` session, disconnect, and later on reconnect and re-attach to the ongoing `screen` session which will have continued on the remote host.

Basic commands to know:
- `screen`: Start a session.
- `screen -r`: Resume a previously detached session. If more than one are detached, you'll need to specify a PID after `-r`.
- `screen -r -d`: Resume a session that wasn't properly detached (e.g. due to the connection timing out).
- `exit` (within a `screen` session): Terminate the session, or if multiple virtual terminals were open, close the current one.

Within a `screen` session, commands can be issued using the <kbd>Ctrl</kbd>+<kbd>A</kbd> hotkey. This hotkey starts an input mode without visual feedback, which can take notably these commands:
- <kbd>C</kbd>: Create a new virtual terminal in the current session.
- <kbd>D</kbd>: Detach from the current session.
- <kbd>0</kbd>-<kbd>9</kbd>: Go to the n-th virtual terminal. When creating a new session, there is only one virtual terminal (<kbd>0</kbd>), but after using the <kbd>C</kbd> command, you can cycle between <kbd>0</kbd> and <kbd>1</kbd>, etc.
  * This is useful when one virtual terminal is busy with a multi-hour build, but you still want to do some checks in a separate terminal (e.g. `tail -n 1 out/logs/*` to review the progress).

See `man screen` for more details and possibly cool tricks.