Building releases
=================

Guidelines for release coordinators to build official releases.

-  Target users: **Release coordinators**.
-  Relevant reading for: **Advanced engine users and contributors**, to
   understand how official builds are made.
-  Other relevant docs: `Building from source `__.

This document starts by presenting how to make release builds in the
most succint way possible, and later sections give more details on
relevant steps or components.

Build process
-------------

We’ll use ``4.4-beta1`` as the dev snapshot being built in this example.
The process is the same for dev snapshots and stable releases.

Login to the build server via SSH. Hostname IP and port are shared in
private.

Then the following commands are the exact set needed to make an official
build:

.. code:: bash

   screen
   cd godot-build-scripts/4.4
   cd git
   git checkout master -b 4.4-beta1
   git pull origin master
   cd ..

   # Start the build.
   sh build.sh -v 4.4-beta1 -g 4.4-beta1
   # Here it takes several hours, you can detach from the `screen` session with 'Ctrl+A' then 'D'.
   # When you resume with `screen -r`, confirm that all builds succeeded with:
   tail -n 1 out/logs/*

   # Package the release (rename and zip files, sign, notarize, etc.).
   sh build-release.sh -v 4.4-beta1 -t 4.4.beta1

   # If the above step succeeds, and only if this is meant to be published officially:
   sh publish-release.sh -v 4.4-beta1

Context to go deeper
--------------------

Reproducible builds via containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Our build pipeline is set up on a physical build server running Fedora.
All platforms are cross-compiled from Fedora using relevant
cross-compilation toolchains.

The Podman/Docker build containers (or images, we tend to use the terms
interchangeably) are built using the Dockerfiles and scripts in this
repository:

https://github.com/godotengine/build-containers

The build server already has the latest version of these containers
prebuilt. You can review the list with ``sudo podman images | less``,
for example:

.. code:: bash

   REPOSITORY                                       TAG                      IMAGE ID      CREATED        SIZE
   localhost/godot-web                              4.3-f40                  ff11195607d6  4 months ago   2.42 GB
   localhost/godot-ios                              4.3-f40                  a6d60e6ad94a  4 months ago   5.49 GB
   localhost/godot-osx                              4.3-f40                  e82d7cc9a7af  4 months ago   4.78 GB
   localhost/godot-android                          4.3-f40                  5b6d3f00e1b4  4 months ago   4.29 GB
   localhost/godot-windows                          4.3-f40                  0e4d07925ffe  4 months ago   2.46 GB
   localhost/godot-linux                            4.3-f40                  1b4e60457cee  4 months ago   2.8 GB
   localhost/godot-fedora                           4.3-f40                  5244d1fe13e6  4 months ago   1.01 GB
   registry.fedoraproject.org/fedora                40                       e442870eb306  4 months ago   233 MB

The “TAG” column is a version string defined at build time when calling
the ``build.sh`` script, in the form
``-``.

Each image can then be referred to as ``localhost/:``
e.g. \ ``localhost/godot-windows/4.3-f40``.

Once a minor version is released, we stick to the same prebuilt images
during its lifetime, to avoid introducing regressions from toolchain
updates.

So in the above example, the latest image is ``4.3-f40`` (used for Godot
4.3.x and based on Fedora 40), and there would likely be a set
eventually named ``4.4-f41`` with Fedora 41 as base and all toolchains
updated.

Check the README file in ``build-containers`` for more details, and a
list of the toolchains currently in use.

In-container build scripts
--------------------------

Once the containers are built locally, they can be used via the scripts
hosted here:

https://github.com/godotengine/godot-build-scripts

On the build server, we cloned this repository multiple times for each
Godot branch, in the ``~/godot-build-scripts//`` folders:

.. code:: bash

   $ ls ~/godot-build-scripts/
   3.5  3.6  4.0  4.1  4.2  4.3  4.4

Each of these is checked out on the relevant branch of
``godot-build-scripts``. Older branches may have slight variations to
the workflow described above.

Some files in these folders are ``.gitignore``\ d but quite important:

* ``config.sh`` is generated by copying ``config.sh.in`` and filling in the blanks. It contains all the details needed to sign, notarize and publish our builds. It’s also the file which defines which image version/tag we’re using for those builds (e.g. ``4.3-f40``).
* Signing keys for various platforms.

Building the binaries
~~~~~~~~~~~~~~~~~~~~~

The entry point to make builds is ``build.sh``, which is responsible
for:

* Downloading the dependencies in ``deps`` if missing.
    * Note: If you update dependencies in the script or at their download location, the easiest way to ensure things are up-to-date is to delete ``deps`` and let ``build.sh`` download everything again.
* Cloning the Godot Git repo in ``git`` (only needed the first time).
* Validating that the specified branch or commit matches the version we claim to build with ``-v``.
* Creating a tarball of the Godot branch or commit we specify with ``-g``.
* Starting builds for all platforms sequentially.

Builds are done by invoking ``podman`` (RedHat’s better Docker
interface) with relevant arguments and the respective scripts found in
``build-windows/build.sh``, ``build-linux/build.sh``, etc. Review those
scripts for an overview of what SCons options are used for each
platform.

   [!TIP] In a situation where you want to build for one specific
   platform or a subset of platforms and not all of them, the easiest
   way currently is to manually edit the main ``build.sh`` file (with
   ``nano`` or command line editor of choice), and comment out the
   ``${podman_run}`` lines you don’t want to build (or ``exit 0``
   early). Remember to reset those changes before making a full official
   build.

   You can also build only “classical” (no .NET) or “mono” builds with
   the ``-b classical|mono`` option.

Packaging the binaries
~~~~~~~~~~~~~~~~~~~~~~

The next step is to run ``build-release.sh``, which basically takes care
of copying, renaming, signing, notarizing, and zipping all editor and
template binaries in the proper way.

   [!TIP] Like ``build.sh``, this file can be hacked together if you
   ever need to do test builds only for some platforms. Just remember to
   reset the changes before making a full official build.

Publishing the binaries
~~~~~~~~~~~~~~~~~~~~~~~

Finally, the ``publish-release.sh`` script takes care of:

-  Uploading binaries to GitHub at
   https://github.com/godotengine/godot-builds

   -  It does this using the scripts in ``~/upload/godot-builds/``, and
      a GitHub write token saved in ``~/.bashrc`` for the @GodotBuilder
      account, which needs to be renewed occasionally.

-  Uploading the Web editor build to our web server. If the Web editor
   build should be marked as the latest stable one, pass the ``-l``
   argument to ``publish-release.sh``.
-  Publish the NuGet packages to NuGet Gallery.
-  Publish the Android library to MavenCentral.

For stable builds only, it additionally uploads the release to:

* GitHub at https://github.com/godotengine/godot
    * This requires getting a GitHub write token for one of our accounts. @GodotBuilder doesn’t have write access to godotengine/godot for safety.
* Steam
* EGS
* itch.io

On Steam, EGS, and itch.io, the builds need to be published manually in
their respective backends (including writing a release post for Steam
and itch.io). This is outside the scope of this howto.

   [!NOTE] Stable releases of the editor should also be uploaded to
   Google Play and the Horizon Store. Until we automate this, this
   should be synced with @m4gr3d.

Using ``screen``
----------------

``screen`` is a “window manager” for the terminal, which lets you have
one or multiple interactive shells which can persist even when logging
off from your main session.

This is useful over SSH, as the connection might disconnect
unexpectedly, or you might simply need to turn off the computer to go to
bed.

If you used ``screen`` before starting a build, you can “detach” from
the ``screen`` session, disconnect, and later on reconnect and re-attach
to the ongoing ``screen`` session which will have continued on the
remote host.

Basic commands to know:

* ``screen``: Start a session.
* ``screen -r``: Resume a previously detached session. If more than one are detached, you’ll need to specify a PID after ``-r``.
* ``screen -r -d``: Resume a session that wasn’t properly detached (e.g. due to the connection timing out).
* ``exit`` (within a ``screen`` session): Terminate the session, or if multiple virtual terminals were open, close the current one.

Within a ``screen`` session, commands can be issued using the Ctrl+A
hotkey. This hotkey starts an input mode without visual feedback, which
can take notably these commands:

* ``C``: Create a new virtual terminal in the current session.
* ``D``: Detach from the current session.
* ``0-9``: Go to the n-th virtual terminal. When creating a new session, there is only one virtual terminal (0), but after using the C command, you can cycle between 0 and 1, etc.
    * This is useful when one virtual terminal is busy with a multi-hour build, but you still want to do some checks in a separate terminal (e.g. ``tail -n 1 out/logs/*`` to review the progress).

See ``man screen`` for more details and possibly cool tricks.
