godotengine/godot-angle-static
1
0
Fork 0
mirror of https://github.com/godotengine/godot-angle-static.git synced 2026-01-06 02:09:55 +03:00
Code Issues Packages Projects Releases Wiki Activity

Add script to automatically download extension data.

Browse Source 
The script uses the 'bb' and 'swarming' tool to download build info
and artifacts. See the script for more info.

Bug: angleproject:6379
Change-Id: I65444771a69dc2f6eee39f6ba8d471fdd8ca2cff
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3198737
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
This commit is contained in:
Jamie Madill
2021-10-01 11:49:52 -04:00
committed by Angle LUCI CQ
parent 2a8be9de18
commit 0b4508e5d9
9 changed files with 270 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
doc/ExtensionSupport.md
View File

@@ -244,29 +244,29 @@ NVIDIA P400 Win10:
* `GL_RENDERER` is `ANGLE (NVIDIA, Vulkan 1.2.133 (NVIDIA Quadro P400 (0x00001CB3)), NVIDIA-451.48.0.0)`
* `GL_VENDOR` is `Google Inc. (NVIDIA)`
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)`
* Data updated September 22, 2021
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)`
* Data updated October 1, 2021
Intel 630 Win10:
* `GL_RENDERER` is `ANGLE (Intel, Vulkan 1.2.135 (Intel(R) HD Graphics 630 (0x00005912)), Intel Corporation-100.100.)`
* `GL_VENDOR` is `Google Inc. (Intel)`
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)`
* Data updated September 22, 2021
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16846 git hash: 22d8115a208b)`
* Data updated October 1, 2021
NVIDIA P400 Linux:
* `GL_RENDERER` is `ANGLE (NVIDIA, Vulkan 1.1.95 (NVIDIA Quadro P400 (0x00001CB3)), NVIDIA-418.56.0.0)`
* `GL_VENDOR` is `Google Inc. (NVIDIA)`
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)`
* Data updated September 22, 2021
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)`
* Data updated October 1, 2021
Intel 630 Linux:
* `GL_RENDERER` is `ANGLE (Intel, Vulkan 1.1.96 (Intel(R) HD Graphics 630 (Kaby Lake GT2) (0x00005912)), Intel open-source Mesa driver-19.0.2)`
* `GL_VENDOR` is `Google Inc. (Intel)`
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)`
* Data updated September 22, 2021
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)`
* Data updated October 1, 2021
SwiftShader Win10:
@@ -279,21 +279,30 @@ Pixel 4 Android 11:
* `GL_RENDERER` is `ANGLE (Qualcomm, Vulkan 1.1.128 (Adreno (TM) 640 (0x06040001)), Qualcomm Technologies Inc. Adreno Vulkan Driver-512.490.0)`
* `GL_VENDOR` is `Google Inc. (Qualcomm)`
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16782 git hash: 368c0ce94472)`
* Data updated September 23, 2021
* `GL_VERSION` is `OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)`
* Data updated October 1, 2021
## How to update supported extension data
Supported extension data is stored in the ANGLE repo as JSON files in
[`scripts/extension_data`](../scripts/extension_data). The JSON data is
sourced from public ANGLE test runs. Look for `angle_end2end_tests` in a bot
run: [example link]
(https://ci.chromium.org/ui/p/angle/builders/ci/win-clang-x64-rel/8183/overview).
run: [example link](https://ci.chromium.org/ui/p/angle/builders/ci/win-clang-x64-rel/8183/overview).
Search for "`angle_end2end_tests`", then click on the "cas output" and find
`GLinfo_ES3_2_Vulkan.json`. Copy and paste the file contents into a file in
the folder above.
`GLinfo_ES3_2_Vulkan.json`.
To update or add a new configuration, first retrieve the JSON data, modify
[`gen_extensions.py`](../src/libANGLE/gen_extensions.py) if necessary, then
run [`scripts/run_code_generation.py`](../scripts/run_code_generation.py) to
refresh generated files.
The Pixel 4 and GLES 3 NVIDIA and Intel data is automatically updated using
the [`update_extension_data.py`](../scripts/update_extension_data.py) script.
To use it first authenticate to the `bb` and `luci-go` tools by running `bb
auth-login` and `./tools/luci-go/swarming login`. Then run the script and
re-run [code generation][CodeGen].
The GLES 1 and SwiftShader data is currently manually updated. Find the relevant
file from the task output (see above) and overwrite the correspoding file.
Re-run [code generation][CodeGen] and create a CL as per our normal process.
To add a new configuration, first retrieve the JSON data, modify
[`gen_extensions.py`](../src/libANGLE/gen_extensions.py) as necessary, then
run [`scripts/run_code_generation.py`][CodeGen] to refresh generated files.
[CodeGen]: ../scripts/run_code_generation.py

14
scripts/code_generation_hashes/Extension_files.json
View File

@@ -1,6 +1,6 @@
{
"doc/ExtensionSupport.md":
"27b0d3eb9282a4b7bcbe4167fd9d3cae",
"1559e88e39e9f797c1dbde50e8bd6cf7",
"scripts/cl.xml":
"f923201d4ea3e1130763b19fa7faa7a2",
"scripts/egl.xml":
@@ -8,15 +8,15 @@
"scripts/egl_angle_ext.xml":
"5bcc01462b355d933cf3ada15198fb68",
"scripts/extension_data/intel_630_linux.json":
"ce0daa93e617ce36bffbcace2a56160e",
"4131eaca8a8e4fca9b0c5ff25b16026f",
"scripts/extension_data/intel_630_win10.json":
"34cf383e82e7ee8f5df22e57a6a3d9e8",
"bd89a6577d818c6d49f36df601cd99dd",
"scripts/extension_data/nvidia_p400_linux.json":
"664ee9c342f0ff9c2e1f883e79946307",
"d5a85767739d7404ffb76cbc9f3dc57d",
"scripts/extension_data/nvidia_p400_win10.json":
"0623157acbc7ba7e8614ef60e399d3be",
"8a8bf15d0fb0168d1eadbb228f32fe29",
"scripts/extension_data/pixel_4_android_11.json":
"3e57bd9a2368657007719c3cfe18ea7e",
"2c05f8906022c6c5f99b56a7678e3d03",
"scripts/extension_data/swiftshader_win10.json":
"396fde2ceb0c833fe2ce405c9f1bd901",
"scripts/extension_data/swiftshader_win10_gles1.json":
@@ -30,7 +30,7 @@
"scripts/wgl.xml":
"c36001431919e1c435f1215a85f7e1db",
"src/libANGLE/gen_extensions.py":
"31c929bbb221600a4ad33114db17364b",
"db7ed1c2245bba536e074a1cf2bb2590",
"src/libANGLE/gles_extensions_autogen.cpp":
"039e36b4e6f02839e380b07fc9cb2855",
"src/libANGLE/gles_extensions_autogen.h":

6
scripts/extension_data/intel_630_linux.json
View File

@@ -1,8 +1,8 @@
{
"Vendor": "Google Inc. (Intel)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Renderer": "ANGLE (Intel, Vulkan 1.1.96 (Intel(R) HD Graphics 630 (Kaby Lake GT2) (0x00005912)), Intel open-source Mesa driver-19.0.2)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Extensions": [
"GL_ANGLE_base_vertex_base_instance",
"GL_ANGLE_client_arrays",
@@ -146,5 +146,5 @@
"GL_OES_vertex_half_float",
"GL_OES_vertex_type_10_10_10_2"
],
"DateRecorded": "September 22, 2021"
"DateRecorded": "October 1, 2021"
}

6
scripts/extension_data/intel_630_win10.json
View File

@@ -1,8 +1,8 @@
{
"Vendor": "Google Inc. (Intel)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16846 git hash: 22d8115a208b)",
"Renderer": "ANGLE (Intel, Vulkan 1.2.135 (Intel(R) HD Graphics 630 (0x00005912)), Intel Corporation-100.100.)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16846 git hash: 22d8115a208b)",
"Extensions": [
"GL_ANGLE_base_vertex_base_instance",
"GL_ANGLE_client_arrays",
@@ -144,5 +144,5 @@
"GL_OVR_multiview",
"GL_OVR_multiview2"
],
"DateRecorded": "September 22, 2021"
"DateRecorded": "October 1, 2021"
}

6
scripts/extension_data/nvidia_p400_linux.json
View File

@@ -1,8 +1,8 @@
{
"Vendor": "Google Inc. (NVIDIA)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Renderer": "ANGLE (NVIDIA, Vulkan 1.1.95 (NVIDIA Quadro P400 (0x00001CB3)), NVIDIA-418.56.0.0)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Extensions": [
"GL_ANGLE_base_vertex_base_instance",
"GL_ANGLE_client_arrays",
@@ -143,5 +143,5 @@
"GL_OES_vertex_half_float",
"GL_OES_vertex_type_10_10_10_2"
],
"DateRecorded": "September 22, 2021"
"DateRecorded": "October 1, 2021"
}

6
scripts/extension_data/nvidia_p400_win10.json
View File

@@ -1,8 +1,8 @@
{
"Vendor": "Google Inc. (NVIDIA)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Renderer": "ANGLE (NVIDIA, Vulkan 1.2.133 (NVIDIA Quadro P400 (0x00001CB3)), NVIDIA-451.48.0.0)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16778 git hash: c3e0eafaf38d)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Extensions": [
"GL_ANGLE_base_vertex_base_instance",
"GL_ANGLE_client_arrays",
@@ -148,5 +148,5 @@
"GL_OVR_multiview",
"GL_OVR_multiview2"
],
"DateRecorded": "September 22, 2021"
"DateRecorded": "October 1, 2021"
}

6
scripts/extension_data/pixel_4_android_11.json
View File

@@ -1,8 +1,8 @@
{
"Vendor": "Google Inc. (Qualcomm)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16782 git hash: 368c0ce94472)",
"Version": "OpenGL ES 3.2.0 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Renderer": "ANGLE (Qualcomm, Vulkan 1.1.128 (Adreno (TM) 640 (0x06040001)), Qualcomm Technologies Inc. Adreno Vulkan Driver-512.490.0)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16782 git hash: 368c0ce94472)",
"ShaderLanguageVersion": "OpenGL ES GLSL ES 3.20 (ANGLE 2.1.16847 git hash: a097ee2ea7a9)",
"Extensions": [
"GL_ANGLE_base_vertex_base_instance",
"GL_ANGLE_client_arrays",
@@ -142,5 +142,5 @@
"GL_OVR_multiview",
"GL_OVR_multiview2"
],
"DateRecorded": "September 23, 2021"
"DateRecorded": "October 1, 2021"
}

204
scripts/update_extension_data.py Normal file
View File

@@ -0,0 +1,204 @@
#!/usr/bin/python3
#
# Copyright 2021 The ANGLE Project Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# update_extension_data.py:
# Downloads and updates auto-generated extension data.
import argparse
import logging
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
EXIT_SUCCESS = 0
EXIT_FAILURE = 1
TEST_SUITE = 'angle_end2end_tests'
BUILDERS = ['angle/ci/android-arm64-rel', 'angle/ci/win-clang-x64-rel', 'angle/ci/linux-clang-rel']
SWARMING_SERVER = 'chromium-swarm.appspot.com'
d = os.path.dirname
THIS_DIR = d(os.path.abspath(__file__))
ANGLE_ROOT_DIR = d(THIS_DIR)
# Host GPUs
INTEL_630 = '8086:5912'
NVIDIA_P400 = '10de:1cb3'
GPUS = [INTEL_630, NVIDIA_P400]
GPU_NAME_MAP = {INTEL_630: 'intel_630', NVIDIA_P400: 'nvidia_p400'}
# OSes
LINUX = 'Linux'
WINDOWS_10 = 'Windows-10'
BOT_OSES = [LINUX, WINDOWS_10]
BOT_OS_NAME_MAP = {LINUX: 'linux', WINDOWS_10: 'win10'}
# Devices
PIXEL_4 = 'flame'
DEVICES_TYPES = [PIXEL_4]
DEVICE_NAME_MAP = {PIXEL_4: 'pixel_4'}
# Device OSes
ANDROID_11 = 'R'
DEVICE_OSES = [ANDROID_11]
DEVICE_OS_NAME_MAP = {ANDROID_11: 'android_11'}
# Result names
INFO_FILES = [
'GLinfo_ES3_2_Vulkan.json',
'GLinfo_ES3_1_Vulkan.json',
]
LOG_LEVELS = ['WARNING', 'INFO', 'DEBUG']
def run_and_get_json(args):
logging.debug(' '.join(args))
output = subprocess.check_output(args)
return json.loads(output)
def get_bb():
return 'bb.bat' if os.name == 'nt' else 'bb'
def run_bb_and_get_output(*args):
bb_args = [get_bb()] + list(args)
return subprocess.check_output(bb_args, encoding='utf-8')
def run_bb_and_get_json(*args):
bb_args = [get_bb()] + list(args) + ['-json']
return run_and_get_json(bb_args)
def get_swarming():
swarming_bin = 'swarming.exe' if os.name == 'nt' else 'swarming'
return os.path.join(ANGLE_ROOT_DIR, 'tools', 'luci-go', swarming_bin)
def run_swarming(*args):
swarming_args = [get_swarming()] + list(args)
logging.debug(' '.join(swarming_args))
subprocess.check_call(swarming_args)
def run_swarming_and_get_json(*args):
swarming_args = [get_swarming()] + list(args)
return run_and_get_json(swarming_args)
def name_device(gpu, device_type):
if gpu:
return GPU_NAME_MAP[gpu]
else:
assert device_type
return DEVICE_NAME_MAP[device_type]
def name_os(bot_os, device_os):
if bot_os:
return BOT_OS_NAME_MAP[bot_os]
else:
assert device_os
return DEVICE_OS_NAME_MAP[device_os]
def get_props_string(gpu, bot_os, device_os, device_type):
d = {'gpu': gpu, 'os': bot_os, 'device os': device_os, 'device': device_type}
return ', '.join('%s %s' % (k, v) for (k, v) in d.items() if v)
def collect_task_and_update_json(task_id, gpu, bot_os, device_os, device_type):
logging.info('Found task with ID: %s, %s' %
(task_id, get_props_string(gpu, bot_os, device_os, device_type)))
target_file_name = '%s_%s.json' % (name_device(gpu, device_type), name_os(bot_os, device_os))
target_file = os.path.join(THIS_DIR, 'extension_data', target_file_name)
with tempfile.TemporaryDirectory() as tempdirname:
run_swarming('collect', '-S', SWARMING_SERVER, '-output-dir=%s' % tempdirname, task_id)
task_dir = os.path.join(tempdirname, task_id)
found = False
for fname in os.listdir(task_dir):
if fname in INFO_FILES:
if found:
logging.warning('Multiple candidates found for %s' % target_file_name)
return
else:
logging.info('%s -> %s' % (fname, target_file))
found = True
source_file = os.path.join(task_dir, fname)
shutil.copy(source_file, target_file)
def get_intersect_or_none(list_a, list_b):
i = [v for v in list_a if v in list_b]
assert not i or len(i) == 1
return i[0] if i else None
def main():
parser = argparse.ArgumentParser(description='Pulls extension support data from ANGLE CI.')
parser.add_argument(
'-v', '--verbose', help='Print additional debugging into.', action='count', default=0)
args = parser.parse_args()
if args.verbose >= len(LOG_LEVELS):
args.verbose = len(LOG_LEVELS) - 1
logging.basicConfig(level=LOG_LEVELS[args.verbose])
name_expr = re.compile(r'^' + TEST_SUITE + r' on (.*) on (.*)$')
for builder in BUILDERS:
# Step 1: Find the build ID.
# We list two builds using 'bb ls' and take the second, to ensure the build is finished.
ls_output = run_bb_and_get_output('ls', builder, '-n', '2', '-id')
build_id = ls_output.splitlines()[1]
logging.info('%s: build id %s' % (builder, build_id))
# Step 2: Get the test suite swarm hashes.
# 'bb get' returns build properties, including cloud storage identifiers for this test suite.
get_json = run_bb_and_get_json('get', build_id, '-p')
test_suite_hash = get_json['output']['properties']['swarm_hashes'][TEST_SUITE]
logging.info('Found swarm hash: %s' % test_suite_hash)
# Step 3: Find all tasks using the swarm hashes.
# 'swarming tasks' can find instances of the test suite that ran on specific systems.
task_json = run_swarming_and_get_json('tasks', '-tag', 'data:%s' % test_suite_hash, '-S',
SWARMING_SERVER)
# Step 4: Download the extension data for each configuration we're monitoring.
# 'swarming collect' downloads test artifacts to a temporary directory.
for task in task_json:
gpu = None
bot_os = None
device_os = None
device_type = None
for bot_dim in task['bot_dimensions']:
if bot_dim['key'] == 'gpu':
logging.debug(bot_dim['value'])
gpu = get_intersect_or_none(GPUS, bot_dim['value'])
if bot_dim['key'] == 'os':
logging.debug(bot_dim['value'])
bot_os = get_intersect_or_none(BOT_OSES, bot_dim['value'])
if bot_dim['key'] == 'device_os':
logging.debug(bot_dim['value'])
device_os = get_intersect_or_none(DEVICE_OSES, bot_dim['value'])
if bot_dim['key'] == 'device_type':
logging.debug(bot_dim['value'])
device_type = get_intersect_or_none(DEVICES_TYPES, bot_dim['value'])
if (gpu or device_type) and (bot_os or device_os):
collect_task_and_update_json(task['task_id'], gpu, bot_os, device_os, device_type)
return EXIT_SUCCESS
if __name__ == '__main__':
sys.exit(main())

25
src/libANGLE/gen_extensions.py
View File

@@ -229,16 +229,25 @@ list of supported EGL extensions in ANGLE's front-end see
Supported extension data is stored in the ANGLE repo as JSON files in
[`scripts/extension_data`](../scripts/extension_data). The JSON data is
sourced from public ANGLE test runs. Look for `angle_end2end_tests` in a bot
run: [example link]
(https://ci.chromium.org/ui/p/angle/builders/ci/win-clang-x64-rel/8183/overview).
run: [example link](https://ci.chromium.org/ui/p/angle/builders/ci/win-clang-x64-rel/8183/overview).
Search for "`angle_end2end_tests`", then click on the "cas output" and find
`GLinfo_ES3_2_Vulkan.json`. Copy and paste the file contents into a file in
the folder above.
`GLinfo_ES3_2_Vulkan.json`.
To update or add a new configuration, first retrieve the JSON data, modify
[`gen_extensions.py`](../src/libANGLE/gen_extensions.py) if necessary, then
run [`scripts/run_code_generation.py`](../scripts/run_code_generation.py) to
refresh generated files.
The Pixel 4 and GLES 3 NVIDIA and Intel data is automatically updated using
the [`update_extension_data.py`](../scripts/update_extension_data.py) script.
To use it first authenticate to the `bb` and `luci-go` tools by running `bb
auth-login` and `./tools/luci-go/swarming login`. Then run the script and
re-run [code generation][CodeGen].
The GLES 1 and SwiftShader data is currently manually updated. Find the relevant
file from the task output (see above) and overwrite the correspoding file.
Re-run [code generation][CodeGen] and create a CL as per our normal process.
To add a new configuration, first retrieve the JSON data, modify
[`gen_extensions.py`](../src/libANGLE/gen_extensions.py) as necessary, then
run [`scripts/run_code_generation.py`][CodeGen] to refresh generated files.
[CodeGen]: ../scripts/run_code_generation.py
"""
_MD_GLES_GPU_CONFIGS = [