Support substituting translated shaders.

Add dumpTranslatedShaders and enableTranslatedShaderSubstitution
frontend ANGLE features, which allow ANGLE developers to prototype
optimizations and other transforms without fully implementing them in
the shader translator.

Tested on macOS with ANGLE's Metal backend, but should work with the
other source-level translator backends.

Add documentation for pre-existing substitution of shader sources, and
of translated shaders added in this CL.

Fixed: angleproject:8280
Change-Id: I24d5ef88a479b23e81cc8169fe813c263acfc71f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4731553
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>

doc/ShaderSubstitution.md

@@ -0,0 +1,104 @@
+# Shader substitution
+
+ANGLE provides two mechanisms for observing, modifying, and substituting
+the application's shaders. This ability to interpose makes it easier to
+diagnose bugs and to prototype new transforms in the shader translator.
+
+## Environment variables controlling reading/writing shaders to disk
+
+For both the source and translated shaders discussed below, the environment
+variable:
+
+```
+ANGLE_SHADER_DUMP_PATH
+```
+
+and the Android property:
+
+```
+debug.angle.shader_dump_path
+```
+
+specify the directory in which shader sources and translated shaders will
+be written to, and, in the case of shader substitution, read from. For
+example, on non-Android platforms:
+
+```
+mkdir -p /path/to/angle_shaders
+export ANGLE_SHADER_DUMP_PATH=/path/to/angle_shaders
+```
+
+will write all data to the `angle_shaders` directory.
+
+On Android, it's necessary to set the `debug.angle.shader_dump_path` property
+and set up the SD card correctly. (Help expanding this documentation is
+appreciated!)
+
+## ESSL shader dumping and substitution
+
+The ANGLE feature `dumpShaderSource`, when enabled, writes all incoming
+ESSL shader sources to disk, in the shader dump directory specified
+above. File names are computed by hashing the shader sources. Shaders will
+only be written to disk if they were not loaded from disk via substitution,
+below.
+
+The ANGLE feature `enableShaderSubstitution`, when enabled, looks for a
+file in the shader dump directory where the filename is the hash of the
+application's shader source, and substitutes its contents for the
+application's shader. This allows you to dump and edit these files at your
+leisure, and rerun the application to pick up the new versions of the
+shaders.
+
+In Chromium, pass the following command line arguments to enable these
+features:
+
+```
+--enable-angle-features=dumpShaderSource
+--enable-angle-features=enableShaderSubstitution
+--enable-angle-features=dumpShaderSource,enableShaderSubstitution
+```
+
+You must also specify `--disable-gpu-sandbox` to allow ANGLE to access
+these on-disk files for reading and writing. **Do not** browse the open web
+with this command line argument specified!
+
+## Translated shader dumping and substitution
+
+The translated shaders produced by ANGLE's shader translator can be dumped
+and substituted as well. This is especially useful when prototyping new
+optimizations in the shader translator.
+
+This mechanism is relatively recent and has not been thoroughly tested. It
+will likely not work in the situation where ANGLE dynamically recompiles
+shaders internally. It should work with all text-based shader translator
+backends (ESSL, GLSL, HLSL, and Metal). See comments in
+`src/libANGLE/Shader.cpp` describing the work needed to make this work with
+the SPIR-V backend.
+
+Translated shaders go into the same shader dump directory specified above
+for shader sources. To enable these features:
+
+```
+--enable-angle-features=dumpTranslatedShaders,enableTranslatedShaderSubstitution --disable-gpu-sandbox
+```
+
+## Putting it all together (example: macOS)
+
+```
+mkdir -p $HOME/tmp/angle_shaders
+export ANGLE_SHADER_DUMP_PATH=$HOME/tmp/angle_shaders
+
+out/Release/Chromium.app/Contents/MacOS/Chromium --disable-gpu-sandbox --use-angle=metal --enable-angle-features=dumpShaderSource,enableShaderSubstitution,dumpTranslatedShaders,enableTranslatedShaderSubstitution
+```
+
+Run the application once to generate the shader dump. Edit source or
+translated shaders as desired. Rerun with the same command line arguments
+to pick up the new versions of the shaders.
+
+Alternatively, and especially if the application doesn't work with all of
+the shaders in the substitution directory, make a new directory and copy in
+only those source or translated shaders you want to substitute, and run:
+
+```
+out/Release/Chromium.app/Contents/MacOS/Chromium --disable-gpu-sandbox --use-angle=metal --enable-angle-features=enableShaderSubstitution,enableTranslatedShaderSubstitution
+```

include/platform/autogen/FrontendFeatures_autogen.h

@@ -147,6 +147,20 @@ struct FrontendFeatures : FeatureSetBase
         &members, "http://anglebug.com/1423136"
     };
 
+    FeatureInfo dumpTranslatedShaders = {
+        "dumpTranslatedShaders",
+        FeatureCategory::FrontendFeatures,
+        "Write translated shaders to temp directory",
+        &members, "http://anglebug.com/8280"
+    };
+
+    FeatureInfo enableTranslatedShaderSubstitution = {
+        "enableTranslatedShaderSubstitution",
+        FeatureCategory::FrontendWorkarounds,
+        "Check the filesystem for translated shaders to use instead of the shader translator's",
+        &members, "http://anglebug.com/8280"
+    };
+
 };
 
 inline FrontendFeatures::FrontendFeatures()  = default;

include/platform/frontend_features.json

@@ -145,6 +145,22 @@
                 "Disables saving programs to the cache"
             ],
             "issue": "http://anglebug.com/1423136"
+        },
+        {
+            "name": "dump_translated_shaders",
+            "category": "Features",
+            "description": [
+                "Write translated shaders to temp directory"
+            ],
+            "issue": "http://anglebug.com/8280"
+        },
+        {
+            "name": "enable_translated_shader_substitution",
+            "category": "Workarounds",
+            "description": [
+                "Check the filesystem for translated shaders to use instead of the shader translator's"
+            ],
+            "issue": "http://anglebug.com/8280"
         }
     ]
 }

src/libANGLE/Shader.cpp

@@ -47,7 +47,7 @@ size_t ComputeShaderHash(const std::string &mergedSource)
     return std::hash<std::string>{}(mergedSource);
 }
 
-std::string GetShaderDumpFilePath(size_t shaderHash)
+std::string GetShaderDumpFilePath(size_t shaderHash, const char *suffix)
 {
     std::stringstream path;
     std::string shaderDumpDir = GetShaderDumpFileDirectory();
@@ -55,7 +55,7 @@ std::string GetShaderDumpFilePath(size_t shaderHash)
     {
         path << shaderDumpDir << "/";
     }
-    path << shaderHash << ".essl";
+    path << shaderHash << "." << suffix;
 
     return path.str();
 }
@@ -236,9 +236,10 @@ void Shader::setSource(const Context *context,
     const angle::FrontendFeatures &frontendFeatures = context->getFrontendFeatures();
 
     bool substitutedShader = false;
+    const char *suffix     = "essl";
     if (frontendFeatures.enableShaderSubstitution.enabled)
     {
-        std::string subsitutionShaderPath = GetShaderDumpFilePath(sourceHash);
+        std::string subsitutionShaderPath = GetShaderDumpFilePath(sourceHash, suffix);
 
         std::string substituteShader;
         if (angle::ReadFileToString(subsitutionShaderPath, &substituteShader))
@@ -253,7 +254,7 @@ void Shader::setSource(const Context *context,
     // back to the file.
     if (frontendFeatures.dumpShaderSource.enabled && !substitutedShader)
     {
-        std::string dumpFile = GetShaderDumpFilePath(sourceHash);
+        std::string dumpFile = GetShaderDumpFilePath(sourceHash, suffix);
 
         writeFile(dumpFile.c_str(), source.c_str(), source.length());
         INFO() << "Dumped shader source: " << dumpFile;
@@ -520,6 +521,52 @@ void Shader::resolveCompile(const Context *context)
     bool isBinaryOutput             = outputType == SH_SPIRV_VULKAN_OUTPUT;
     mState.mCompiledShaderState.buildCompiledShaderState(compilerHandle, isBinaryOutput);
 
+    const angle::FrontendFeatures &frontendFeatures = context->getFrontendFeatures();
+    bool substitutedTranslatedShader                = false;
+    const char *suffix                              = "translated";
+    if (frontendFeatures.enableTranslatedShaderSubstitution.enabled)
+    {
+        // To support reading/writing compiled binaries (SPIR-V
+        // representation), need more file input/output facilities,
+        // and figure out the byte ordering of writing the 32-bit
+        // words to disk.
+        if (isBinaryOutput)
+        {
+            INFO() << "Can not substitute compiled binary (SPIR-V) shaders yet";
+        }
+        else
+        {
+            std::string substituteShaderPath = GetShaderDumpFilePath(mState.mSourceHash, suffix);
+
+            std::string substituteShader;
+            if (angle::ReadFileToString(substituteShaderPath, &substituteShader))
+            {
+                mState.mCompiledShaderState.translatedSource = std::move(substituteShader);
+                substitutedTranslatedShader                  = true;
+                INFO() << "Trasnslated shader substitute found, loading from "
+                       << substituteShaderPath;
+            }
+        }
+    }
+
+    // Only dump translated shaders that have not been previously substituted. It would write the
+    // same data back to the file.
+    if (frontendFeatures.dumpTranslatedShaders.enabled && !substitutedTranslatedShader)
+    {
+        if (isBinaryOutput)
+        {
+            INFO() << "Can not dump compiled binary (SPIR-V) shaders yet";
+        }
+        else
+        {
+            std::string dumpFile = GetShaderDumpFilePath(mState.mSourceHash, suffix);
+
+            const std::string &translatedSource = mState.mCompiledShaderState.translatedSource;
+            writeFile(dumpFile.c_str(), translatedSource.c_str(), translatedSource.length());
+            INFO() << "Dumped translated source: " << dumpFile;
+        }
+    }
+
 #if !defined(NDEBUG)
     if (outputType != SH_SPIRV_VULKAN_OUTPUT)
     {

util/autogen/angle_features_autogen.cpp

@@ -99,6 +99,7 @@ constexpr PackedEnumMap<Feature, const char *> kFeatureNames = {{
     {Feature::DoubleDepthBiasConstantFactor, "doubleDepthBiasConstantFactor"},
     {Feature::DoWhileGLSLCausesGPUHang, "doWhileGLSLCausesGPUHang"},
     {Feature::DumpShaderSource, "dumpShaderSource"},
+    {Feature::DumpTranslatedShaders, "dumpTranslatedShaders"},
     {Feature::EglColorspaceAttributePassthrough, "eglColorspaceAttributePassthrough"},
     {Feature::EmulateAbsIntFunction, "emulateAbsIntFunction"},
     {Feature::EmulateAdvancedBlendEquations, "emulateAdvancedBlendEquations"},
@@ -133,6 +134,7 @@ constexpr PackedEnumMap<Feature, const char *> kFeatureNames = {{
     {Feature::EnableProgramBinaryForCapture, "enableProgramBinaryForCapture"},
     {Feature::EnableShaderSubstitution, "enableShaderSubstitution"},
     {Feature::EnableTimestampQueries, "enableTimestampQueries"},
+    {Feature::EnableTranslatedShaderSubstitution, "enableTranslatedShaderSubstitution"},
     {Feature::EnsureNonEmptyBufferIsBoundForDraw, "ensureNonEmptyBufferIsBoundForDraw"},
     {Feature::ExpandIntegerPowExpressions, "expandIntegerPowExpressions"},
     {Feature::ExplicitlyCastMediumpFloatTo16Bit, "explicitlyCastMediumpFloatTo16Bit"},

util/autogen/angle_features_autogen.h

@@ -99,6 +99,7 @@ enum class Feature
     DoubleDepthBiasConstantFactor,
     DoWhileGLSLCausesGPUHang,
     DumpShaderSource,
+    DumpTranslatedShaders,
     EglColorspaceAttributePassthrough,
     EmulateAbsIntFunction,
     EmulateAdvancedBlendEquations,
@@ -133,6 +134,7 @@ enum class Feature
     EnableProgramBinaryForCapture,
     EnableShaderSubstitution,
     EnableTimestampQueries,
+    EnableTranslatedShaderSubstitution,
     EnsureNonEmptyBufferIsBoundForDraw,
     ExpandIntegerPowExpressions,
     ExplicitlyCastMediumpFloatTo16Bit,