godotengine/godot-angle-static
1
0
Fork 0
mirror of https://github.com/godotengine/godot-angle-static.git synced 2026-01-07 06:09:57 +03:00
Code Issues Packages Projects Releases Wiki Activity

Roll chromium_revision 6c5859c895..399855f4bb (890175:891326)

Browse Source 
Manual changes:
1. Fix generate_test_spec_json.py to correctly handle ADDITIONAL_MIXINS
   Needed after angle_skia_gold_test mixin removed from Chromium in
   crrev.com/890333
2. Copy third_party/logdog from Chromium
   Needed after crrev.com/890539 in order to create isolates on Android
   and for tools/perf/process_perf_results.py

Change log: 6c5859c895..399855f4bb
Full diff: 6c5859c895..399855f4bb

Changed dependencies
* build: 8870cb4120..2192a63c23
* buildtools: c793cca886..9d8449e380
* buildtools/linux64: git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810..git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa
* buildtools/mac: git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810..git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa
* buildtools/third_party/libc++abi/trunk: 7e3b76855b..2c53623d59
* buildtools/win: git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810..git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa
* testing: a62f8260df..ab567bcc84
* third_party/abseil-cpp: 7949d87093..bd17c406ba
* third_party/android_deps: 887e8d9009..fa47598982
* third_party/android_sdk: 1cfc90728e..816daa2545
* third_party/catapult: https://chromium.googlesource.com/catapult.git/+log/{catapult_..17cf72ca75
* third_party/depot_tools: b508ecd932..6d099d543d
* third_party/nasm: 19f3fad68d..e9be5fd6d7
* third_party/protobuf: 82f8803671..f4241bd0f2
* third_party/turbine: _iPtB_ThhxlMOt2TsYqVppwriEEn0mp-NUNRwDwYLUAC..Om6yIEXgJxuqghErK29h9RcMH6VaymMbxwScwXmcN6EC
* tools/clang: 09481f56be..fd14318cc9
* tools/luci-go: git_revision:2cc9805d5ad186367461ef1c4f0c59098b450418..git_revision:725192cc79f07ea946e10a64baac06625c206968
* tools/luci-go: git_revision:2cc9805d5ad186367461ef1c4f0c59098b450418..git_revision:725192cc79f07ea946e10a64baac06625c206968
* tools/luci-go: git_revision:2cc9805d5ad186367461ef1c4f0c59098b450418..git_revision:725192cc79f07ea946e10a64baac06625c206968
* tools/mb: 94630dfc19..12c8fa872b
* tools/perf: 5c84710692..2e6477d751
DEPS diff: 6c5859c895..399855f4bb/DEPS

Clang version changed llvmorg-13-init-11999-g50c0aaed:llvmorg-13-init-12491-g055770d5
Details: 09481f56be..fd14318cc9/scripts/update.py

Bug: angleproject:4483, angleproject:6037
Change-Id: I9035126bce55642d4dfce54eeace85093bdd1782
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2954241
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
This commit is contained in:
Yuly Novikov
2021-06-10 16:09:01 -04:00
committed by Angle LUCI CQ
parent ac704c8003
commit 166dd0c75c
12 changed files with 997 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
DEPS
View File

@@ -27,7 +27,7 @@ vars = {
'checkout_android_native_support': 'checkout_android or checkout_chromeos',
# Version of Chromium our Chromium-based DEPS are mirrored from.
'chromium_revision': '6c5859c895f578a3fecf477f3b33e4e1720003c3',
'chromium_revision': '399855f4bba6e724e465aca24b4b18fd284a2b98',
# We never want to checkout chromium,
# but need a dummy DEPS entry for the autoroller
'dummy_checkout_chromium': False,
@@ -72,12 +72,12 @@ vars = {
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling catapult
# and whatever else without interference from each other.
'catapult_revision': 'd598390f022179e4b4dcdf505c8545650f63d47d',
'catapult_revision': '17cf72ca75dd1b7b8c16f3b5376d2185bd72a2c3',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling luci-go
# and whatever else without interference from each other.
'luci_go': 'git_revision:2cc9805d5ad186367461ef1c4f0c59098b450418',
'luci_go': 'git_revision:725192cc79f07ea946e10a64baac06625c206968',
# Three lines of non-changing comments so that
# the commit queue can handle CLs rolling android_sdk_build-tools_version
@@ -112,12 +112,12 @@ vars = {
deps = {
'build': {
'url': '{chromium_git}/chromium/src/build.git@8870cb41201b9f4a384188af6d3c110ef083dba7',
'url': '{chromium_git}/chromium/src/build.git@2192a63c23ac58bae5f85cc11f537adb7834d326',
'condition': 'not build_with_chromium',
},
'buildtools': {
'url': '{chromium_git}/chromium/src/buildtools.git@c793cca886a1d820d03b3cb440d2f1c26a3381cc',
'url': '{chromium_git}/chromium/src/buildtools.git@9d8449e380d493e074e6cabdaef9a2e5ccd5c4f3',
'condition': 'not build_with_chromium',
},
@@ -130,7 +130,7 @@ deps = {
'packages': [
{
'package': 'gn/gn/linux-amd64',
'version': 'git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810',
'version': 'git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa',
}
],
'dep_type': 'cipd',
@@ -141,7 +141,7 @@ deps = {
'packages': [
{
'package': 'gn/gn/mac-${{arch}}',
'version': 'git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810',
'version': 'git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa',
}
],
'dep_type': 'cipd',
@@ -154,7 +154,7 @@ deps = {
},
'buildtools/third_party/libc++abi/trunk': {
'url': '{chromium_git}/external/github.com/llvm/llvm-project/libcxxabi.git@7e3b76855b76ce7b4c975df50674734357056612',
'url': '{chromium_git}/external/github.com/llvm/llvm-project/libcxxabi.git@2c53623d59004dd0482dfc20df6ef00c76341db3',
'condition': 'not build_with_chromium',
},
@@ -162,7 +162,7 @@ deps = {
'packages': [
{
'package': 'gn/gn/windows-amd64',
'version': 'git_revision:39a87c0b36310bdf06b692c098f199a0d97fc810',
'version': 'git_revision:393dab000d704a4364d085fa4c01ec7af176c8fa',
}
],
'dep_type': 'cipd',
@@ -170,12 +170,12 @@ deps = {
},
'testing': {
'url': '{chromium_git}/chromium/src/testing@a62f8260df45a43023b00ee1fcc2ad19f8ae1146',
'url': '{chromium_git}/chromium/src/testing@ab567bcc84a3d5cd12f576a8e405a5ef88bab351',
'condition': 'not build_with_chromium',
},
'third_party/abseil-cpp': {
'url': '{chromium_git}/chromium/src/third_party/abseil-cpp@7949d870935409c5c195447bc23ad1bf3ff80c91',
'url': '{chromium_git}/chromium/src/third_party/abseil-cpp@bd17c406bae17650d19bee34ee87c592714ada94',
'condition': 'not build_with_chromium',
},
@@ -218,7 +218,7 @@ deps = {
},
'third_party/android_deps': {
'url': '{chromium_git}/chromium/src/third_party/android_deps@887e8d90095085239a43c2aa1c27d2394d933d2b',
'url': '{chromium_git}/chromium/src/third_party/android_deps@fa4759898250078804382ecaa7d31dcdfc5f8d10',
'condition': 'checkout_android and not build_with_chromium',
},
@@ -233,7 +233,7 @@ deps = {
},
'third_party/android_sdk': {
'url': '{chromium_git}/chromium/src/third_party/android_sdk@1cfc90728e0c42cbd68d8c900a3d46d5f8ba86ec',
'url': '{chromium_git}/chromium/src/third_party/android_sdk@816daa254586f27a5746133f20a4aaba9f44b632',
'condition': 'checkout_android and not build_with_chromium',
},
@@ -320,7 +320,7 @@ deps = {
},
'third_party/depot_tools': {
'url': '{chromium_git}/chromium/tools/depot_tools.git@b508ecd932fd2653b4d3e9bccd80b3b7ac98c36a',
'url': '{chromium_git}/chromium/tools/depot_tools.git@6d099d543d26bef583e58c9f55a6cf7b6c5d6207',
'condition': 'not build_with_chromium',
},
@@ -406,7 +406,7 @@ deps = {
},
'third_party/nasm': {
'url': '{chromium_git}/chromium/deps/nasm.git@19f3fad68da99277b2882939d3b2fa4c4b8d51d9',
'url': '{chromium_git}/chromium/deps/nasm.git@e9be5fd6d723a435ca2da162f9e0ffcb688747c1',
'condition': 'not build_with_chromium',
},
@@ -427,7 +427,7 @@ deps = {
},
'third_party/protobuf': {
'url': '{chromium_git}/chromium/src/third_party/protobuf@82f8803671681950cbc32b4b681fa6ae0bb1febe',
'url': '{chromium_git}/chromium/src/third_party/protobuf@f4241bd0f2642a3e90c3c7b428c1b6cbd083d2b4',
'condition': 'not build_with_chromium',
},
@@ -492,7 +492,7 @@ deps = {
'packages': [
{
'package': 'chromium/third_party/turbine',
'version': '_iPtB_ThhxlMOt2TsYqVppwriEEn0mp-NUNRwDwYLUAC',
'version': 'Om6yIEXgJxuqghErK29h9RcMH6VaymMbxwScwXmcN6EC',
},
],
'condition': 'checkout_android and not build_with_chromium',
@@ -524,7 +524,7 @@ deps = {
},
'tools/clang': {
'url': '{chromium_git}/chromium/src/tools/clang.git@09481f56be7b1bbaf5a466be5d81691902825fcf',
'url': '{chromium_git}/chromium/src/tools/clang.git@fd14318cc948b67e5ec8a0f2f7563142ccd6dac2',
'condition': 'not build_with_chromium',
},
@@ -559,7 +559,7 @@ deps = {
},
'tools/mb': {
'url': '{chromium_git}/chromium/src/tools/mb@94630dfc19de559cb3603f0f7a7c9e0ba2068d49',
'url': '{chromium_git}/chromium/src/tools/mb@12c8fa872b58311e03a0307c86d641bf9b75e3a2',
'condition': 'not build_with_chromium',
},
@@ -574,7 +574,7 @@ deps = {
},
'tools/perf': {
'url': '{chromium_git}/chromium/src/tools/perf@5c84710692637a3b3618e88b7d145b2189da71c9',
'url': '{chromium_git}/chromium/src/tools/perf@2e6477d751563260f32470dec52c8407a0b8f662',
'condition': 'not build_with_chromium',
},
@@ -3077,7 +3077,7 @@ deps = {
'packages': [
{
'package': 'chromium/third_party/android_deps/libs/org_jetbrains_kotlin_kotlin_stdlib',
'version': 'version:2@1.4.32.cr0',
'version': 'version:2@1.5.10.cr0',
},
],
'condition': 'checkout_android and not build_with_chromium',
@@ -3088,7 +3088,7 @@ deps = {
'packages': [
{
'package': 'chromium/third_party/android_deps/libs/org_jetbrains_kotlin_kotlin_stdlib_common',
'version': 'version:2@1.4.32.cr0',
'version': 'version:2@1.5.10.cr0',
},
],
'condition': 'checkout_android and not build_with_chromium',

7
infra/specs/generate_test_spec_json.py
View File

@@ -104,14 +104,13 @@ def main():
assert isinstance(test, dict)
seen_mixins = seen_mixins.union(test.get('mixins', set()))
found_mixins = {}
found_mixins = ADDITIONAL_MIXINS.copy()
for mixin in seen_mixins:
if mixin in found_mixins:
continue
assert (mixin in chromium_generator.mixins)
found_mixins[mixin] = chromium_generator.mixins[mixin]
for mixin_name, mixin in ADDITIONAL_MIXINS.items():
found_mixins[mixin_name] = mixin
pp = pprint.PrettyPrinter(indent=2)
format_data = {

4
scripts/code_generation_hashes/Test_spec_JSON.json
View File

@@ -2,7 +2,7 @@
"infra/specs/angle.json":
"af1d088e331b9c7865a245c36bb443df",
"infra/specs/generate_test_spec_json.py":
"162566b21bca4ef0b815e411920c9f2d",
"348921bf70270ec6ee51fbb7e97d6925",
"infra/specs/mixins.pyl":
"937e107ab606846d61eec617d09e50d0",
"infra/specs/test_suite_exceptions.pyl":
@@ -16,5 +16,5 @@
"testing/buildbot/generate_buildbot_json.py":
"9f5d39d58f4a8baf4b74349147feb606",
"testing/buildbot/mixins.pyl":
"d6f859d47f1823376217f335101b22ea"
"bcd371a244be7e942852a60366f62c77"
}

1
third_party/logdog/OWNERS vendored Normal file
View File

@@ -0,0 +1 @@
file://build/android/OWNERS

15
third_party/logdog/README.chromium vendored Normal file
View File

@@ -0,0 +1,15 @@
Name: logdog
Short Name: logdog
URL: https://chromium.googlesource.com/infra/luci/luci-py/client/libs/logdog
Version: 9a84af84d3fa62b230569cf1d3abf69cc7c576e2
Revision: 9a84af84d3fa62b230569cf1d3abf69cc7c576e2
License: Apache 2.0
License File: NOT_SHIPPED
Security Critical: no
Description:
This is used from build/android/pylib/utils/logdog_helper.py
Local Modifications:
See get.sh and this files is also generated by the script.

38
third_party/logdog/get.sh vendored Executable file
View File

@@ -0,0 +1,38 @@
#!/bin/bash
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
set -eux
revision=9a84af84d3fa62b230569cf1d3abf69cc7c576e2
cd $(dirname $0)
rm -rf logdog
git clone https://chromium.googlesource.com/infra/luci/luci-py/client/libs/logdog
(
cd logdog
git checkout $revision
# remove unnecessary files.
rm -rf .git tests
)
cat <<EOF > README.chromium
Name: logdog
Short Name: logdog
URL: https://chromium.googlesource.com/infra/luci/luci-py/client/libs/logdog
Version: $revision
Revision: $revision
License: Apache 2.0
License File: NOT_SHIPPED
Security Critical: no
Description:
This is used from build/android/pylib/utils/logdog_helper.py
Local Modifications:
See get.sh and this files is also generated by the script.
EOF

1
third_party/logdog/logdog/OWNERS vendored Normal file
View File

@@ -0,0 +1 @@
iannucci@chromium.org

3
third_party/logdog/logdog/__init__.py vendored Normal file
View File

@@ -0,0 +1,3 @@
# Copyright 2016 The LUCI Authors. All rights reserved.
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.

90
third_party/logdog/logdog/bootstrap.py vendored Normal file
View File

@@ -0,0 +1,90 @@
# Copyright 2016 The LUCI Authors. All rights reserved.
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
import collections
import os
from . import stream, streamname
class NotBootstrappedError(RuntimeError):
"""Raised when the current environment is missing Butler bootstrap variables.
"""
_ButlerBootstrapBase = collections.namedtuple(
'_ButlerBootstrapBase',
('project', 'prefix', 'streamserver_uri', 'coordinator_host', 'namespace'))
class ButlerBootstrap(_ButlerBootstrapBase):
"""Loads LogDog Butler bootstrap parameters from the environment.
LogDog Butler adds variables describing the LogDog stream parameters to the
environment when it bootstraps an application. This class probes the
environment and identifies those parameters.
"""
# TODO(iannucci): move all of these to LUCI_CONTEXT
_ENV_PROJECT = 'LOGDOG_STREAM_PROJECT'
_ENV_PREFIX = 'LOGDOG_STREAM_PREFIX'
_ENV_STREAM_SERVER_PATH = 'LOGDOG_STREAM_SERVER_PATH'
_ENV_COORDINATOR_HOST = 'LOGDOG_COORDINATOR_HOST'
_ENV_NAMESPACE = 'LOGDOG_NAMESPACE'
@classmethod
def probe(cls, env=None):
"""Returns (ButlerBootstrap): The probed bootstrap environment.
Args:
env (dict): The environment to probe. If None, `os.getenv` will be used.
Raises:
NotBootstrappedError if the current environment is not boostrapped.
"""
if env is None:
env = os.environ
def _check(kind, val):
if not val:
return val
try:
streamname.validate_stream_name(val)
return val
except ValueError as exp:
raise NotBootstrappedError('%s (%s) is invalid: %s' % (kind, val, exp))
streamserver_uri = env.get(cls._ENV_STREAM_SERVER_PATH)
if not streamserver_uri:
raise NotBootstrappedError('No streamserver in bootstrap environment.')
return cls(
project=env.get(cls._ENV_PROJECT, ''),
prefix=_check("Prefix", env.get(cls._ENV_PREFIX, '')),
streamserver_uri=streamserver_uri,
coordinator_host=env.get(cls._ENV_COORDINATOR_HOST, ''),
namespace=_check("Namespace", env.get(cls._ENV_NAMESPACE, '')))
def stream_client(self, reg=None):
"""Returns: (StreamClient) stream client for the bootstrap streamserver URI.
If the Butler accepts external stream connections, it will export a
streamserver URI in the environment. This will create a StreamClient
instance to operate on the streamserver if one is defined.
Args:
reg (stream.StreamProtocolRegistry or None): The stream protocol registry
to use to create the stream. If None, the default global registry will
be used (recommended).
Raises:
ValueError: If no streamserver URI is present in the environment.
"""
reg = reg or stream._default_registry
return reg.create(
self.streamserver_uri,
project=self.project,
prefix=self.prefix,
coordinator_host=self.coordinator_host,
namespace=self.namespace)

561
third_party/logdog/logdog/stream.py vendored Normal file
View File

@@ -0,0 +1,561 @@
# Copyright 2016 The LUCI Authors. All rights reserved.
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
import collections
import contextlib
import json
import os
import posixpath
import socket
import sys
import threading
import time
from . import streamname, varint
if sys.platform == "win32":
from ctypes import GetLastError
_StreamParamsBase = collections.namedtuple('_StreamParamsBase',
('name', 'type', 'content_type', 'tags'))
# Magic number at the beginning of a Butler stream
#
# See "ProtocolFrameHeaderMagic" in:
# <luci-go>/logdog/client/butlerlib/streamproto
BUTLER_MAGIC = 'BTLR1\x1e'
class StreamParams(_StreamParamsBase):
"""Defines the set of parameters to apply to a new stream."""
# A text content stream.
TEXT = 'text'
# A binary content stream.
BINARY = 'binary'
# A datagram content stream.
DATAGRAM = 'datagram'
@classmethod
def make(cls, **kwargs):
"""Returns (StreamParams): A new StreamParams instance with supplied values.
Any parameter that isn't supplied will be set to None.
Args:
kwargs (dict): Named parameters to apply.
"""
return cls(**{f: kwargs.get(f) for f in cls._fields})
def validate(self):
"""Raises (ValueError): if the parameters are not valid."""
streamname.validate_stream_name(self.name)
if self.type not in (self.TEXT, self.BINARY, self.DATAGRAM):
raise ValueError('Invalid type (%s)' % (self.type,))
if self.tags is not None:
if not isinstance(self.tags, collections.Mapping):
raise ValueError('Invalid tags type (%s)' % (self.tags,))
for k, v in self.tags.items():
streamname.validate_tag(k, v)
def to_json(self):
"""Returns (str): The JSON representation of the StreamParams.
Converts stream parameters to JSON for Butler consumption.
Raises:
ValueError: if these parameters are not valid.
"""
self.validate()
obj = {
'name': self.name,
'type': self.type,
}
def _maybe_add(key, value):
if value is not None:
obj[key] = value
_maybe_add('contentType', self.content_type)
_maybe_add('tags', self.tags)
# Note that "dumps' will dump UTF-8 by default, which is what Butler wants.
return json.dumps(obj, sort_keys=True, ensure_ascii=True, indent=None)
class StreamProtocolRegistry(object):
"""Registry of streamserver URI protocols and their client classes.
"""
def __init__(self):
self._registry = {}
def register_protocol(self, protocol, client_cls):
assert issubclass(client_cls, StreamClient)
if self._registry.get(protocol) is not None:
raise KeyError('Duplicate protocol registered.')
self._registry[protocol] = client_cls
def create(self, uri, **kwargs):
"""Returns (StreamClient): A stream client for the specified URI.
This uses the default StreamProtocolRegistry to instantiate a StreamClient
for the specified URI.
Args:
uri (str): The streamserver URI.
kwargs: keyword arguments to forward to the stream. See
StreamClient.__init__.
Raises:
ValueError: if the supplied URI references an invalid or improperly
configured streamserver.
"""
uri = uri.split(':', 1)
if len(uri) != 2:
raise ValueError('Invalid stream server URI [%s]' % (uri,))
protocol, value = uri
client_cls = self._registry.get(protocol)
if not client_cls:
raise ValueError('Unknown stream client protocol (%s)' % (protocol,))
return client_cls._create(value, **kwargs)
# Default (global) registry.
_default_registry = StreamProtocolRegistry()
create = _default_registry.create
class StreamClient(object):
"""Abstract base class for a streamserver client.
"""
class _StreamBase(object):
"""ABC for StreamClient streams."""
def __init__(self, stream_client, params):
self._stream_client = stream_client
self._params = params
@property
def params(self):
"""Returns (StreamParams): The stream parameters."""
return self._params
@property
def path(self):
"""Returns (streamname.StreamPath): The stream path.
Raises:
ValueError: if the stream path is invalid, or if the stream prefix is
not defined in the client.
"""
return self._stream_client.get_stream_path(self._params.name)
def get_viewer_url(self):
"""Returns (str): The viewer URL for this stream.
Raises:
KeyError: if information needed to construct the URL is missing.
ValueError: if the stream prefix or name do not form a valid stream
path.
"""
return self._stream_client.get_viewer_url(self._params.name)
class _BasicStream(_StreamBase):
"""Wraps a basic file descriptor, offering "write" and "close"."""
def __init__(self, stream_client, params, fd):
super(StreamClient._BasicStream, self).__init__(stream_client, params)
self._fd = fd
@property
def fd(self):
return self._fd
def fileno(self):
return self._fd.fileno()
def write(self, data):
return self._fd.write(data)
def close(self):
return self._fd.close()
class _DatagramStream(_StreamBase):
"""Wraps a stream object to write length-prefixed datagrams."""
def __init__(self, stream_client, params, fd):
super(StreamClient._DatagramStream, self).__init__(stream_client, params)
self._fd = fd
def send(self, data):
varint.write_uvarint(self._fd, len(data))
self._fd.write(data)
def close(self):
return self._fd.close()
def __init__(self, project=None, prefix=None, coordinator_host=None, namespace=''):
"""Constructs a new base StreamClient instance.
Args:
project (str or None): If not None, the name of the log stream project.
prefix (str or None): If not None, the log stream session prefix.
coordinator_host (str or None): If not None, the name of the Coordinator
host that this stream client is bound to. This will be used to
construct viewer URLs for generated streams.
namespace (str): The prefix to apply to all streams opened by this client.
"""
self._project = project
self._prefix = prefix
self._coordinator_host = coordinator_host
self._namespace = namespace
self._name_lock = threading.Lock()
self._names = set()
@property
def project(self):
"""Returns (str or None): The stream project, or None if not configured."""
return self._project
@property
def prefix(self):
"""Returns (str or None): The stream prefix, or None if not configured."""
return self._prefix
@property
def coordinator_host(self):
"""Returns (str or None): The coordinator host, or None if not configured.
"""
return self._coordinator_host
@property
def namespace(self):
"""Returns (str): The namespace for all streams opened by this client.
Empty if not configured.
"""
return self._namespace
def get_stream_path(self, name):
"""Returns (streamname.StreamPath): The stream path.
Args:
name (str): The name of the stream.
Raises:
KeyError: if information needed to construct the path is missing.
ValueError: if the stream path is invalid, or if the stream prefix is
not defined in the client.
"""
if not self._prefix:
raise KeyError('Stream prefix is not configured')
return streamname.StreamPath.make(self._prefix, name)
def get_viewer_url(self, name):
"""Returns (str): The LogDog viewer URL for the named stream.
Args:
name (str): The name of the stream. This can also be a query glob.
Raises:
KeyError: if information needed to construct the URL is missing.
ValueError: if the stream prefix or name do not form a valid stream
path.
"""
if not self._coordinator_host:
raise KeyError('Coordinator host is not configured')
if not self._project:
raise KeyError('Stream project is not configured')
return streamname.get_logdog_viewer_url(self._coordinator_host, self._project,
self.get_stream_path(name))
def _register_new_stream(self, name):
"""Registers a new stream name.
The Butler will internally reject any duplicate stream names. However, there
isn't really feedback when this happens except a closed stream client. This
is a client-side check to provide a more user-friendly experience in the
event that a user attempts to register a duplicate stream name.
Note that this is imperfect, as something else could register stream names
with the same Butler instance and this library has no means of tracking.
This is a best-effort experience, not a reliable check.
Args:
name (str): The name of the stream.
Raises:
ValueError if the stream name has already been registered.
"""
with self._name_lock:
if name in self._names:
raise ValueError("Duplicate stream name [%s]" % (name,))
self._names.add(name)
@classmethod
def _create(cls, value, **kwargs):
"""Returns (StreamClient): A new stream client instance.
Validates the streamserver parameters and creates a new StreamClient
instance that connects to them.
Implementing classes must override this.
"""
raise NotImplementedError()
def _connect_raw(self):
"""Returns (file): A new file-like stream.
Creates a new raw connection to the streamserver. This connection MUST not
have any data written to it past initialization (if needed) when it has been
returned.
The file-like object must implement `write`, `fileno`, `flush`, and `close`.
Implementing classes must override this.
"""
raise NotImplementedError()
def new_connection(self, params):
"""Returns (file): A new configured stream.
The returned object implements (minimally) `write` and `close`.
Creates a new LogDog stream with the specified parameters.
Args:
params (StreamParams): The parameters to use with the new connection.
Raises:
ValueError if the stream name has already been used, or if the parameters
are not valid.
"""
self._register_new_stream(params.name)
params_json = params.to_json()
fobj = self._connect_raw()
fobj.write(BUTLER_MAGIC)
varint.write_uvarint(fobj, len(params_json))
fobj.write(params_json)
return fobj
@contextlib.contextmanager
def text(self, name, **kwargs):
"""Context manager to create, use, and teardown a TEXT stream.
This context manager creates a new butler TEXT stream with the specified
parameters, yields it, and closes it on teardown.
Args:
name (str): the LogDog name of the stream.
kwargs (dict): Log stream parameters. These may be any keyword arguments
accepted by `open_text`.
Returns (file): A file-like object to a Butler UTF-8 text stream supporting
`write`.
"""
fobj = None
try:
fobj = self.open_text(name, **kwargs)
yield fobj
finally:
if fobj is not None:
fobj.close()
def open_text(self, name, content_type=None, tags=None):
"""Returns (file): A file-like object for a single text stream.
This creates a new butler TEXT stream with the specified parameters.
Args:
name (str): the LogDog name of the stream.
content_type (str): The optional content type of the stream. If None, a
default content type will be chosen by the Butler.
tags (dict): An optional key/value dictionary pair of LogDog stream tags.
Returns (file): A file-like object to a Butler text stream. This object can
have UTF-8 text content written to it with its `write` method, and must
be closed when finished using its `close` method.
"""
params = StreamParams.make(
name=posixpath.join(self._namespace, name),
type=StreamParams.TEXT,
content_type=content_type,
tags=tags)
return self._BasicStream(self, params, self.new_connection(params))
@contextlib.contextmanager
def binary(self, name, **kwargs):
"""Context manager to create, use, and teardown a BINARY stream.
This context manager creates a new butler BINARY stream with the specified
parameters, yields it, and closes it on teardown.
Args:
name (str): the LogDog name of the stream.
kwargs (dict): Log stream parameters. These may be any keyword arguments
accepted by `open_binary`.
Returns (file): A file-like object to a Butler binary stream supporting
`write`.
"""
fobj = None
try:
fobj = self.open_binary(name, **kwargs)
yield fobj
finally:
if fobj is not None:
fobj.close()
def open_binary(self, name, content_type=None, tags=None):
"""Returns (file): A file-like object for a single binary stream.
This creates a new butler BINARY stream with the specified parameters.
Args:
name (str): the LogDog name of the stream.
content_type (str): The optional content type of the stream. If None, a
default content type will be chosen by the Butler.
tags (dict): An optional key/value dictionary pair of LogDog stream tags.
Returns (file): A file-like object to a Butler binary stream. This object
can have UTF-8 content written to it with its `write` method, and must
be closed when finished using its `close` method.
"""
params = StreamParams.make(
name=posixpath.join(self._namespace, name),
type=StreamParams.BINARY,
content_type=content_type,
tags=tags)
return self._BasicStream(self, params, self.new_connection(params))
@contextlib.contextmanager
def datagram(self, name, **kwargs):
"""Context manager to create, use, and teardown a DATAGRAM stream.
This context manager creates a new butler DATAAGRAM stream with the
specified parameters, yields it, and closes it on teardown.
Args:
name (str): the LogDog name of the stream.
kwargs (dict): Log stream parameters. These may be any keyword arguments
accepted by `open_datagram`.
Returns (_DatagramStream): A datagram stream object. Datagrams can be
written to it using its `send` method.
"""
fobj = None
try:
fobj = self.open_datagram(name, **kwargs)
yield fobj
finally:
if fobj is not None:
fobj.close()
def open_datagram(self, name, content_type=None, tags=None):
"""Creates a new butler DATAGRAM stream with the specified parameters.
Args:
name (str): the LogDog name of the stream.
content_type (str): The optional content type of the stream. If None, a
default content type will be chosen by the Butler.
tags (dict): An optional key/value dictionary pair of LogDog stream tags.
Returns (_DatagramStream): A datagram stream object. Datagrams can be
written to it using its `send` method. This object must be closed when
finished by using its `close` method.
"""
params = StreamParams.make(
name=posixpath.join(self._namespace, name),
type=StreamParams.DATAGRAM,
content_type=content_type,
tags=tags)
return self._DatagramStream(self, params, self.new_connection(params))
class _NamedPipeStreamClient(StreamClient):
"""A StreamClient implementation that connects to a Windows named pipe.
"""
def __init__(self, name, **kwargs):
r"""Initializes a new Windows named pipe stream client.
Args:
name (str): The name of the Windows named pipe to use (e.g., "\\.\name")
"""
super(_NamedPipeStreamClient, self).__init__(**kwargs)
self._name = '\\\\.\\pipe\\' + name
@classmethod
def _create(cls, value, **kwargs):
return cls(value, **kwargs)
ERROR_PIPE_BUSY = 231
def _connect_raw(self):
# This is a similar procedure to the one in
# https://github.com/microsoft/go-winio/blob/master/pipe.go (tryDialPipe)
while True:
try:
return open(self._name, 'wb+', buffering=0)
except (OSError, IOError):
if GetLastError() != self.ERROR_PIPE_BUSY:
raise
time.sleep(0.001) # 1ms
_default_registry.register_protocol('net.pipe', _NamedPipeStreamClient)
class _UnixDomainSocketStreamClient(StreamClient):
"""A StreamClient implementation that uses a UNIX domain socket.
"""
class SocketFile(object):
"""A write-only file-like object that writes to a UNIX socket."""
def __init__(self, sock):
self._sock = sock
def fileno(self):
return self._sock
def write(self, data):
self._sock.sendall(data)
def flush(self):
pass
def close(self):
self._sock.close()
def __init__(self, path, **kwargs):
"""Initializes a new UNIX domain socket stream client.
Args:
path (str): The path to the named UNIX domain socket.
"""
super(_UnixDomainSocketStreamClient, self).__init__(**kwargs)
self._path = path
@classmethod
def _create(cls, value, **kwargs):
if not os.path.exists(value):
raise ValueError('UNIX domain socket [%s] does not exist.' % (value,))
return cls(value, **kwargs)
def _connect_raw(self):
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(self._path)
return self.SocketFile(sock)
_default_registry.register_protocol('unix', _UnixDomainSocketStreamClient)

198
third_party/logdog/logdog/streamname.py vendored Normal file
View File

@@ -0,0 +1,198 @@
# Copyright 2016 The LUCI Authors. All rights reserved.
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
import collections
import re
import string
# third_party/
from six.moves import urllib
_STREAM_SEP = '/'
_ALNUM_CHARS = string.ascii_letters + string.digits
_VALID_SEG_CHARS = _ALNUM_CHARS + ':_-.'
_SEGMENT_RE_BASE = r'[a-zA-Z0-9][a-zA-Z0-9:_\-.]*'
_SEGMENT_RE = re.compile('^' + _SEGMENT_RE_BASE + '$')
_STREAM_NAME_RE = re.compile('^(' + _SEGMENT_RE_BASE + ')(/' + _SEGMENT_RE_BASE + ')*$')
_MAX_STREAM_NAME_LENGTH = 4096
_MAX_TAG_KEY_LENGTH = 64
_MAX_TAG_VALUE_LENGTH = 4096
def validate_stream_name(v, maxlen=None):
"""Verifies that a given stream name is valid.
Args:
v (str): The stream name string.
Raises:
ValueError if the stream name is invalid.
"""
maxlen = maxlen or _MAX_STREAM_NAME_LENGTH
if len(v) > maxlen:
raise ValueError('Maximum length exceeded (%d > %d)' % (len(v), maxlen))
if _STREAM_NAME_RE.match(v) is None:
raise ValueError('Invalid stream name: %r' % v)
def validate_tag(key, value):
"""Verifies that a given tag key/value is valid.
Args:
k (str): The tag key.
v (str): The tag value.
Raises:
ValueError if the tag is not valid.
"""
validate_stream_name(key, maxlen=_MAX_TAG_KEY_LENGTH)
validate_stream_name(value, maxlen=_MAX_TAG_VALUE_LENGTH)
def normalize_segment(seg, prefix=None):
"""Given a string (str|unicode), mutate it into a valid segment name (str).
This operates by replacing invalid segment name characters with underscores
(_) when encountered.
A special case is when "seg" begins with non-alphanumeric character. In this
case, we will prefix it with the "prefix", if one is supplied. Otherwise,
raises ValueError.
See _VALID_SEG_CHARS for all valid characters for a segment.
Raises:
ValueError: If normalization could not be successfully performed.
"""
if not seg:
if prefix is None:
raise ValueError('Cannot normalize empty segment with no prefix.')
seg = prefix
else:
def replace_if_invalid(ch, first=False):
ret = ch if ch in _VALID_SEG_CHARS else '_'
if first and ch not in _ALNUM_CHARS:
if prefix is None:
raise ValueError('Segment has invalid beginning, and no prefix was '
'provided.')
return prefix + ret
return ret
seg = ''.join(replace_if_invalid(ch, i == 0) for i, ch in enumerate(seg))
if _SEGMENT_RE.match(seg) is None:
raise AssertionError('Normalized segment is still invalid: %r' % seg)
# v could be of type unicode. As a valid stream name contains only ascii
# characters, it is safe to transcode v to ascii encoding (become str type).
if isinstance(seg, unicode):
return seg.encode('ascii')
return seg
def normalize(v, prefix=None):
"""Given a string (str|unicode), mutate it into a valid stream name (str).
This operates by replacing invalid stream name characters with underscores (_)
when encountered.
A special case is when any segment of "v" begins with an non-alphanumeric
character. In this case, we will prefix the segment with the "prefix", if one
is supplied. Otherwise, raises ValueError.
See _STREAM_NAME_RE for a description of a valid stream name.
Raises:
ValueError: If normalization could not be successfully performed.
"""
normalized = _STREAM_SEP.join(
normalize_segment(seg, prefix=prefix) for seg in v.split(_STREAM_SEP))
# Validate the resulting string.
validate_stream_name(normalized)
return normalized
class StreamPath(collections.namedtuple('_StreamPath', ('prefix', 'name'))):
"""StreamPath is a full stream path.
This consists of both a stream prefix and a stream name.
When constructed with parse or make, the stream path must be completely valid.
However, invalid stream paths may be constructed by manually instantiation.
This can be useful for wildcard query values (e.g., "prefix='foo/*/bar/**'").
"""
@classmethod
def make(cls, prefix, name):
"""Returns (StreamPath): The validated StreamPath instance.
Args:
prefix (str): the prefix component
name (str): the name component
Raises:
ValueError: If path is not a full, valid stream path string.
"""
inst = cls(prefix=prefix, name=name)
inst.validate()
return inst
@classmethod
def parse(cls, path):
"""Returns (StreamPath): The parsed StreamPath instance.
Args:
path (str): the full stream path to parse.
Raises:
ValueError: If path is not a full, valid stream path string.
"""
parts = path.split('/+/', 1)
if len(parts) != 2:
raise ValueError('Not a full stream path: [%s]' % (path,))
return cls.make(*parts)
def validate(self):
"""Raises: ValueError if this is not a valid stream name."""
try:
validate_stream_name(self.prefix)
except ValueError as e:
raise ValueError('Invalid prefix component [%s]: %s' % (
self.prefix,
e.message,
))
try:
validate_stream_name(self.name)
except ValueError as e:
raise ValueError('Invalid name component [%s]: %s' % (
self.name,
e.message,
))
def __str__(self):
return '%s/+/%s' % (self.prefix, self.name)
def get_logdog_viewer_url(host, project, *stream_paths):
"""Returns (str): The LogDog viewer URL for the named stream(s).
Args:
host (str): The name of the Coordiantor host.
project (str): The project name.
stream_paths: A set of StreamPath instances for the stream paths to
generate the URL for.
"""
return urllib.parse.urlunparse((
'https', # Scheme
host, # netloc
'v/', # path
'', # params
'&'.join(('s=%s' % (urllib.parse.quote('%s/%s' % (project, path), ''))
for path in stream_paths)), # query
'', # fragment
))

63
third_party/logdog/logdog/varint.py vendored Normal file
View File

@@ -0,0 +1,63 @@
# Copyright 2016 The LUCI Authors. All rights reserved.
# Use of this source code is governed under the Apache License, Version 2.0
# that can be found in the LICENSE file.
import os
import sys
def write_uvarint(w, val):
"""Writes a varint value to the supplied file-like object.
Args:
w (object): A file-like object to write to. Must implement write.
val (number): The value to write. Must be >= 0.
Returns (int): The number of bytes that were written.
Raises:
ValueError if 'val' is < 0.
"""
if val < 0:
raise ValueError('Cannot encode negative value, %d' % (val,))
count = 0
while val > 0 or count == 0:
byte = (val & 0b01111111)
val >>= 7
if val > 0:
byte |= 0b10000000
w.write(chr(byte))
count += 1
return count
def read_uvarint(r):
"""Reads a uvarint from a stream.
This is targeted towards testing, and will not be used in production code.
Args:
r (object): A file-like object to read from. Must implement read.
Returns: (value, count)
value (int): The decoded varint number.
count (int): The number of bytes that were read from 'r'.
Raises:
ValueError if the encoded varint is not terminated.
"""
count = 0
result = 0
while True:
byte = r.read(1)
if len(byte) == 0:
raise ValueError('UVarint was not terminated')
byte = ord(byte)
result |= ((byte & 0b01111111) << (7 * count))
count += 1
if byte & 0b10000000 == 0:
break
return result, count