Skip to content

Commit

Permalink
Generate files to use Emscripten bindings from outside Emscripten
Browse files Browse the repository at this point in the history
We are stabilizing webgpu.h, and that process requires many (mostly
small) breaking changes to the header. Right now, some projects are
relying on Emscripten's and Dawn's implementations of webgpu.h to be in
sync, but this creates challenges because we need to make many separate
breaking changes.

This change forks Emscripten's WebGPU bindings into Dawn (in
//third_party) in such a way that they don't have to be checked into the
Emscripten repository. This way we can keep Dawn's bindings and Dawn's
fork of the Emscripten bindings in sync (approximately) in the Dawn
repository.

Projects depending on Emscripten and Dawn to be in sync should use this
fork instead, until we've finished the webgpu.h changes and contributed
the bindings back up to Emscripten in one big breaking release that
should have minimal to no breakage afterward.

For ease of diffing this commit overwrites the verbatim copies of
library_{,html5_}webgpu.js added in the previous commit, with a few
updates to bring it in sync with Dawn (though it may not be fully in
sync - it has only been minimally tested so far).

Example of using the resulting generated files from outside GN:
https://github.com/kainino0x/webgpu-cross-platform-demo/compare/dawnwasm

The generated files can only be generated using the GN build; CMake
rules haven't been implemented, and the existing Emscripten build rules
for CMake have been removed.

Note, since these bindings were designed to live inside Emscripten, they
rely on some Emscripten internals.

Symbols in rough order from less stable to more stable:

- `$stackSave` - changed recently
- `$stackRestore` - changed recently
- `$callUserCallback`
- `$stringToUTF8OnStack`
- `$stringToNewUTF8`
- `$stringToUTF8`
- `$lengthBytesUTF8`
- `$readI53FromI64`
- `$findCanvasEventTarget`
- `$warnOnce`

Generator internals, these seem stable:

- `mergeInto`
- `C_STRUCTS`
- `LibraryManager.library`

The build rules also depend on:

- the location of `gen_struct_info.py` (which recently changed from
  `tools/` to `tools/maint/`)

Bug: 346806934
Change-Id: I6ca10ea37bc7307d4d9571dcdc696e7d025663d3
Reviewed-on: https://dawn-review.googlesource.com/c/dawn/+/193722
Reviewed-by: Loko Kung <lokokung@google.com>
Auto-Submit: Kai Ninomiya <kainino@chromium.org>
Reviewed-by: dan sinclair <dsinclair@chromium.org>
Commit-Queue: dan sinclair <dsinclair@chromium.org>
  • Loading branch information
kainino0x authored and Dawn LUCI CQ committed Jul 3, 2024
1 parent a10fbea commit 1178400
Show file tree
Hide file tree
Showing 23 changed files with 851 additions and 844 deletions.
12 changes: 12 additions & 0 deletions BUILD.gn
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,24 @@ group("android") {
deps = [ "tools/android" ]
}

group("emdawnwebgpu") {
# Without a copy of Emscripten, we can only generate a few files.
deps = [
"src/emdawnwebgpu:emdawnwebgpu_headers_gen",
"src/emdawnwebgpu:emdawnwebgpu_js_gen",
]
if (dawn_emscripten_dir != "") {
deps += [ "src/emdawnwebgpu:emdawnwebgpu" ]
}
}

group("all") {
testonly = true
deps = [
":android",
":benchmarks",
":cmds",
":emdawnwebgpu",
":fuzzers",
":libs",
":tests",
Expand Down
7 changes: 7 additions & 0 deletions build_overrides/dawn.gni
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,13 @@ dawn_vulkan_validation_layers_dir = "//third_party/vulkan-validation-layers/src"
dawn_dxc_dir = "//third_party/dxc"
dawn_dxheaders_dir = "//third_party/dxheaders"

declare_args() {
# Enables generating the Emscripten WebGPU bindings.
# Note this must point at an emscripten source checkout; the emsdk copy
# installed at emsdk/upstream/emscripten does not have gen_struct_info.py.
dawn_emscripten_dir = ""
}

# PartitionAlloc is an optional dependency.
# It does not fully support the MSVC compiler at the moment.
if (is_clang || !is_win) {
Expand Down
2 changes: 1 addition & 1 deletion docs/dawn/codegen.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ At this time it is used to generate:
- the Dawn, Emscripten, and upstream webgpu-native `webgpu.h` C header
- the Dawn and Emscripten `webgpu_cpp.h` C++ wrapper over the C header
- libraries that implements `webgpu.h` by calling in a static or `thread_local` proc table
- other parts of the [Emscripten](https://emscripten.org/) WebGPU implementation
- the [Emscripten](https://emscripten.org/) WebGPU binding implementation
- a GMock version of the API with its proc table for testing
- validation helper functions for dawn_native
- the definition of dawn_native's proc table
Expand Down
1 change: 1 addition & 0 deletions generator/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ function(DawnGenerator)
"SCRIPT;OUTPUT_HEADERS;OUTPUT_SOURCES;PRINT_NAME"
"EXTRA_PARAMETERS"
)
message(STATUS "Dawn: Configuring DawnGenerator for ${arg_PRINT_NAME}.")

if (arg_UNPARSED_ARGUMENTS)
message(FATAL_ERROR
Expand Down
74 changes: 57 additions & 17 deletions generator/dawn_json_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -928,6 +928,37 @@ def as_jsEnumValue(value):
return "'" + value.name.js_enum_case() + "'"


def has_wasmType(return_type, args):
return all(map(lambda x: len(as_wasmType(x)) == 1, [return_type] + args))


# Returns a single character wasm type (v/p/i/j/f/d) if valid, a "(longer string)" if not
def as_wasmType(x):
if isinstance(x, RecordMember):
if x.annotation == 'value':
x = x.type
elif '*' in x.annotation:
return 'p'
else:
return f'({x})'

if isinstance(x, Type):
if x.category == 'enum':
return 'i'
elif x.category == 'bitmask':
# TODO(crbug.com/347732150): Change to 'j' when bitmasks are 64-bit
return 'i'
elif x.category in ['object', 'function pointer']:
return 'p'
elif x.category == 'native':
return x.json_data.get('wasm type', f'({x.name.name})')
elif x.category in ['structure', 'callback info']:
return f'({x.name.name})' # Invalid
else:
assert False, 'Type -> ' + x.category
assert False, x


def convert_cType_to_cppType(typ, annotation, arg, indent=0):
if typ.category == 'native':
return arg
Expand Down Expand Up @@ -1122,6 +1153,8 @@ def as_cProc(type_name, method_name):
'as_cType': lambda name: as_cType(c_prefix, name),
'as_cppType': as_cppType,
'as_jsEnumValue': as_jsEnumValue,
'has_wasmType': has_wasmType,
'as_wasmType': as_wasmType,
'convert_cType_to_cppType': convert_cType_to_cppType,
'as_varName': as_varName,
'decorate': decorate,
Expand Down Expand Up @@ -1270,38 +1303,45 @@ def get_file_renders(self, args):
FileRender('api.h', 'webgpu-headers/' + api + '.h',
[RENDER_PARAMS_BASE, params_upstream]))

if 'emscripten_bits' in targets:
if 'emdawnwebgpu_headers' in targets:
assert api == 'webgpu'
params_emscripten = parse_json(
loaded_json, enabled_tags=['compat', 'emscripten'])
# system/include/webgpu
renders.append(
FileRender('api.h',
'emscripten-bits/system/include/webgpu/webgpu.h',
FileRender('api.h', 'src/emdawnwebgpu/include/webgpu/webgpu.h',
[RENDER_PARAMS_BASE, params_emscripten]))
renders.append(
FileRender(
'api_cpp.h',
'emscripten-bits/system/include/webgpu/webgpu_cpp.h', [
RENDER_PARAMS_BASE, params_emscripten, {
'c_header': api + '/' + api + '.h',
'c_namespace': None,
}
]))
FileRender('api_cpp.h',
'src/emdawnwebgpu/include/webgpu/webgpu_cpp.h', [
RENDER_PARAMS_BASE, params_emscripten, {
'c_header': api + '/' + api + '.h',
'c_namespace': None,
}
]))
renders.append(
FileRender(
'api_cpp_chained_struct.h',
'emscripten-bits/system/include/webgpu/webgpu_cpp_chained_struct.h',
'src/emdawnwebgpu/include/webgpu/webgpu_cpp_chained_struct.h',
[RENDER_PARAMS_BASE, params_emscripten]))
# Snippets to paste into existing Emscripten files

if 'emdawnwebgpu_js' in targets:
assert api == 'webgpu'
params_emscripten = parse_json(
loaded_json, enabled_tags=['compat', 'emscripten'])
renders.append(
FileRender('api_struct_info.json',
'emscripten-bits/webgpu_struct_info.json',
FileRender('emdawnwebgpu/webgpu_struct_info.json',
'src/emdawnwebgpu/webgpu_struct_info.json',
[RENDER_PARAMS_BASE, params_emscripten]))
renders.append(
FileRender('library_api_enum_tables.js',
'emscripten-bits/library_webgpu_enum_tables.js',
FileRender('emdawnwebgpu/library_webgpu_enum_tables.js',
'src/emdawnwebgpu/library_webgpu_enum_tables.js',
[RENDER_PARAMS_BASE, params_emscripten]))
renders.append(
FileRender(
'emdawnwebgpu/library_webgpu_generated_sig_info.js',
'src/emdawnwebgpu/library_webgpu_generated_sig_info.js',
[RENDER_PARAMS_BASE, params_emscripten]))

if 'mock_api' in targets:
mock_params = [
Expand Down
2 changes: 1 addition & 1 deletion generator/templates/api_cpp.h
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
{% set CAPI = metadata.c_prefix %}
{% if 'dawn' in enabled_tags %}
#ifdef __EMSCRIPTEN__
#error "Do not include this header. Emscripten already provides headers needed for {{metadata.api}}."
#error "This header is for native Dawn. Use Dawn's or Emscripten's Emscripten bindings instead."
#endif
{% endif %}
{% set PREFIX = "" if not c_namespace else c_namespace.SNAKE_CASE() + "_" %}
Expand Down
91 changes: 91 additions & 0 deletions generator/templates/emdawnwebgpu/library_webgpu_enum_tables.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
//* Copyright 2024 The Dawn & Tint Authors
//*
//* Redistribution and use in source and binary forms, with or without
//* modification, are permitted provided that the following conditions are met:
//*
//* 1. Redistributions of source code must retain the above copyright notice, this
//* list of conditions and the following disclaimer.
//*
//* 2. Redistributions in binary form must reproduce the above copyright notice,
//* this list of conditions and the following disclaimer in the documentation
//* and/or other materials provided with the distribution.
//*
//* 3. Neither the name of the copyright holder nor the names of its
//* contributors may be used to endorse or promote products derived from
//* this software without specific prior written permission.
//*
//* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
//* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
//* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
//* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
//* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
//* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
//* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
//* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
//* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
//* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//*
//*
//* This generator is used to produce the number-to-string mappings for
//* Emscripten's library_webgpu.js.
//* https://github.com/emscripten-core/emscripten/blob/main/src/library_webgpu.js
//*
{{'{{{'}}
globalThis.__HAVE_EMDAWNWEBGPU_ENUM_TABLES = true;

// Constant values used at code-generation time in an Emscripten build.
// These will not appear in the final build result, so we can just dump
// every enum here without affecting binary size.
globalThis.WEBGPU_ENUM_CONSTANT_TABLES = {
{% for type in by_category["enum"] + by_category["bitmask"] %}
{{type.name.CamelCase()}}: {
{% for value in type.values %}
'{{value.name.CamelCase()}}': {{value.value}},
{% endfor %}
},
{% endfor %}
};

// Maps from enum string back to enum number, for callbacks.
// These appear in the final build result so should be kept minimal.
globalThis.WEBGPU_STRING_TO_INT_TABLES = `
{% for type in by_category["enum"] if type.json_data.get("emscripten_string_to_int", False) %}
Int_{{type.name.CamelCase()}}: {
{% for value in type.values if value.json_data.get("emscripten_string_to_int", True) %}
{% if type.name.name == 'device lost reason' and value.name.name == 'unknown' %}
'undefined': {{value.value}}, // For older browsers
{% endif %}
{{as_jsEnumValue(value)}}: {{value.value}},
{% endfor %}
},
{% endfor %}
Int_PreferredFormat: {
{% for value in types['texture format'].values if value.name.name in ['RGBA8 unorm', 'BGRA8 unorm'] %}
{{as_jsEnumValue(value)}}: {{value.value}},
{% endfor %}
},
`;

// Maps from enum number to enum string.
// These appear in the final build result so should be kept minimal.
globalThis.WEBGPU_INT_TO_STRING_TABLES = `
{% for type in by_category["enum"] if not type.json_data.get("emscripten_no_enum_table") %}
{{type.name.CamelCase()}}: {% if type.contiguousFromZero -%}
[
{% for value in type.values %}
{{as_jsEnumValue(value)}},
{% endfor %}
]
{%- else -%}
{
{% for value in type.values %}
{{value.value}}: {{as_jsEnumValue(value)}},
{% endfor %}
}
{%- endif -%}
,
{% endfor %}
`;

null;
{{'}}}'}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
//* Copyright 2024 The Dawn & Tint Authors
//*
//* Redistribution and use in source and binary forms, with or without
//* modification, are permitted provided that the following conditions are met:
//*
//* 1. Redistributions of source code must retain the above copyright notice, this
//* list of conditions and the following disclaimer.
//*
//* 2. Redistributions in binary form must reproduce the above copyright notice,
//* this list of conditions and the following disclaimer in the documentation
//* and/or other materials provided with the distribution.
//*
//* 3. Neither the name of the copyright holder nor the names of its
//* contributors may be used to endorse or promote products derived from
//* this software without specific prior written permission.
//*
//* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
//* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
//* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
//* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
//* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
//* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
//* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
//* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
//* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
//* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//*
//*
//* This generator is used to produce the Wasm signatures for any C functions implemented in JS.
//* It is equivalent to the emscripten_webgpu_* and wgpu* signatures in:
//* https://github.com/emscripten-core/emscripten/blob/main/src/library_sigs.js
//*
{{'{{{'}}
globalThis.__HAVE_EMDAWNWEBGPU_SIG_INFO = true;
null;
{{'}}}'}}
{%- macro render_function_or_method_sig(object_or_none, function) -%}
{%- if not has_wasmType(function.return_type, function.arguments) -%}
//
{%- endif -%}
{{as_cMethod(object_or_none.name if object_or_none else None, function.name)}}__sig: '
{{- as_wasmType(function.return_type) -}}
{%- if object_or_none -%}
{{- as_wasmType(object_or_none) -}}
{%- endif -%}
{%- for arg in function.arguments -%}
{{- as_wasmType(arg) -}}
{%- endfor -%}
',
{%- endmacro -%}
const webgpuSigs = {
// emscripten_webgpu_*
emscripten_webgpu_get_device__sig: 'p',
emscripten_webgpu_release_js_handle__sig: 'vi',
{% for type in by_category["object"] %}
emscripten_webgpu_export_{{type.name.snake_case()}}__sig: 'ip',
emscripten_webgpu_import_{{type.name.snake_case()}}__sig: 'pi',
{% endfor %}

// wgpu*
{% for function in by_category["function"] %}
{{render_function_or_method_sig(None, function)}}
{% endfor %}
{% for type in by_category["object"] %}
{% for method in c_methods(type) %}
{{render_function_or_method_sig(type, method)}}
{% endfor %}
{% endfor %}
};

// Delete all of the WebGPU sig info from Emscripten's builtins first.
for (const k of Object.keys(LibraryManager.library)) {
if (k.endsWith('__sig') && (k.startsWith('emscripten_webgpu_') || k.startsWith('wgpu'))) {
delete LibraryManager.library[k];
}
}
mergeInto(LibraryManager.library, webgpuSigs, {allowMissing: true});
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,13 @@
//* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//*
//*
//* This generator is used to produce part of Emscripten's struct_info.json,
//* which is a list of struct fields that it uses to generate field offset
//* information for its own code generators.
//* https://github.com/emscripten-core/emscripten/blob/main/src/struct_info.json
//* This generator is used to produce a struct info file, in the format of
//* Emscripten's struct_info.json file, which is simply a list of struct fields
//* that it uses to generate field offset information for JS code generation.
//* <https://github.com/emscripten-core/emscripten/blob/main/src/struct_info.json>
//* It can be fed directly into gen_struct_info.py, or copy-pasted into the
//* struct_info.json as part of an update to Emscripten's built-in bindings.
//* <https://github.com/emscripten-core/emscripten/blob/main/system/include/webgpu/README.md>
//*
[
// ===========================================
Expand Down
Loading

0 comments on commit 1178400

Please sign in to comment.