Protobuf use static uid (#344)

* adds option to make fields optional in protobuf generator
* adds Documentation for proto generator
* makes vss2proto fail when running with staticUID flag but found does not have staticUID attribute

Signed-off-by: Sven Erik Jeroschewski <svenerik.jeroschewski@bosch.com>

README.md

@@ -33,7 +33,7 @@ Examples on tool usage can be found in the [VSS Makefile](https://github.com/COV
 [vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported   | Check [vspec2x documentation](docs/vspec2x.md)                                                                        |
 [vspec2c.py](obsolete/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Obsolete (2022-11-01) | [Documentation](obsolete/vspec2c/README.md)                                                                           |
 [vspec2ocf.py](obsolete/ocf/vspec2ocf.py) | Parses and expands a VSS and generates a OCF specification | Obsolete (2022-11-01) | -                                                                                                                     |
-[vspec2protobuf.py](vspec2protobuf.py) | Parses and expands a VSS and generates a Protobuf specification | Contrib               | -                                                                                                                     |
+[vspec2protobuf.py](vspec2protobuf.py) | Parses and expands a VSS tree and generates a Protobuf message definition | Contrib               | [Documentation](docs/vspec2proto.md)                                                                                                                     |
 [vspec2ttl.py](contrib/vspec2ttl/vspec2ttl.py) | Parses and expands a VSS and generates a TTL specification | Contrib               | -                                                                                                                     |
 [vspec2graphql.py](vspec2graphql.py) | Parses and expands a VSS and generates a GraphQL specification | Community Supported   | [Documentation](docs/VSS2GRAPHQL.md)                                                                                  |
 [vspec2id.py](vspec2id.py) | Generates and validates static UIDs for a VSS | WIP                   | [vspec2id Documentation](./docs/vspec2id.md)                                                                          |

docs/vspec2proto.md

@@ -0,0 +1,98 @@
+# vspec2proto - vspec Protocol Buffer generator
+
+The vspec2proto.py script generates [Protocol Buffer message definitions](https://protobuf.dev) for all nodes in the tree. You can use these proto definitions to serialize the data of a VSS tree, e.g., as part of a gRPC API.
+
+## Example
+
+```bash
+./vspec2protobuf.py outputIds_v2.vspec indentity.proto  -q ../vehicle_signal_specification/spec/quantities.yaml -u ../vehicle_signal_specification/spec/units.yaml -e staticUID --static-uid --add-optional
+```
+
+This example assumes that you checked out the COVESA VSS repository next to the vss-tools repository.
+
+## Exporter specific arguments
+
+In addition, to the general arguments of each exporter, the vspec2proto exporter supports the following arguments:
+
+```bash
+--static-uid          Expect staticUID attribute in the vspec input and use it as field number.
+--add-optional        Set each field to optional
+
+```
+
+## Field Numbers and Backwards Compatibility
+
+Part of the serialization with protocol buffers is the replacement of the field identifier with a field number. It is possible to set the used field number in the protobuf file. For instance, the field 'Speed' in the message 'Vehicle' could get the identifier 5:
+
+```bash
+message Vehicle {
+  ...
+  float Speed = 5
+  ...
+}
+```
+
+An encoded speed value of 20, would look something like this:
+
+```bash
+5: 20
+```
+
+Instead of sending the complete signal path like 'Vehicle.Speed', the serialization only contains the field number to reduce the amount of data to send. However, using field numbers as identifiers requires the same mapping between field numbers and fields on the encoding and decoding side.
+
+The vspec2proto generator supports two approaches for setting the field numbers with different advantages and drawbacks.
+
+### Incremental Field Numbers
+
+By default, the generator numbers the fields from a single branch, which it represents as a message in protobuf, starting with 1. As an example, for the  `Vehicle` node we would get a message like this:
+
+```bash
+message Vehicle {
+  VehicleVersionVSS VersionVSS = 1;
+  VehicleVehicleIdentification VehicleIdentification = 2;
+  string LowVoltageSystemState = 3;
+  VehicleLowVoltageBattery LowVoltageBattery = 4;
+  float Speed = 5;
+  float TraveledDistance = 6;
+  ...
+}
+```
+
+We could create a newer version of the vspec-file with an additional signal like `MyNewSignal`. This addition may happen locally through a custom overlay or upstream in a new minor version of VSS. Either way, we would expect backward compatibility of the model with the same major version of the VSS model, even if the decoding side does not know about the updates. However, the resulting protobuf could be something like:
+
+```bash
+VehicleVersionVSS VersionVSS = 1;
+  ...
+  VehicleLowVoltageBattery LowVoltageBattery = 4;
+  float Speed = 5;
+  float MyNewSignal = 6;
+  float TraveledDistance = 7;
+  ...
+```
+
+Since the proto generator did not know the field numbers which it assigned to the fields in previous runs, it sorted `MyNewSignal` together with the old signals. As a result, the field numbers for the fields following the new signal have changed.
+
+Thus, when using this protobuf file, the generator may introduce breaking changes even if the underlying vspec does not contain breaking changes. Hence, we recommend using the exact same proto file for en- and decoding, which leads to a stronger coupling between both sides.
+
+### static-uid as Field Number
+
+One solution to overcome breaking changes in the serialization caused by non-breaking changes in the VSS model is to define the numeric identifiers within the VSS model. This way, the proto generator is able to reuse these identifiers as field numbers and does not have to come up with its own numbers. However, such identifiers are not part of the upstream VSS model yet.
+
+But there is the option to generate static uids with the [`vspec2id.py`](./vspec2id.md).
+By adding the flag `--static-uid` you can instruct the proto generator to expect static uids in the input file and use them as field numbers.
+
+This comes with the following drawbacks:
+
+* **no bi-directional mapping between static-uid and protobuf field number**: The static uid is 32-bit long, and the length of a field number in proto files cannot exceed 29-bit.
+Because of that, the generator cuts off the three least significant bits, which removes the option to map a field number back to the static-uid.
+* **potential collisions**: The id generator uses a hashing algorithm to compute the 32-bit long id. This hashing may lead to id collisions. The proto generator increases the collision probability since it cuts the id to 29-bit to be compatible with the maximum size of a protobuf field number.
+Because of that, the proto generator re-checks for collisions. In case of a detected collision, the id and the proto generator stop and request the user to change the input model until no collisions occur.
+In addition, protobuf reserves the field numbers 19.000 to 20.000. The proto generator thus treats field numbers in this range like a collision.
+* **overhead on the wire**: The [protobuf documentation](https://protobuf.dev/programming-guides/proto3/#assigning) recommends using field numbers that are as low as possible to reduce the number of required bytes for encoding the number.
+For the incrementally assigned field numbers, we only need a few bits per number, while the id-based field number, in most cases, requires the full 4 bytes for 29 bits.
+
+## Mark field as optional
+
+In proto3, one can mark a field as `optional` to change the behavior of how to deal with values that are not present during the encoding. By default, the fields are not optional, and you can use the flag `--add-optional` to make all fields optional.
+
+See the [Protocol Buffers Language Guide](https://protobuf.dev/programming-guides/proto3/#field-labels) for the implications of using `optional`.

vspec/vssexporters/vss2protobuf.py

@@ -53,7 +53,7 @@ def setup_file(self, fp: Path, package_name: str):
             proto_file.write(f"package {package_name};\n\n")
             self.out_files.add(fp)
 
-    def traverse_data_type_tree(self, tree: VSSNode):
+    def traverse_data_type_tree(self, tree: VSSNode, static_uid, add_optional):
         """
         All structs in a branch are written to a single .proto file.
         The file's base name is same as the branch's name
@@ -100,12 +100,12 @@ def traverse_data_type_tree(self, tree: VSSNode):
                 proto_file.write("\n")
 
                 proto_file.write(f"message {type_qn[-1]} {{" + "\n")
-                print_message_body(tree_node.children, proto_file)
+                print_message_body(tree_node.children, proto_file, static_uid, add_optional)
                 proto_file.write("}\n\n")
                 logging.info(f"Wrote {type_qn[-1]} to {fp}")
 
 
-def traverse_signal_tree(tree: VSSNode, proto_file):
+def traverse_signal_tree(tree: VSSNode, proto_file, static_uid, add_optional):
     proto_file.write("syntax = \"proto3\";\n\n")
     tree_node: VSSNode
 
@@ -127,18 +127,45 @@ def traverse_signal_tree(tree: VSSNode, proto_file):
     # write proto messages to file
     for tree_node in filter(lambda n: n.is_branch(), PreOrderIter(tree)):
         proto_file.write(f"message {tree_node.qualified_name('')} {{" + "\n")
-        print_message_body(tree_node.children, proto_file)
+        print_message_body(tree_node.children, proto_file, static_uid, add_optional)
         proto_file.write("}\n\n")
 
 
-def print_message_body(nodes, proto_file):
+def print_message_body(nodes, proto_file, static_uid, add_optional):
+    usedKeys = {}
     for i, node in enumerate(nodes, 1):
-        data_type = node.qualified_name("")
         if not (node.is_branch() or node.is_struct()):
             dt_val = node.get_datatype()
             data_type = mapped.get(dt_val.strip("[]"), dt_val.strip("[]"))
-            data_type = ("repeated " if dt_val.endswith("[]") else "") + data_type
-        proto_file.write(f"  {data_type} {node.name} = {i};" + "\n")
+            if dt_val.endswith("[]"):
+                data_type = "repeated " + data_type
+            elif add_optional:
+                data_type = "optional " + data_type
+        else:
+            data_type = node.qualified_name("")
+            if add_optional:
+                data_type = "optional " + data_type
+        if static_uid:
+            if 'staticUID' not in node.extended_attributes:
+                logging.fatal((f"Aborting because {node.qualified_name('.')} does not have the staticUID attribute. "
+                               f"When using the option --static-uid each node must have the attribute staticUID."))
+                sys.exit(-1)
+            fieldNumber = int(int(node.extended_attributes['staticUID'], 0) / 8)
+            if (fieldNumber < 20000 and fieldNumber >= 19000):
+                logging.fatal('''Aborting because field number {fieldNumber} for signal {node.name} is in
+                reservered range between 19000 and 20000. Consider changing the signal to alter the staticUID.''')
+                sys.exit(-1)
+            if fieldNumber in usedKeys:
+                logging.fatal((f"Aborting, due to collision for fieldNumber {fieldNumber}. "
+                               f"It is used by {node.qualified_name('.')} and {usedKeys[fieldNumber]}. "
+                               "Consider changing the signals to alter the staticUID."))
+                proto_file.truncate(0)
+                sys.exit(-1)
+            else:
+                usedKeys[fieldNumber] = node.qualified_name('.')
+        else:
+            fieldNumber = i
+        proto_file.write(f"  {data_type} {node.name} = {fieldNumber};" + "\n")
 
 
 class Vss2Protobuf(Vss2X):
@@ -148,11 +175,10 @@ def __init__(self, vspec2vss_config: Vspec2VssConfig):
         vspec2vss_config.uuid_supported = False
 
     def add_arguments(self, parser: argparse.ArgumentParser) -> None:
-        parser.add_argument('--json-all-extended-attributes', action='store_true',
-                            help="Generate all extended attributes found in the model "
-                                 "(default is generating only those given by the -e/--extended-attributes parameter).")
-        parser.add_argument('--json-pretty', action='store_true',
-                            help=" Pretty print JSON output.")
+        parser.add_argument('--static-uid', action='store_true',
+                            help=" Expect staticUID attribute in the vspec input and use it as field number.")
+        parser.add_argument('--add-optional', action='store_true',
+                            help=" Set each field to optional.")
 
     def generate(self, config: argparse.Namespace, signal_root: VSSNode, vspec2vss_config: Vspec2VssConfig,
                  data_type_root: Optional[VSSNode] = None) -> None:
@@ -165,7 +191,7 @@ def generate(self, config: argparse.Namespace, signal_root: VSSNode, vspec2vss_c
                 exporter_path = Path(Path.cwd())
             logging.debug(f"Will use {exporter_path} for type exports")
             exporter = ProtoExporter(exporter_path)
-            exporter.traverse_data_type_tree(data_type_root)
+            exporter.traverse_data_type_tree(data_type_root, config.static_uid, config.add_optional)
 
         with open(config.output_file, 'w') as f:
-            traverse_signal_tree(signal_root, f)
+            traverse_signal_tree(signal_root, f, config.static_uid, config.add_optional)