feat(new transform): Introduce WASM Plugins #2006

Hoverbear · 2020-03-09T15:03:47Z

Closes #2280, related to #2341

Platform support:

Linux (x86_64)
Linux (ARMv7) (Likely never)
Linux (Aarch64) (Probably coming, ARM contributing)
Mac (x86_64)
- Multiplatform signals bytecodealliance/lucet#437
Windows (x86_64)
FreeBSD (x86_64)
- Support FreeBSD bytecodealliance/lucet#419
- Multiplatform signals bytecodealliance/lucet#437

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear · 2020-03-31T00:13:08Z

@lukesteensen @LucioFranco Could you take an initial look at this (the RFC only?)

I suggest we merge this with just transforms specced and amend it with sinks and sources later. I'll work on getting the POC woven in and the RFC less draft-style.

Signed-off-by: Ana Hobden <operator@hoverbear.org>

LucioFranco · 2020-03-31T16:21:05Z

Cargo.toml

@@ -177,11 +183,11 @@ dirs = "2.0.2"

 [features]
 # Default features for *-unknown-linux-gnu and *-apple-darwin
-default = ["sources", "transforms", "sinks", "vendored", "unix", "leveldb-plain", "rdkafka-plain"]
+default = ["sources", "transforms", "sinks", "vendored", "unix", "leveldb-plain", "rdkafka-plain", "engine"]


prob a better name than "engine"?

edit: I guess you do call it the vector engine but from reading this I wouldn't assume its the wasm engine.

Definitely open to new terms

LucioFranco

As sinks? For this we should consider how we handle batching, partitioning, etc, more.

I think we will want to expose premade host fn that basically just call out to and configure our current implementations.

What about the idea of codecs being separate?

I'm not totally following this question, could you explain more?

We should consider if we are satisfied with the idea of hostcalls being used for get and other API. We could also let
the host call the Guest allocate function and then pass it a C String pointer to let it work on. This, however, requires
serializing and deserializing the entire Event each time, which is a huge performance bottleneck.

Is this something where we could expose both? Would be nice to let the user choose the trade offs.

Do we one day want to support a cross platform cargo install vector option for installing a Vector binary?

To me this isn't that important since we should be having users go through their system package manager or download the binary from our hosted repo.

Should a user import vector::... to use the Vector guest API in their Rust module?

I would imagine we want to expose a separate smaller crate for rust users that want to compile to wasm? We probably also want to support rust as the best language to write in?

How can we let users see what's happening in WASM modules? Can we use tracing somehow? Lucet supports tracing, perhaps
we could hook in somehow?

Probably via the same method we have users interact with events via some host fn.

Overall this looks fantastic! Super excited to get WASM support! Great work :)

rfcs/2020-03-05-wasm-engine.md

LucioFranco · 2020-03-31T17:04:03Z

rfcs/2020-03-05-wasm-engine.md

+We noted that the existing lua runtime was able to accomplish these tasks quite elegantly, however it was an order of
+magnitude slower than a native transform.
+
+> TODO: Proof


proof is just "lua" :)

I'd like to do some real testing. :)

rfcs/2020-03-05-wasm-engine.md

Hoverbear · 2020-06-09T20:43:18Z

Ok lets leave it off by default and I'll remove the WATs and have them work through /target

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear · 2020-06-10T19:26:28Z

scripts/generate/templates/_partials/_component_header.md.erb

@@ -7,7 +7,7 @@ description: <%= "The Vector `#{component.name}` #{component.type} #{component_s
 event_types: <%= component.event_types.to_json %>
 function_category: <%= component.function_category.to_json %>
 issues_url: <%= metadata.links.fetch("urls.#{component.id}_issues") %>
-<%- if !component.transform? -%>
+<%- if !component.transform? || component.respond_to?(:operating_systems) -%>


@binarylogic I tried just removing this line but that broke, is this the correct way to introduce this value (operating_systems) to the transforms page?

src/event/mod.rs

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear · 2020-06-10T21:09:09Z

I asked @lukesteensen to take a final peak at this before merge to make sure I didn't miss anything. :)

lukesteensen

Looks good! Excited to start experimenting with this 🎉

lukesteensen · 2020-06-10T22:29:15Z

src/wasm/fingerprint.rs

+        let age = modified.duration_since(std::time::UNIX_EPOCH)?;
+        Ok(Self(age.as_secs()))


nit: Using the time itself instead of duration_since would be a more unique fingerprint. Not that I'd really expect a collision either way.

Heck, you're right! I wanted to improve the fingerprinter in general later, but it seems sufficient right now.

* Add wip Signed-off-by: Ana Hobden <operator@hoverbear.org> * wip Signed-off-by: Ana Hobden <operator@hoverbear.org> * wip Signed-off-by: Ana Hobden <operator@hoverbear.org> * WIP Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix bindings Signed-off-by: Ana Hobden <operator@hoverbear.org> * Integrate Make task Signed-off-by: Ana Hobden <operator@hoverbear.org> * Remove old cruft Signed-off-by: Ana Hobden <operator@hoverbear.org> * More refinements to rfc Signed-off-by: Ana Hobden <operator@hoverbear.org> * Get function passing working Signed-off-by: Ana Hobden <operator@hoverbear.org> * Tracing and proper FFI strings Signed-off-by: Ana Hobden <operator@hoverbear.org> * Wip Signed-off-by: Ana Hobden <operator@hoverbear.org> * 🚀 Signed-off-by: Ana Hobden <operator@hoverbear.org> * More RFC notes Signed-off-by: Ana Hobden <operator@hoverbear.org> * Format wasm rfc Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add API notes Signed-off-by: Ana Hobden <operator@hoverbear.org> * Write more of the RFC Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add plan of attack and platform support Signed-off-by: Ana Hobden <operator@hoverbear.org> * Minor fixes. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Command line test passes Signed-off-by: Ana Hobden <operator@hoverbear.org> * Extract foreign module APIs. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Do some documentation. Signed-off-by: Ana Hobden <operator@hoverbear.org> * More RFC writing Signed-off-by: Ana Hobden <operator@hoverbear.org> * Cleanup and RFC updating. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Checkpoint Signed-off-by: Ana Hobden <operator@hoverbear.org> * Passing a Raw registration up works Signed-off-by: Ana Hobden <operator@hoverbear.org> * Update RFC to reflect new simpler APIs. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Various cleaning of structure. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Some refining on docs and roles Signed-off-by: Ana Hobden <operator@hoverbear.org> * Some new events Signed-off-by: Ana Hobden <operator@hoverbear.org> * Remove RFC present in other PR Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix release builds and add new metric Signed-off-by: Ana Hobden <operator@hoverbear.org> * Rename somethings Signed-off-by: Ana Hobden <operator@hoverbear.org> * Caching and fingerprinting works now. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add benchmark for wasm protobuf Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add noop bench Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add wasm CI Signed-off-by: Ana Hobden <operator@hoverbear.org> * Update lockfile Signed-off-by: Ana Hobden <operator@hoverbear.org> * Clean up make jobs, benching Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup make jobs Signed-off-by: Ana Hobden <operator@hoverbear.org> * Use WATS Signed-off-by: Ana Hobden <operator@hoverbear.org> * Figure out less hazardous memory management (demo) Signed-off-by: Ana Hobden <operator@hoverbear.org> * Responsible memory management Signed-off-by: Ana Hobden <operator@hoverbear.org> * process takes pointer/len, emit works Signed-off-by: Ana Hobden <operator@hoverbear.org> * Better benchmarks Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add docs * Fixup nits. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add website files Signed-off-by: Ana Hobden <operator@hoverbear.org> * Rework registration. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Some refining of the protobuf transform and tests. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup benches. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Make emit more flexible. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add tests for other mods Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add transform changes Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add error handling Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add panic handling Signed-off-by: Ana Hobden <operator@hoverbear.org> * No more stale wats Signed-off-by: Ana Hobden <operator@hoverbear.org> * wasm interop exposed and more protobuf guide Signed-off-by: Ana Hobden <operator@hoverbear.org> * Clean up Signed-off-by: Ana Hobden <operator@hoverbear.org> * Various small fixes Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add cached metric to compilation event Signed-off-by: Ana Hobden <operator@hoverbear.org> * Clean up test spans * More robust protobuf Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add options calls to WASM modules. Signed-off-by: Ana Hobden <operator@hoverbear.org> * checkpoint Signed-off-by: Ana Hobden <operator@hoverbear.org> * Improve fingerprinter. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add wasm-timings feature Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add assert config Signed-off-by: Ana Hobden <operator@hoverbear.org> * Extract fingerprint and artifact cache. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Remove protobuf guide. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Make things much more safe. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fix a whole swack of integer sizing issues. Signed-off-by: Ana Hobden <operator@hoverbear.org> * Update to git master lucet Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add nix env Signed-off-by: Ana Hobden <operator@hoverbear.org> * Update lucet/tracing Signed-off-by: Ana Hobden <operator@hoverbear.org> * Note we support wat as well Signed-off-by: Ana Hobden <operator@hoverbear.org> * Raise some debug to info Signed-off-by: Ana Hobden <operator@hoverbear.org> * fmt Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add artifact_cache knob Signed-off-by: Ana Hobden <operator@hoverbear.org> * Remove WATs Signed-off-by: Ana Hobden <operator@hoverbear.org> * Clean up formatting and some clippy lints Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup features in benches Signed-off-by: Ana Hobden <operator@hoverbear.org> * Checker now needs cmake Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fmt Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup bench features Signed-off-by: Ana Hobden <operator@hoverbear.org> * Add OS support to transforms on website Signed-off-by: Ana Hobden <operator@hoverbear.org> * Wasm doesn't need wabt anymore Signed-off-by: Ana Hobden <operator@hoverbear.org> * Fixup CI Signed-off-by: Ana Hobden <operator@hoverbear.org> * Remove bench features haha Signed-off-by: Ana Hobden <operator@hoverbear.org> * Correct lib authors Signed-off-by: Ana Hobden <operator@hoverbear.org> * Yank utf-8 handling change Signed-off-by: Ana Hobden <operator@hoverbear.org> * generate Signed-off-by: Ana Hobden <operator@hoverbear.org> * Decouple wasm modules from workspace Signed-off-by: Ana Hobden <operator@hoverbear.org> Signed-off-by: Brian Menges <brian.menges@anaplan.com>

Hoverbear added 4 commits March 2, 2020 15:16

Add wip

e1b5c2c

Signed-off-by: Ana Hobden <operator@hoverbear.org>

wip

47f3231

Signed-off-by: Ana Hobden <operator@hoverbear.org>

wip

13bac4d

Signed-off-by: Ana Hobden <operator@hoverbear.org>

WIP

3aca385

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear self-assigned this Mar 9, 2020

Fix bindings

aaf2a4d

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear added needs: rfc Needs an RFC before work can begin. type: new feature labels Mar 10, 2020

Hoverbear mentioned this pull request Mar 10, 2020

Ability to call transform functions within a runtime #2015

Closed

Hoverbear added 7 commits March 11, 2020 11:43

Integrate Make task

764cd51

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Remove old cruft

d58b370

Signed-off-by: Ana Hobden <operator@hoverbear.org>

More refinements to rfc

e6c368f

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Get function passing working

48ba013

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Tracing and proper FFI strings

5400b35

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Wip

6ef2bae

Signed-off-by: Ana Hobden <operator@hoverbear.org>

🚀

28597be

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear linked an issue Mar 23, 2020 that may be closed by this pull request

New wasm transform #1802

Closed

Hoverbear added 4 commits March 24, 2020 08:12

More RFC notes

dd66604

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Format wasm rfc

a3b0ec3

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Add API notes

b8cca2b

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Write more of the RFC

4ec5cc7

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear changed the title ~~DRAFT: Wasm engine~~ feature: RFC - Wasm engine Mar 31, 2020

Hoverbear added domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources domain: transforms Anything related to Vector's transform components and removed needs: rfc Needs an RFC before work can begin. labels Mar 31, 2020

Add plan of attack and platform support

059cc68

Signed-off-by: Ana Hobden <operator@hoverbear.org>

LucioFranco reviewed Mar 31, 2020

View reviewed changes

Hoverbear added 7 commits June 9, 2020 14:56

Remove WATs

c5be593

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Merge branch 'master' into wasm-engine

f7d938d

Clean up formatting and some clippy lints

07c4699

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fixup features in benches

0627d7a

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Checker now needs cmake

b6d3064

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fmt

c2764a0

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fixup bench features

850671d

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear requested a review from MOZGIII as a code owner June 10, 2020 15:38

Hoverbear force-pushed the wasm-engine branch from f8e2422 to 850671d Compare June 10, 2020 15:40

Add OS support to transforms on website

03e921f

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear force-pushed the wasm-engine branch from f3e4b51 to 03e921f Compare June 10, 2020 17:40

Hoverbear added 4 commits June 10, 2020 11:42

Wasm doesn't need wabt anymore

9455eb5

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Fixup CI

8ac1555

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Remove bench features haha

84750b2

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Correct lib authors

f2d2483

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear commented Jun 10, 2020

View reviewed changes

src/event/mod.rs Outdated Show resolved Hide resolved

Hoverbear added 3 commits June 10, 2020 12:28

Yank utf-8 handling change

7d6f2b2

Signed-off-by: Ana Hobden <operator@hoverbear.org>

generate

e12b6a2

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Decouple wasm modules from workspace

ba92982

Signed-off-by: Ana Hobden <operator@hoverbear.org>

Hoverbear changed the title ~~feature(new transform): Introduce WASM Plugins~~ feat(new transform): Introduce WASM Plugins Jun 10, 2020

lukesteensen approved these changes Jun 10, 2020

View reviewed changes

Hoverbear merged commit 5706d2a into master Jun 11, 2020

binarylogic added type: feature A value-adding code addition that introduce new functionality. and removed type: new feature labels Jun 16, 2020

binarylogic deleted the wasm-engine branch July 23, 2020 17:32

JeanMertz added the transform: wasm label Aug 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new transform): Introduce WASM Plugins #2006

feat(new transform): Introduce WASM Plugins #2006

Hoverbear commented Mar 9, 2020 •

edited

Loading

Hoverbear commented Mar 31, 2020 •

edited

Loading

LucioFranco Mar 31, 2020 •

edited

Loading

Hoverbear Mar 31, 2020

LucioFranco left a comment

LucioFranco Mar 31, 2020

Hoverbear Apr 2, 2020

Hoverbear commented Jun 9, 2020

Hoverbear Jun 10, 2020 •

edited

Loading

Hoverbear commented Jun 10, 2020

lukesteensen left a comment

lukesteensen Jun 10, 2020

Hoverbear Jun 11, 2020

		let age = modified.duration_since(std::time::UNIX_EPOCH)?;
		Ok(Self(age.as_secs()))

feat(new transform): Introduce WASM Plugins #2006

feat(new transform): Introduce WASM Plugins #2006

Conversation

Hoverbear commented Mar 9, 2020 • edited Loading

Hoverbear commented Mar 31, 2020 • edited Loading

LucioFranco Mar 31, 2020 • edited Loading

Choose a reason for hiding this comment

Hoverbear Mar 31, 2020

Choose a reason for hiding this comment

LucioFranco left a comment

Choose a reason for hiding this comment

LucioFranco Mar 31, 2020

Choose a reason for hiding this comment

Hoverbear Apr 2, 2020

Choose a reason for hiding this comment

Hoverbear commented Jun 9, 2020

Hoverbear Jun 10, 2020 • edited Loading

Choose a reason for hiding this comment

Hoverbear commented Jun 10, 2020

lukesteensen left a comment

Choose a reason for hiding this comment

lukesteensen Jun 10, 2020

Choose a reason for hiding this comment

Hoverbear Jun 11, 2020

Choose a reason for hiding this comment

Hoverbear commented Mar 9, 2020 •

edited

Loading

Hoverbear commented Mar 31, 2020 •

edited

Loading

LucioFranco Mar 31, 2020 •

edited

Loading

Hoverbear Jun 10, 2020 •

edited

Loading