Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update openDAL to address AWS/GHAC issues #21779

Merged
merged 16 commits into from
Jan 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/notes/2.25.x.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ one in `remote_store_headers` or `remote_execution_headers`.

Pants now supports the `{chroot}` replacement marker in remote execution contexts. (With local and Docker execution, the `{chroot}` marker is replaced with the absolute path of the sandbox directory if it appears in program arguments or environment variables. Pants will do the same as well in remote execution contexts. This requires `/bin/bash` to be available on the remote execution server.)

The OpenDAL library powering the Github Actions cache backend has been updated, picking up some bug fixes for Github Enterprise Server instances using AWS S3 as backing storage for the Github Actions cache.

### New Options System

The "legacy" options system is removed in this release. All options parsing is now handled by the new, native parser.
Expand Down
110 changes: 40 additions & 70 deletions src/rust/engine/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion src/rust/engine/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ notify = { git = "https://github.com/pantsbuild/notify", rev = "276af0f3c5f300bf
num_cpus = "1"
num_enum = "0.5"
once_cell = "1.20"
opendal = { version = "0.41", default-features = false, features = [
opendal = { version = "0.45.1", default-features = false, features = [
"services-memory",
"services-fs",
"services-ghac",
Expand Down
22 changes: 13 additions & 9 deletions src/rust/engine/remote_provider/remote_provider_opendal/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,7 @@ impl Provider {
// TODO: record Metric::RemoteStoreRequestTimeouts for timeouts
TimeoutLayer::new()
.with_timeout(options.timeout)
// TimeoutLayer requires specifying a non-zero minimum transfer speed too.
.with_speed(1),
.with_io_timeout(options.timeout),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with_speed has a deprecation warning, suggesting to use with_io_timeout instead. As I understand it, with_timeout and with_io_timeout each apply a timeout to different operations, so both are required: https://opendal.apache.org/docs/rust/opendal/layers/struct.TimeoutLayer.html#notes

)
// TODO: RetryLayer doesn't seem to retry stores, but we should
.layer(RetryLayer::new().with_max_times(options.retries + 1))
Expand All @@ -78,7 +77,7 @@ impl Provider {

pub fn fs(path: &str, scope: String, options: RemoteStoreOptions) -> Result<Provider, String> {
let mut builder = opendal::services::Fs::default();
builder.root(path).enable_path_check();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compiler told me "path always checked since RFC-3243 List Prefix" and this enable_path_check call is deprecated, so it seems we can just get rid of it. This has knock-on effects of builder not needing to be mutable, and path no longer being required as a parameter

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I think we do still need to set root (and use path): there's a dynamic/run-time error if it isn't specified (for testing locally, ./cargo test -p store shows this, CI hasn't got to it yet due to earlier errors related to the error reporting): https://github.com/apache/opendal/blob/50791255c6ef3ed259c88dcc04a4295fa60fa443/core/src/services/fs/backend.rs#L97-L103

thread 'remote_tests::smoke_test_from_options_file_provider' panicked at fs/store/src/remote_tests.rs:94:6:
called `Result::unwrap()` on an `Err` value: "failed to initialise fs remote store provider: ConfigInvalid (permanent) at  => root is not specified"

I think it's just enable_path_check that's no longer required, and this code should remain as builder.root(path);.

Background: these calls are on the same line but operate independently: using the "builder pattern" in Rust, each of them mutates builder, and returns a mutable reference to allow chaining. A "clearer" version of the same code might've been:

let mut builder = ...;
builder.root(path);
builder.enable_path_check();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense. Thanks for the info! I've put path back in, and it seems to be working fine

builder.root(path);
Provider::new(builder, scope, options)
}

Expand Down Expand Up @@ -161,19 +160,24 @@ impl Provider {

match mode {
LoadMode::Validate => {
let correct_digest = async_verified_copy(digest, false, &mut reader, destination)
.await
.map_err(|e| format!("failed to read {}: {}", path, e))?;
let correct_digest =
match async_verified_copy(digest, false, &mut reader, destination).await {
Ok(correct_digest) => correct_digest,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false),
Err(e) => return Err(format!("failed to read {}: {}", path, e)),
};

if !correct_digest {
// TODO: include the actual digest here
return Err(format!("Remote CAS gave wrong digest: expected {digest:?}"));
}
}
LoadMode::NoValidate => {
tokio::io::copy(&mut reader, destination)
.await
.map_err(|e| format!("failed to read {}: {}", path, e))?;
match tokio::io::copy(&mut reader, destination).await {
Ok(result) => result,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false),
Err(e) => return Err(format!("failed to read {}: {}", path, e)),
};
}
}
Ok(true)
Expand Down
Loading