Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SmTP, client push flow: ignoring errors when sending data blocks #908

Closed
s373r opened this issue Oct 16, 2024 · 0 comments · Fixed by #918
Closed

SmTP, client push flow: ignoring errors when sending data blocks #908

s373r opened this issue Oct 16, 2024 · 0 comments · Fixed by #918
Assignees
Labels
bug Something isn't working

Comments

@s373r
Copy link
Member

s373r commented Oct 16, 2024

Describe the bug

While investigating kamu-data/kamu-node#147, this bug was also discovered that causes an unsuccessful result to be masked.

Steps To Reproduce

Please use the instructions from this issue: kamu-data/kamu-node#147

Within this bug report, we are interested in the sending aspect from the client (kamu).aspect of sending:

$ kamu push ca.bankofcanada.exchange-rates.daily --to odf+http://192.168.49.2:30201/s373r/ca.bankofcanada.exchange-rates.daily

Current Behavior

# kamu push: successful output

  Updated repository to 6d1c36e4 (6 block(s)) (ca.bankofcanada.exchange-rates.daily > odf+http://192.168.49.2:30201/s373r/ca.bankofcanada.exchange-rates.daily)1 dataset(s) pushed

Expected Behavior

# kamu push: failure output

A message indicating that an error occurred while sending the block date

kamu system info

build:
  appVersion: 0.205.0
  ...

Logs

No response

Anything else?

Some technical details:

WsSmartTransferProtocolClient::export_group_of_object_files():

  1. within export_task we asynchronously iterate over the tasks of sending date blocks to the recipient. If at least one result is unsuccessful, iterating is terminated:
        let mut export_task = tokio::spawn(async move {
            let src_ref = src.as_ref();
            use futures::stream::{StreamExt, TryStreamExt};
            futures::stream::iter(task_data)
                .map(Ok)
                .try_for_each_concurrent(
                    /* limit */ transfer_options.max_parallel_transfers,
                    |(s, counter)| async move {
                        let export_result = dataset_export_object_file(src_ref, s).await;
                        counter.fetch_add(1, Ordering::Relaxed);
                        export_result
                    },
                )
                .await
        });
  1. We expect the task (export_task) to finish, but ignore its outcome (_) (which could potentially be failure):
        loop {
            tokio::select! {
                _ = read_payload::<DatasetPushObjectsUploadProgressRequest>(socket) => {
                        let uploaded_files_count: i32 = uploaded_files_counter.load(Ordering::Relaxed);
                        tracing::debug!(%uploaded_files_count, "Notifying server about the upload progress");
                        self
                            .push_send_objects_upload_progress(socket, uploaded_files_count)
                            .await
                            .map_err(
                                |e| SyncError::Internal(e.int_err())
                            )?;
                    }
                _ = &mut export_task => break // <---
            }
        }

Expectations from fix:

E1) Dispatch error logged
E2) If an error occurs, the kamu push command should terminate with an error

@s373r s373r added the bug Something isn't working label Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants