Downloaded files are corrupted and cannot be unzipped

[Hoshiyar0703]

Issue : The files which are downloaded via CLI (Netstorage) aren't accessible (not openable). However, the same file when downloaded using SFTP via Netstorage is accessible without any issue.

Upload domain : newsbyteslds.upload.akamai.com
Cpcode : 719450
File name : /719450/newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz

Error message when trying to open the file using gunzip after it was downloaded via CLI :
gunzip: newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz: not in gzip format

Steps used to replicate the Issue

Installed CLI using "brew install akamai"

Installed Netstorage package using "akamai install netstorage"

Performed Netstorage setup using "akamai netstorage setup"

Downloaded the file using command below

"akamai netstorage download newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz"

Upon accessing the file using gunzip : Error Received
$ gunzip newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz
gunzip: newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz: not in gzip format

Tests performed :
Downloaded the same file using filezilla --> File was successfully opened using gunzip
Downloaded the same using CLI --> File wasn't opened using gunzip

The MD5sum of the same file downloaded via CLI and download via SFTP ( Filezilla ) aren't same. They are different

MD5 of the file downloaded via CLI
$ md5 newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz
MD5 (newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz) = a1cd5aaf96941f01e7ef52d772590502

md5sum of the file on the storage server

\md5sum newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz
3941b9dbfcdf77cf88a2fd014cf14d5b newsbytesapp_714978_714978.esw3c_S.201808210000-2400-0.gz
\

[GeoffLThompson]

Hi @Hoshiyar0703 ,

I fixed this issue by modifying the bin/netstorage.js file.

The node.js request module defaults to UTF-8 when pulling the requested file. If you set the request attribute, encoding:null this returns a Buffer object which enables the fs.Write() method, invoked lower in the call stack, to write out the binary data.

download(options) {
        return this.parseFileCpCode(options)
        .then(options => {
            let path = this.buildPath([options.cpcode, options.path, options.file])
            console.info("Downloading file: " + path ); 
            let request = {
                action: "version=1&action=download",
                method: "GET",
                path: path,
                encoding: null    //<---- Added encoding attribute
        }
            return this.makeRequest(request)
    }).then(response => {
        if (!options.outfile) {options.outfile=options.file}

        return writeDownloadFile(options.outfile, response.body)
        }).catch(error => {
            console.log("ERROR from download: " + error)
        })
    }

Once you have updated the src/netstorage file you, can test by renaming the bin/akamaiNetstorage executable and use the uncompiled akamaiNetstorage script. This will then pick up the changes you made in the bin/netstorage file and let you test the download.

Edit: Request package information: https://www.npmjs.com/package/request; See Request Options | encoding.

[pjnchai]

Hi @GeoffLThompson,

I am experiencing this issue as well. Have attempted to implement your workaround, but was not able to make the updated src/netstorage.js to take effect in the bin/akamai-netstorage executable.

Can you please provide more details on how to commit the updated change into the bin/akamai-netstorage executable?

thanks,
Peter

[GeoffLThompson]

Hi Peter (@pjnchai )

To clarify, I was incorrect in my previous post regarding the storage.js file. The file is located in the src/ directory and NOT the bin/ directory.

If this still does not address your issue then read on.

Quick

The quick method to test the fix and ensure readable files after download, is as follows:

1. If you have installed the akamai-cli tool and depending on your OS, locate your .akamai-cli folder. Under Windows, it should be located in your profile directory: C:\Users\<your_name>\.akamai-cli . On, Linux it will be in your home directory: /home/<your_name>/.akamai-cli .
2. Navigate to the .akamai-cli/src/cli-netstorage directory.
3. If you modified the src/netstorage.js file, you can test by removing or moving the compiled akamai-netstorage binaries in the bin/ directory.
4. When you run the akamai-cli tool again it should use the files in the src/ directory and thus let you test your changes.

Not Quick

The long term fix is to compile the plugin again with the fix applied. I am assuming a Windows environment here. Other environments will vary but should follow a similar approach.

1. Download and install node.js onto you computer.
2. Download and install an IDE like VSCode. This makes editing stuff easier. Read this: https://code.visualstudio.com/docs/nodejs/nodejs-tutorial
3. Download and install git.
4. Git clone this repo into wherever you store projects. Example: C:\Projects\akamai\
5. In the root of the project you just cloned, there is a file called package.json containing the projects dependencies. You will need to acquire, via the node package manager (npm), these and THEIR dependencies. There are a terrifying amount of these. You have been warned.
6. Once you have all the required dependencies you can then install (npm and run nexe. This reads the release.js script and build binaries for multiple platforms.
7. After nexe has run successfully the binaries will plop out where ever the pkg ... --output location points to.
8. Copy the binary to the C:\Users\<your_name>\.akamai-cli\src\cli-netstorage\bin folder and you should be good to go.

This is not a comprehensive set of instructions but it should get you at least further ahead.
Happy to field further questions and share what little I know. ;)

@synedra What would really help things along, would be a setup guide for working on bug fixes and features covering things like development tools and acceptable patterns, naming conventions, etc.
Is this something that is already covered or planned?

[kimdaeho-git]

If there is a modified version, please share it.