my five cents // Upload to S3 bucket from bash

Upload to S3 bucket from bash

October 13th, 2023

If because of any reason you just need to be able to upload objects (files) to an AWS S3 bucket, you may start with the following script.

Initialize a few variables as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# path to the file to be uploaded
file=...
# identifier of the S3 bucket
s3Bucket=...
# credentials of an identity granted to upload objects
s3AccessKey=
s3SecretKey=
# identifier of the AWS region (like eu-north-1)
awsRegion=
# fixed value of the S3 service
awsService=s3
# select one of the required storage class
storageClass=STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW

Calculate a few supporting values:

1
2
3
4
5
6
# extract the filename out of the source path - it will be used for the target object name
baseFilename=$(basename ${file})
# get the current timestamp in the ISO8601 format and UTC time zone
dateTimeUtc=`date -u +'%Y%m%dT%H%M%SZ'`
# get an SHA-256 hash of the file's contents (hex string)
payloadHash=`sha256sum ${file} | awk '{print $1;}'`

Calculate the signature key according to the AWS4 standard as described here:

1
2
3
4
5
6
7
8
# date key is the current date (year, month and day only) hashed with the your secret key (prefixed with AWS4)
dateKey=`echo -n ${dateTimeUtc:0:8} | openssl dgst -sha256 -hmac "AWS4${s3SecretKey}" -binary | xxd -c 256 -p`
# date&region key is the requested AWS region hashed with the date key
dateRegionKey=`echo -n ${awsRegion} | openssl dgst -sha256 -mac hmac -macopt hexkey:${dateKey} -binary | xxd -c 256 -p`
# date&region&service key is the requested service (S3) hashed with the date&region key
dateRegionServiceKey=`echo -n ${awsService} | openssl dgst -sha256 -mac hmac -macopt hexkey:${dateRegionKey} -binary | xxd -c 256 -p`
# finally, the signing key is the fixed string aws4_requested hashed with the date&region&service key
signingKey=`echo -n "aws4_request" | openssl dgst -sha256 -mac hmac -macopt hexkey:${dateRegionServiceKey} -binary | xxd -c 256 -p`

Prepare the HTTP headers for the request. Caution: this is an example utilizing the minimum set of headers. If you need any additional ones, you may need to add them accordingly.

1
2
3
4
5
6
7
8
# not really a header, but an HTTP request method
hdrMethod=PUT
# hostname (there are other patterns possible)
hdrHost=${s3Bucket}.${awsService}.${awsRegion}.amazonaws.com
# content type of the file to be uploaded (this header is optional)
hdrContentType=`file -b --mime-type ${file}`
# the timestamp of the request (x-amz-date takes precedence over Date and allows for the ISO8601 format)
hdrXAmzDate=${dateTimeUtc}

Create a canonical request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# the target path (you may need to adapt it if the files are going to be placed somewhere else)
canonUri=/${baseFilename}
# query parameters for the request (we will not be using any)
# if needed: a collection of (uri-enoded key, equal-symbol, uri-encoded value), separated by an ampersand and sorted by the uri-encoded keys
canonQuery=
# signed headers and their values (all lowercase, alphabetical order, no whitespaces, new-line after each header)
canonHdrs=`echo -n "content-type:${hdrContentType}\nhost:${hdrHost}\nx-amz-content-sha256:${payloadHash}\nx-amz-date:${hdrXAmzDate}\nx-amz-storage-class:${storageClass}\n"`
# a list of headers being signed (all lowercase, alphabetical order, no whitespaces, semicolon-separated)
signedHdrs=`echo -n "content-type;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class"`

# finally, the canonical request (new-line separated)
canonRequest=`echo -ne "${hdrMethod}\n${canonUri}\n${canonQuery}\n${canonHdrs}\n${signedHdrs}\n${payloadHash}"`

Create the final string-to-sign:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# set the name of the algorithm
algorithm="AWS4-HMAC-SHA256"
# set the request timestamp (for clearness, the value is the same as of the original variable)
requestDateTime=${dateTimeUtc}
# set the credential scope (request date, AWS region, AWS service and a fixed string aws4_request)
credentialScope=${dateTimeUtc:0:8}/${awsRegion}/${awsService}/aws4_request

# calculate an SHA-256 hash of the canonical request
canonRequestHash=`echo -n "${canonRequest}" | sha256sum | awk '{print $1;}'`

# finally, merge all components to a string that will be signed (new-line separated)
stringToSign=`echo -ne "${algorithm}\n${requestDateTime}\n${credentialScope}\n${canonRequestHash}"`

Calculate the signature:

1
2
# the value of the string-to-sign hashed with the signing key
signature=`echo -n "${stringToSign}" | openssl dgst -sha256 -mac hmac -macopt hexkey:${signingKey} -binary | xxd -c 256 -p`

Construct the Authorization HTTP header:

1
2
# the algorithm followed by the credentials (id and scope, not the secret!), list of the signed headers and the signature itself
hdrAuthorization="${algorithm} Credential=${s3AccessKey}/${credentialScope}, SignedHeaders=${signedHdrs}, Signature=${signature}"

Upload the data using curl:

1
2
3
4
5
6
7
8
curl -X PUT -T "${file}" \
  -H "Host: ${hdrHost}" \
  -H "x-amz-date: ${hdrXAmzDate}" \
  -H "Content-Type: ${hdrContentType}" \
  -H "x-amz-storage-class: ${storageClass}" \
  -H "x-amz-content-sha256: ${payloadHash}" \
  -H "Authorization: ${hdrAuthorization}" \
  https://${hdrHost}/${baseName}

If there is any issue with the signature, the AWS S3 will precisely report, how it calculated it, giving you a lot of help for troubleshooting. Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <AWSAccessKeyId>...</AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256
        20231013T174747Z
        20231013/eu-north-1/s3/aws4_request
        -- here the hash of the canonical request --
    </StringToSign>
    <SignatureProvided> -- here hex value of the signature -- </SignatureProvided>
    <StringToSignBytes> -- here hex value of the string-to-sign -- </StringToSignBytes>
    <CanonicalRequest>PUT
        /your-file-name.ext
        ...
        content-type:...
        host:your-bucket.s3.eu-north-1.amazonaws.com
        x-amz-content-sha256: -- SHA-256 of the file --
        x-amz-date:20231013T174747Z
        x-amz-storage-class:STANDARD

        content-type;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class
        -- here the hash of the payload --
    </CanonicalRequest>
    <CanonicalRequestBytes> -- here hex value of the canonical request -- </CanonicalRequestBytes>
    <RequestId>...</RequestId>
    <HostId>....</HostId>
</Error>

Additional resources:

Next: nginx + Uvicorn + FastAPI + systemd

Previous: Convert Samsung Memo to Quillnote

Main Menu