Teach One To Fish

AWS S3 CLI

feature image

AWS S3 CLI

You can use the AWS CLI in Mac, Linux or Windows. Make sure you did setup the AKSK in your computer or use Cloud9 to do these tasks.

CLI help

Run:

aws help
aws s3 help
aws s3 lshelp

Find current region

Run:

aws configure get region

Output:

ap-east-1

Create bucket

Create bucket current region

Run:

aws s3 mb s3://aaa-demo-123

Output:

make_bucket: aaa-demo-123

Create bucket in another region

Run:

aws s3 mb s3://aaa-demo-456 --region us-west-1

Output:

make_bucket: aaa-demo-456

Remove bucket

Run:

aws s3 rb aaa-demo-123

Output:

remove_bucket: aaa-demo-123

List bucket

Run:

aws s3 ls

Output:

2022-10-12 16:01:51 aaa-demo-123
2022-08-10 17:10:09 aaanew123
[...]

low level

Run:

aws s3api list-buckets --query 'Buckets[].Name'

Output:

[
    "aaa-demo-123",
    "aaanew123",
[...]

List all files inside one bucket

List all files at root folder

Run:

aws s3 ls s3://developingonaws

Output:

                           PRE images/
2022-08-12 00:56:48         14 secret-3.txt
2022-01-15 01:33:57        123 test.py

List all files inside one bucket recursively

Run:

aws s3 ls s3://developingonaws --recursive

Output:

2022-10-12 17:10:14          0 images/
2022-10-12 17:13:19    8546824 images/cloud1.jpg
2022-10-12 17:13:19    7891201 images/cloud2.jpg
2022-10-12 17:13:20   13305362 images/cloud3.jpg
2022-08-12 00:56:48         14 secret-3.txt
2022-01-15 01:33:57        123 test.py

Check bucket region

Run:

aws s3api get-bucket-location --bucket developingonaws

Output:

{
    "LocationConstraint": "ap-east-1"
}

Upload file

Run:

aws s3 cp secret-1.txt s3://developingonaws

Output:

upload: ./secret-1.txt to s3://developingonaws/secret-1.txt

Upload file

Download a file to local computer

Syntax:

aws s3api get-object --bucket YOUR_BUCKET --key path/to/key path/to/local

Run:

aws s3api get-object --bucket developingonaws --key secret-3.txt ./download-secret-3.txt

Output:

{
    "AcceptRanges": "bytes",
    "LastModified": "2022-08-11T16:56:48+00:00",
    "ContentLength": 14,
    "ETag": "\"2ca28e5f67a5c1b828dcc7808b12433e\"",
    "VersionId": "null",
    "ContentType": "binary/octet-stream",
    "Metadata": {
        "mykey": "myvalue"
    }
}

Check your folder to see if the file is downloaded.

Download a specific range of bytes from S3

We can use the following command to download a fixed range of bytes of the object from S3:

Syntax:

aws s3api get-object --bucket YOUR_BUCKET --key path/to/key --range bytes=10-20 path/to/local

Download a file that has been modified recently

We can use the following command to download a file only if has been modified since the specified timestamp.

Syntax:

aws s3api get-object --bucket YOUR_BUCKET --key path/to/key --if-modified-since 2022-08-01 path/to/local

Bucket versioning

Check current bucket versioning settings

Run:

aws s3api get-bucket-versioning --bucket developingonaws --generate-cli-skeleton output

Output:

{
    "Status": "Status",
    "MFADelete": "MFADelete"
}

Enable versioning

Run:

aws s3api put-bucket-versioning --bucket developingonaws --versioning-configuration Status=Enabled

Output:

Check Enabled versioning

Run:

aws s3api get-bucket-versioning --bucket developingonaws

Output:

{
    "Status": "Enabled"
}

list versions of the file

Run:

aws s3api list-object-versions --bucket developingonaws --prefix secret-3.txt

Output:

{
    "Versions": [
        {
            "ETag": "\"62ded210ab573df57c527909015af1e8\"",
            "Size": 9,
            "StorageClass": "STANDARD",
            "Key": "secret-3.txt",
            "VersionId": ".H0XC02kCGtWT0qXEglDXOMig8dLD4Jw",
            "IsLatest": true,
            "LastModified": "2022-10-14T10:54:45+00:00",
            "Owner": {
                "ID": "b3dd23fdfeb2c3789775cd137126836542beb3444f51cd9c4b03c854309dc633"
            }
        },
        {
            "ETag": "\"822b3744d55897ad91f12c821769a2da\"",
            "Size": 8,
            "StorageClass": "STANDARD",
            "Key": "secret-3.txt",
            "VersionId": "9Zw2H45QAq.AK98Q2z0ZO1UgWY1PPmAt",
            "IsLatest": false,
            "LastModified": "2022-10-14T10:54:22+00:00",
            "Owner": {
                "ID": "b3dd23fdfeb2c3789775cd137126836542beb3444f51cd9c4b03c854309dc633"
            }
        },
        {
            "ETag": "\"2ca28e5f67a5c1b828dcc7808b12433e\"",
            "Size": 14,
            "StorageClass": "STANDARD",
            "Key": "secret-3.txt",
            "VersionId": "null",
            "IsLatest": false,
            "LastModified": "2022-10-13T00:05:44+00:00",
            "Owner": {
                "ID": "b3dd23fdfeb2c3789775cd137126836542beb3444f51cd9c4b03c854309dc633"
            }
        }
    ]
}

Download a specific version of the object from S3

We can specific the version id of the object that we want to download.

Syntax:

aws s3api get-object --bucket YOUR_BUCKET --key path/to/key --version-id version_id path/to/local

Run:

aws s3api get-object --bucket developingonaws --key secret-3.txt --version-id 9Zw2H45QAq.AK98Q2z0ZO1UgWY1PPmAt ./download-secret-3.txt

Output:

{
    "AcceptRanges": "bytes",
    "LastModified": "2022-10-14T10:54:22+00:00",
    "ContentLength": 8,
    "ETag": "\"822b3744d55897ad91f12c821769a2da\"",
    "VersionId": "9Zw2H45QAq.AK98Q2z0ZO1UgWY1PPmAt",
    "ContentType": "text/plain",
    "Metadata": {}
}

Open download-secret-3.txt to see what is in the previous version.

Suspended versioning

Run:

aws s3api put-bucket-versioning --bucket developingonaws --versioning-configuration Status=Suspended

Output:

Check Suspended versioning

Run:

aws s3api get-bucket-versioning --bucket developingonaws

Output:

{
    "Status": "Suspended"
}

Storage Class

list object with storage class

Run:

aws s3api list-objects --bucket developingonaws

Output:

{
    "Contents": [
        {
            "Key": "images/",
            "LastModified": "2022-10-12T09:10:14+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "b3dd23fdfeb2c3789775cd137126836542beb3444f51cd9c4b03c854309dc633"
            }
        },
[...]

list object and storage class only

Run:

aws s3api list-objects --bucket developingonaws --query 'Contents[].{Key: Key, StorageClass: StorageClass}'

Output:

[...]
    {
        "Key": "images/cloud3.jpg",
        "StorageClass": "STANDARD"
    },
    {
        "Key": "secret-1.txt",
        "StorageClass": "GLACIER"
    },
    {
        "Key": "secret-3.txt",
        "StorageClass": "STANDARD"
    },
[...]

Restore object from Glacier

Run:

aws s3api restore-object --bucket developingonaws --key secret-1.txt --restore-request '{"Days":40,"GlacierJobParameters":{"Tier":"Standard"}}'

Output:

[...]

Monitor the status of your restore request

Run the following command to monitor the status of your restore request:

Run:

aws s3api head-object --bucket developingonaws --key secret-1.txt

Output:

If the restore is still in progress after you run the command, you receive a response similar to the following:

{
    "Restore": "ongoing-request=\"true\"",
    ...
    "StorageClass": "GLACIER | DEEP_ARCHIVE",
    "Metadata": {}
}

After the restore is complete, you receive a response similar to the following:

{
    "Restore": "ongoing-request=\"false\", expiry-date=\"Sun, 13 Aug 2017 00:00:00 GMT\"",
    ...
    "StorageClass": "GLACIER | DEEP_ARCHIVE",
    "Metadata": {}
}

Change the object’s storage class to Amazon S3 Standard

To increase the multipart threshold of the AWS CLI, run the following command:

Run:

aws configure set default.s3.multipart_threshold 5GB

To overwrite the existing object with the Amazon S3 Standard storage class, run the following command:

Run:

aws s3 cp s3://awsexamplebucket/dir1/example.obj s3://awsexamplebucket/dir1/example.obj --storage-class STANDARD

To perform a recursive copy for an entire prefix and overwrite existing objects with the Amazon S3 Standard storage class, run the following command:

Run:

aws s3 cp s3://awsexamplebucket/dir1/ s3://awsexamplebucket/dir1/ --storage-class STANDARD --recursive --force-glacier-transfer

Note: Objects that are archived to S3 Glacier Flexible Retrieval have a minimum storage duration of 90 days. Objects that are archived to S3 Glacier Deep Archive have a minimum storage duration of 180 days. If you have overwritten an object in S3 Glacier Flexible Retrieval before the 90-day minimum, you are charged for 90 days. Similarly, objects that are in S3 Glacier Deep Archive and are overwritten before the 180-day minimum, you are charged for 180 days.

Generate pre-signed URL

default pre-signed URL

To create a pre-signed URL with the default one hour lifetime that links to an object in an S3 bucket

Run:

aws s3 presign s3://developingonaws/secret-3.txt

Output:

https://developingonaws.s3.ap-east-1.amazonaws.com/secret-3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZJY3635BS423QE2M%2F20221025%2Fap-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221025T085221Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e30b4917a1ec66e7218587a7fee279b70fd519fc785739e499a68e9b2944c2de

7 days pre-signed URL

To create a pre-signed URL with a custom lifetime 7 days that links to an object in an S3 bucket. A new key will be generated.

Run:

aws s3 presign s3://developingonaws/secret-3.txt --expires-in 604800

Output:

https://developingonaws.s3.ap-east-1.amazonaws.com/secret-3.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAZJY3635BS423QE2M%2F20221025%2Fap-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221025T085526Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=6ca0658b0c1d3f88c6543bb17c7a7310f3a41d0b1760a1a3824b55f8db1c58b6

Teach One To Fish

, ,
, ,