s3cmd

Last Updated: 6/23/2024

S3CMD is a robust command line tool for uploading, retrieving, and managing S3 storage. It can be used with AWS as well as other S3 storage services including “minio”.

Installation:

root@node1:~# apt install s3cmd

Configure:

This is the quick edit version.

Use the “s3cmd –configure” command to ask a lot of questions the configuration to be used. In this example I am using a local version of S3 that is hosted provided for by “minio”.

s3cmd --configure

This will interactively prompt you for a number of values and populate a basic configuration file for you.

PRO TIP: – just the miniumum

Let’s make our lives easier and install a command line utility which will make it easier to edit INI files with.

apt install crudini

Once the “crudini” utility is installed it’s easy to make sure a few basic values are in place.

The configuration file for S3MD is named “.s3cfg” and for Unix hangs off your user directory.

My experience has led me to believe that there is a minimum number of settings s3cmd expects.

crudini is … well it’s still crude. While I wish there was a way to create the configuration file using crudini alone… alas… there isn’t a nice way. So let’s just admit that and create a file which is going to model INI style configuration the old fashioned way.

cat <<EOF | sudo tee .s3cfg
[default]
EOF

There now that is set. We now have a configuration file with the “default” section.

From there there are a number of basic settings that I found that are useful for a simple installation. Please note I set the check_ssl_certificate for False. Your mileage may vary depending on how much detail you have put into configure say “minio”

crudini --set .s3cfg default access_key AfFKQQ8gSg84ZYkutQ5J
crudini --set .s3cfg default secret_key QGHHGIk87jiAquoz37DZuLkWl04SkpajY3tiaqsK
crudini --set .s3cfg default check_ssl_certificate False
crudini --set .s3cfg default host_base s3.sparelab.net:9000
crudini --set .s3cfg default host_bucket s3.sparelab.net:9000
crudini --set .s3cfg default cloudfront_host s3.sparelab.net:9000
crudini --set .s3cfg default website_endpoint s3.sparelab.net:9000

To find visible root level buckets.

root@node1:~# s3cmd ls
2024-05-15 22:38  s3://mp3
2024-05-15 22:20  s3://spare
2024-05-15 20:30  s3://thomas

Finding contents of a particular bucket.

root@node1:~# s3cmd ls s3://thomas/
                          DIR  s3://thomas/subfolder1/
2024-05-15 21:14            2  s3://thomas/note.txt
2024-05-18 01:32            9  s3://thomas/ok.txt
2024-05-15 22:22           20  s3://thomas/other.txt

To find the diskspace

root@node1:~# s3cmd du
           0       0 objects s3://mp3/
          72       1 objects s3://spare/
          31       4 objects s3://thomas/
------------
103          Total

To create a bucket:

s3cmd mb s3://mybucket

To remove a bucket

s3cmd rb s3://mybucket

To retrieve/copy a file

s3cmd get s3://thomas/ok.txt
download: 's3://thomas/ok.txt' -> './ok.txt'  [1 of 1]
 9 of 9   100% in    0s     3.40 KB/s  done

And use the put command to put a file into the s3 bucket.

root@nodey:/home/ubuntu/h# s3cmd put helloworld_1.0-1_amd64.deb s3://thomas/
upload: 'helloworld_1.0-1_amd64.deb' -> 's3://thomas/helloworld_1.0-1_amd64.deb'  [1 of 1]
 980338 of 980338   100% in    0s     6.24 MB/s  done

Using the get command with wildcard

root@nodex:~# s3cmd get s3://thomas/hello*
download: 's3://thomas/helloworld_1.0-1_amd64.deb' -> './helloworld_1.0-1_amd64.deb'  [1 of 1]
 980338 of 980338   100% in    0s    42.07 MB/s  done

reference:
https://manpages.ubuntu.com/manpages/jammy/man1/s3cmd.1.html
https://s3tools.org/s3cmd-howto
https://s3tools.org/usage