Cloud Storage: Introduction to gcloud storage

Lab Introduction

In addition to using the console to access Cloud Storage, you have the option of using gcloud. Although a lot of this comes down to personal preference, there are many things which are objectively easier to use gcloud for.

Imagine, for example, if you have dozens, hundreds, or even thousands of files that you are working with in a storage bucket. It is a lot easier to deal with the files programmatically rather than manually.

Understanding how gcloud works with Cloud Storage will allow you to work more efficiently and see possible ways to automate the management of your buckets in automated or scheduled scripts as well.

Every Cloud Storage action in the console has a corresponding gcloud command. You will learn what some of those are in addition to others that you cannot do in the console.

We will work in Cloud Shell, a Linux terminal that runs in your browser with the Google Cloud tools already installed. There is nothing to install on your own machine.

Requirements to Pass This Lab

A bucket with a name starting with gcpstudyhub-lab-cli-, located in us-central1
An object named report-final.txt at the bucket root
An object named archive/report.txt
The object scratch.txt must be deleted
Five objects uploaded under logs/ in a single command

Before You Begin

Step 0: Authentication

How this works: You need a Google account that uses the same email address as your GCP Study Hub account. When you sign into that Google account, the system will be able to grant you temporary access to a GCP project when you start the lab.

Click the button below to start your lab:

Start Lab

Instructions

Step 1: Open Google Cloud Console

Once your lab starts, use the "Open Lab in GCP Console" button from the lab portal to open the Google Cloud Console with your temporary lab project already selected.

Step 2: Open Cloud Shell

Click the Activate Cloud Shell icon in the top right of the console. It looks like a terminal prompt (>_)
A terminal panel opens at the bottom of the screen. The first start can take a minute or two
A dialog appears saying "Cloud Shell needs permission to use your credentials to make Google Cloud API calls." Click Authorize
When it is ready you will see the shell prompt, which is the text the terminal prints to show it is waiting for you to type. It ends with your project in brackets:

yourname@cloudshell:~ (your-lab-project)$

Everything after that $ is what you type.

Why your project is already set: you opened Cloud Shell from inside your lab project in the console, so it opened already pointing at that project. That is the setup you are skipping. Without it you would have to run gcloud config set project PROJECT_ID, or work through gcloud init to create a configuration, before any command would go to the right place.

Step 3: Create a Bucket

Bucket names are globally unique across all of Google Cloud, not just your project, so pick some numbers of your own to put on the end. Replace 1234 below with anything you like.

export BUCKET=gcpstudyhub-lab-cli-1234

That stores the name in a variable so you do not have to retype it. Now create the bucket:

gcloud storage buckets create gs://$BUCKET --location=us-central1

About us-central1: every bucket has a location, you choose it when you create the bucket, and it can never be changed afterwards. us-central1 is a single region in Iowa, and it is what this lab's validation checks for. In real work the choice matters: you put a bucket near the users or the compute that will read it, because that is what determines latency and what you pay to move data out.

You should see:

Creating gs://gcpstudyhub-lab-cli-1234/...

Your output will show your bucket name rather than gcpstudyhub-lab-cli-1234, since you picked your own numbers. That is true of every output in this lab.

If you get an error that the name is already taken, change the numbers and run both commands again.

Step 4: Create a File and Upload It

Make a small file to work with:

echo 'quarterly numbers' > report.txt

Upload it to your bucket:

gcloud storage cp report.txt gs://$BUCKET/

You should see something like:

Copying file://report.txt to gs://gcpstudyhub-lab-cli-1234/report.txt
  Completed files 1/1 | 18.0B/18.0B

Step 5: List and Inspect

List what is in the bucket:

gcloud storage ls gs://$BUCKET

gs://gcpstudyhub-lab-cli-1234/report.txt

That one line is the file you just created. You wrote it locally with echo, put it in the bucket with gcloud storage cp, and are now reading it back with gcloud storage ls. You have not touched the console once, and if you open the bucket in the console you will find the same object sitting there.

Now ask for details with the -l flag:

gcloud storage ls -l gs://$BUCKET/report.txt

As you can see in the output, which should look like this, -l adds the size in bytes and the creation time in front of the object name:

        18  <creation timestamp>  gs://gcpstudyhub-lab-cli-1234/report.txt
TOTAL: 1 objects, 18 bytes (18.00B)

Step 6: Copy an Object into a Folder

Copy the file into a folder called archive:

gcloud storage cp gs://$BUCKET/report.txt gs://$BUCKET/archive/

Now list the bucket again, this time recursively with -r:

gcloud storage ls -r gs://$BUCKET

You should see:

gs://gcpstudyhub-lab-cli-1234/:
gs://gcpstudyhub-lab-cli-1234/report.txt

gs://gcpstudyhub-lab-cli-1234/archive/:
gs://gcpstudyhub-lab-cli-1234/archive/report.txt

Read that output as two groups. The first is the root of the bucket, holding report.txt. The second is archive/, holding its own report.txt. The -r flag is what makes the listing descend into the second group instead of stopping at the top. Your bucket name will differ from the one shown here.

Why this works without creating a folder first: Cloud Storage actually has no folders. They are a kind of illusion. A bucket is a flat namespace, meaning it is one long list of objects with no structure underneath it. The second file is a single object whose full name is archive/report.txt, and the slash is just a character inside that name. The tools see slashes and draw you a folder tree, which is why no folder had to exist before you copied into it, and why there is no such thing as an empty folder here.

Note: both the source and the destination in that copy are gs:// paths. The gs:// scheme always means Cloud Storage, in the same way https:// means a web address, and everything after it is a bucket name followed by an object name. Because both sides were gs://, the copy happened entirely inside Google Cloud. Nothing was downloaded to Cloud Shell and uploaded again.

Step 7: Download an Object

The same cp command downloads, just by swapping which side is a gs:// path:

gcloud storage cp gs://$BUCKET/report.txt ./downloaded.txt
cat downloaded.txt

You should see the contents printed back:

quarterly numbers

That file has actually been downloaded onto your Cloud Shell machine, into your home directory. It is a real local file now and you can do anything you like with it from here. You could edit it with vim downloaded.txt, or open it in the Cloud Shell Editor, or feed it to a script. You can imagine how useful that is: pull a config file or a data export out of a bucket, change it, and push it back, without a browser anywhere in the loop.

Upload, download and server to server copy are all the same command. gcloud storage cp works out what you meant from which paths you gave it.

Step 8: Move and Rename

Rename the object at the bucket root:

gcloud storage mv gs://$BUCKET/report.txt gs://$BUCKET/report-final.txt

Confirm the result, using the same ls and the same -r recursive flag you used in Step 6:

gcloud storage ls -r gs://$BUCKET

gs://gcpstudyhub-lab-cli-1234/:
gs://gcpstudyhub-lab-cli-1234/report-final.txt

gs://gcpstudyhub-lab-cli-1234/archive/:
gs://gcpstudyhub-lab-cli-1234/archive/report.txt

Two things to notice. There is no longer a report.txt at the root. A move is not a copy: the object was renamed, so the old name is gone and only report-final.txt remains there. Note that downloaded.txt is not here either: that was downloaded onto Cloud Shell in Step 7 and never uploaded, so it only exists on the local machine.

But archive/report.txt is still sitting there untouched. Renaming one object does not affect a copy of it somewhere else, because they are two separate objects that merely happen to have the same contents.

Step 9: Delete an Object

Create and upload a throwaway file:

echo 'delete me' > scratch.txt
gcloud storage cp scratch.txt gs://$BUCKET/

Confirm it is there:

gcloud storage ls gs://$BUCKET

gs://gcpstudyhub-lab-cli-1234/report-final.txt
gs://gcpstudyhub-lab-cli-1234/scratch.txt
gs://gcpstudyhub-lab-cli-1234/archive/

There it is, listed alongside the others. Note the third line, archive/ with a trailing slash: without -r the listing does not descend into it, so it shows the prefix itself rather than the object inside.

Now remove it:

gcloud storage rm gs://$BUCKET/scratch.txt

Removing objects:
Removing gs://gcpstudyhub-lab-cli-1234/scratch.txt...
  Completed 1/1

List one more time to confirm it is gone:

gcloud storage ls -r gs://$BUCKET

Compare this against the listing you got in step 2 above. scratch.txt is no longer there. The root now holds only report-final.txt, and archive/report.txt is still below it.

Note that rm asked for no confirmation. There is no recycle bin at the command line, which is exactly why teams put guardrails such as versioning and retention policies on buckets that matter.

Step 10: Do It for Many Files at Once

Everything so far has been one file at a time, which the console does perfectly well. This step is the reason the command line exists.

Create a folder of five files locally:

mkdir -p logs
for i in 1 2 3 4 5; do echo "log line $i" > logs/day$i.txt; done
ls logs

Those three lines are ordinary Linux, not Google Cloud. mkdir -p logs makes a directory called logs and does not complain if it already exists. The middle line is a loop: for each number in the list 1 to 5 it writes the text log line 1, log line 2 and so on into a file named logs/day1.txt, logs/day2.txt and so on. The last line lists the directory so you can see the five files you just made.

Upload the whole folder with a single command, using -r for recursive:

gcloud storage cp -r logs gs://$BUCKET/

You should see five files copied:

Copying file://logs/day4.txt to gs://gcpstudyhub-lab-cli-1234/logs/day4.txt
Copying file://logs/day3.txt to gs://gcpstudyhub-lab-cli-1234/logs/day3.txt
Copying file://logs/day1.txt to gs://gcpstudyhub-lab-cli-1234/logs/day1.txt
Copying file://logs/day2.txt to gs://gcpstudyhub-lab-cli-1234/logs/day2.txt
Copying file://logs/day5.txt to gs://gcpstudyhub-lab-cli-1234/logs/day5.txt
  Completed files 5/5 | 55.0B/55.0B

Underneath that you will also get a line reporting the average throughput, something like Average throughput: 479.6B/s. Do not expect to see that exact figure: it depends on the size of the files and how fast the transfer happened to run, so it will be different for you and different again each time you run it.

Look at the order. The files did not finish in the order 1 to 5. gcloud storage transfers multiple files at the same time rather than waiting for each one, which is why it stays fast when there are thousands of them. Your own output will almost certainly be in a different order again. Doing this by hand in the console would be five separate uploads, and five thousand files would be impossible.

Now select files by pattern, something that, again, the console cannot do, at least not easily:

gcloud storage ls gs://$BUCKET/logs/*.txt

The * is a wildcard, and it stands for "any characters at all". So *.txt means every object under logs/ whose name ends in .txt, whatever comes before it. You could just as easily ask for day1* to get one day, or *.csv to get none of these. On a bucket with thousands of objects that is the difference between a command and an afternoon.

And ask how much data the whole bucket holds:

gcloud storage du -s gs://$BUCKET

That prints a single line with the total size in bytes and the bucket name.

Wildcards, searching files by patterns, recursion and parallel transfer are some of the practical reasons that you might use Cloud Storage through gcloud instead of through the console.

Validate Your Work

Get Your Completion Token

Click the button below to open the lab checker in a new window. Sign in with Google to verify your work and get your completion token.

Open Lab Checker

Submit Your Token

Paste your token below and click Validate. If valid, a button will appear to mark the lesson complete.

What You Learned

How to open Cloud Shell and run Google Cloud commands with no local setup, and why it already knows your project
How to create a bucket and upload, list, copy, download, move and delete objects with gcloud storage
That cp handles upload, download and server to server copy, deciding which you meant from the paths
That gs:// always means Cloud Storage, and that a copy between two gs:// paths never leaves Google Cloud
That folders are an illusion over a flat namespace, created simply by putting a slash in an object's name
That a move renames an object, so the old name disappears, while a separate copy elsewhere is unaffected
That rm deletes immediately with no confirmation, which is why buckets that matter need versioning or retention
That recursion, wildcards and parallel transfer are the real reasons to script Cloud Storage rather than click it

Troubleshooting

My commands stopped working, or $BUCKET is empty

Cloud Shell closes idle sessions, and reconnecting gives you a fresh shell that has forgotten your variables. Your bucket and files are unaffected. Just set the variable again with the same name you used in Step 3:

export BUCKET=gcpstudyhub-lab-cli-1234

If you cannot remember the name, run gcloud storage ls to list your buckets.

Cloud Shell is slow to start

If you have not used Cloud Shell before, or not recently, it may say it is provisioning a machine or unarchiving your disk. This is normal and only happens on the first start. Wait for the prompt to appear.

A command says permission denied

Check that the prompt shows your lab project in brackets, for example (your-lab-project)$. If it shows a different project, your commands are running somewhere you do not have access. Close Cloud Shell and reopen it from the console tab that the lab portal opened for you.

Validation Fails

If your validation fails, check that:

Your bucket name starts with gcpstudyhub-lab-cli- and the bucket is in us-central1. Location cannot be changed later, so if it is wrong, create a new bucket
gcloud storage ls -r gs://$BUCKET shows report-final.txt at the root and archive/report.txt below it
scratch.txt no longer appears in the listing
gcloud storage ls gs://$BUCKET/logs/ lists five files. If you see fewer, rerun the recursive upload from Step 10
You are using the same email address for validation that you used to start the lab