cloud_storage
A library for connecting to Google Cloud Storage through Lua.
Tutorial
You can learn more about authenticating with Google Cloud Storage here:
https://cloud.google.com/storage/docs/authentication.
Here's a quick guide to get you started:
The easiest way use this library is to create a service account for your
project. You'll need to generate a private key and store it alongside your
configuration.
Go to the cloud console: https://console.cloud.google.com. Enable Cloud
Storage if you haven't done so already. You may also need to enter billing
information.
Navigate to IAM & admin » Service accounts, located on the sidebar.
Click Create service account.

Select Furnish a new private key and select JSON as the type. After
creating the service account your browser will download a .json file. Store
that securely, since it grants access to your project.
The .json is used to create a new API client with this library:
local google = require "cloud_storage.google"
local storage = google.CloudStorage:from_json_key_file("path/to/my-key.json")
-- you can now use any of the methods on the storage object
local files = assert(storage:get_bucket("my_bucket"))
Using a p12/pem secret key
Use these directions only if you have a key created with the P12 type.
The private key downloaded from the Cloud Console is a .p12 file. The key
uses a hard coded password notasecret.
In order to use it, it must be converted to a .pem file. Run the following
command (replacing key.p12 and key.pem with the input filename and desired
output filename). Enter notasecret for the password.
openssl pkcs12 -in key.p12 -out key.pem -nodes -clcerts
One more piece of information is needed: the service account email address.
You'll find it labeled Service account ID on the service account list. It
might look something like cloud-storage@my-project.iam.gserviceaccount.com.
You can now connect to the API. Create a new client like this:
local oauth = require "cloud_storage.oauth"
local google = require "cloud_storage.google"
-- replace with your service account ID
local o = oauth.OAuth("cloud-storage@my-project.iam.gserviceaccount.com", "path/to/key.pem")
-- use your id as the second argument, everything before the @ in your service account ID
local storage = google.CloudStorage(o, "cloud-storage")
local files = assert(storage:get_bucket("my_bucket"))
HTTP Client Customization
By default, this library uses socket.http for making HTTP requests. You can provide a custom HTTP client that supports the LuaSocket interface:
local http = require("cloud_storage.http")
-- Replace with your custom HTTP client
-- Must implement a request() function compatible with LuaSocket
http.set(require("lapis.http"))
-- Now all future requests will use the `request` function in your http client...
This allows for custom timeout handling, proxy configuration, or mock clients for testing.
Reference
Module cloud_storage.oauth
This module contains the OAuth implementation used by the storage API.
local oauth = require "cloud_storage.oauth"
ouath_instance = oauth.OAuth(service_email, path_to_pem_file=nil)
Create a new OAuth object. Handles OAuth authenticated requests.
It's not necessary to use this module directly if loading private key from a
.json file.
Module cloud_storage.google
Communicates with the Google cloud storage API.
local google = require "cloud_storage.google"
Error handling
Any methods that fail to execute will return nil, an error message, and an
object that represents the error. Successful responses will return a Lua table
containing the details of the operation.
storage = google.CloudStorage(ouath_instance, project_id)
Create a new instance of a storage object from an OAuth instance. Note that the
from_json_key_file interface is a simpler way of constructing the storage
interface.
local oauth = require "cloud_storage.oauth"
local google = require "cloud_storage.google"
local o = oauth.OAuth("me@my-project.iam.gserviceaccount.com", "key.pem")
local storage = google.CloudStorage(o, "111111111111")
storage = google.CloudStorage:from_json_key_file(json_key_file_name)
Create a new instance of a storage object from a .json private key file
generated by the Google Cloud Console.
local google = require "cloud_storage.google"
local storage = google.CloudStorage:from_json_key_file("my-storage-auth.json")
storage:get_service()
https://cloud.google.com/storage/docs/xml-api/get-service
storage:get_bucket(bucket)
https://cloud.google.com/storage/docs/xml-api/get-bucket
storage:get_file(bucket, key)
https://cloud.google.com/storage/docs/xml-api/get-object
storage:delete_file(bucket, key)
https://cloud.google.com/storage/docs/xml-api/delete-object
storage:head_file(bucket, key)
https://cloud.google.com/storage/docs/xml-api/head-object
storage:copy_file(source_bucket, source_key, dest_bucket, dest_key, options={})
https://cloud.google.com/storage/docs/xml-api/put-object-copy
Options:
acl: value to use for x-goog-acl, defaults to public-read
headers: table of additional headers to include in request
storage:compose(bucket, key, source_keys, options={})
https://cloud.google.com/storage/docs/xml-api/put-object-compose
Composes a new file from multiple source files in the same bucket by
concatenating them.
source_keys is an array of keys as strings or an array of key declarations as
tables.
The table format for a source key is structured like this:
{
name = "file.png",
-- optional fields:
generation = "1361471441094000",
if_generation_match = "1361471441094000",
}
Options:
acl: value to use for x-goog-acl, defaults to public-read
mimetype: sets Content-type header for the composed file
headers: table of additional headers to include in request
storage:put_file_string(bucket, key, data, opts={})
Note: this API previously had key as an option but it was moved to
second argument
Uploads the string data to the bucket at the specified key.
Options include:
mimetype: sets Content-type header for the file, defaults to not being set
acl: sets the x-goog-acl header for file, defaults to public-read
headers: an optional array table of any additional headers to send
storage:put_file_string("my_bucket", "message.txt", "hello world!", {
mimetype = "text/plain",
acl = "private"
})
storage:put_file(bucket, fname, opts={})
Reads fname from disk and uploads it. The key of the file will be the name of
the file unless opts.key is provided. The mimetype of the file is guessed
based on the extension unless opts.mimetype is provided.
storage:put_file("my_bucket", "source.lua", {
mimetype = "text/lua"
})
All the same options from put_file_string are available for this method.
Note: This method is currently inefficient . The whole file is loaded
into memory and sent at once in the request. An alternative implementation
will be added in the future. Open a ticket if you need it.
storage:start_resumable_upload(bucket, key, options={})
https://cloud.google.com/storage/docs/xml-api/resumable-upload
Options:
acl: value to use for x-goog-acl, defaults to public-read
mimetype: sets Content-type header for the composed file
headers: table of additional headers to include in request
storage:bucket_url(bucket, opts={})
Returns full URL for a bucket
Options:
subdomain: use the {bucket}.domain format instead of default domain/{bucket} format
scheme: protocol of returned URL, default https
storage:bucket_url(bucket, opts={})
Returns full URL for a key on a bucket. Supports the same options as storage:bucket_url method.
storage:file_url("my-bucket", "pics/leafo.png") --> "https://commondatastorage.googleapis.com/my-bucket/pics/leafo.png"
storage:file_url("my-bucket", "pics/leafo.png", {
scheme = "http",
subdomain = true,
}) --> "http://my-bucket.commondatastorage.googleapis.com/pics/leafo.png"
Options:
subdomain: use the {bucket}.domain format instead of default domain/{bucket} format
scheme: protocol of returned URL, default https
storage:signed_url(bucket, key, expiration, opts={})
Creates a temporarily URL for downloading an object regardless of it's ACL.
expiration is a Unix timestamp in the future, like one generated from
os.time().
print(storage:signed_url("my_bucket", "message.txt", os.time() + 100))
Options:
headers: table of additional x-goog headers to embed into the signature
verb: HTTP verb to embed into the signature, deafult GET
scheme: protocol of returned URL, default https
storage:upload_url(bucket, key, opts={})
Generate a signed URL for browser-based upload.
Options:
content_disposition
filename If provided, will append to content disposition as filename= (and set content disposition to attachment if none is provided)
acl default project-private
success_action_redirect
expires unix timestamp. Deafult: 1 hour from now
size_limit
scheme protocol of returned URL, default https
storage:put_file_acl(bucket, key, acl)
https://cloud.google.com/storage/docs/xml-api/put-object-acls
Changelog
Future updates
All future changelogs will be post on github release notes: https://github.com/leafo/cloud_storage/releases
1.0.0 Mar 31, 2018
- Changed
put_file_string now takes key as second argument, instead of within options table
- Replace
luacrypto with luaossl
- Add support for
json private keys
- Better error handling for all API calls. (Failures return
nil, error message, and error object)
- Add
copy_file and compose methods
- Add
upload_url method to create upload URLs
- Add
start_resumable_upload method for creating resumable uploads
- Headers are canonicalized and sorted to make them consistent between calls
- Fix bug where special characters in key were not being URL encoded
- Fix bug where some special characters were not being encoded for signed URLs
- Rewrite documentation tutorial
0.1.0 Sep 29, 2013