Amazon Import and Export Disk for Dummies

AWS

Amazon Import Export is a service that makes it easier to export large amounts of data to EBS volumes, s3 and Glacier. You can ship your own portable storage device to AWS and after receiving, they will mount the device and copy to an s3 bucket or an EBS volume. Not only can you import data to AWS, you can also export data from AWS using this service. It used to be Amazon Import Export until 2015 re:Invent, but with the announcement of Snowball, shipping disks is now Amazon Import Export Disk.

We had to export about 10 terabytes of data to a couple of S3 buckets for one of our clients. We had tried Tsunami and Aspera already and exported a subset of data, but since we had a deadline to meet, we figured it would be faster to ship a disk from the US data center and load data directly to s3 buckets. Since I didn't find many articles about Import/Export in detail, I thought I would write one.

Choose a compatible disk

It is very important to choose a compatible external storage device to ship. S3 export supports NTFS, EXT2, EXT3, EXT4, exFAT, and FAT32 filesystem types and up to 16 TB. A full list of supported file systems, maximum capacity, device type, adapter power requirements are available here.

Create an Import Job

You need to create an Import job to get started. There are a couple of ways - using Import/Export web service tool or AWS cli. I have used web service tool for this article.

  • download the tool from here
  • Extract the contents from the downloaded zip file
  • You will find a file called AWSCredentials.properties file. You need secret and access keys of an IAM user that has enough access to create and view Import/Export jobs. Since we are importing data to an S3 bucket, the user should also have read and write access on the target S3 bucket(s).
  • Fill the AccessKey and SecretKey on the above file and save it.

Create An Import Manifest File

Depending on the kind of import job, we need to create a manifest file in order to create the import job. This will have details of an s3 bucket for logging information, notification email, return addresses and import job destination. Web services folder we have extracted in the above step comes with example manifest files for EBS/S3/Glacier import/exports. My manifest file ImportThis.txt for S3 Import would look like this

manifestVersion:2.0  
generator:Text editor  
bucket:myappdatabucket  
logBucket:myappdatabucket-log  
eraseDevice:no  
notificationEmail:joker@gotham.com;bane@gotham.com  
acl:private  
serviceLevel:standard  
returnAddress:  
    name:Bruce Wayne
    company:Wayne Enterprises
    street1:1007,Wayne Manor, Mountain Drive
    city:Gotham
    country:USA
    phoneNumber:201-xxx-xxxx

Pay attention to eraseDevice option. This is a directive for AWS to whether or not the device will be erased before shipping back. The default is to erase data unless you explicitly state that you don't want it erased.

Creating the job using web service tool

Open a command prompt and cd to the directory you have extracted above. and run the following command.

java -jar lib/AWSImportExportWebServiceTool-1.0.jar CreateJob Import ImportThis.txt  

This will create a job using the details provided inside the manifest file and also create a file named SIGNATURE in your working directory. If you check the file, you will see something like this -

version:2.0  
signingMethod:HmacSHA1  
jobId:7Y24H  
signature:PyRzwSFPKdVINuT5rl+hgOJyT8Y=  

Save this signature file because we will need to copy it to the root of the device before shipping it.

Get shipping address

Though you created the job successfully, you still don't know where to ship to. In order to get the shipping address, use GetShippingLable command to generated a pre-paid shipping label.

Run the following command

java -jar lib\AWSImportExportWebServiceTool-1.0.jar GetShippingLabel  

It will prompt you a series of questions like JobID that you created on the above step, the address specified in the manifest file etc. Since I have specified an s3 bucket for logging, my shipping label was saved as a pdf file on it. It looks like this.

You are now nearly done. On to the final step now.

Shipping the device

  • Copy the SIGNATURE file to the root of the storage device.
  • Download the packaging slip, print it and fill the values:
Date: mm\dd\yyyy  
AWS Account Email: The email you signed up with, root account  
Contact: Bruce Wayne  
Phone\Email: 425-xxx-xxxx\xyz@gotham.com  
JobID: 7Y24H (you should use this same jobid)  
DeviceIdentifier\SerialNumber: Someidentifier\YourDeviceSerial  (If your device has a serial number)  
  • Use the prepaid shipping label you got above, print it and securely attach it to your storage device. It has the destination address.
  • Ship the device and if the storage device has cables and power supply, ship them too via UPS. We have to do this ourselves. You can arrange for a pickup from UPS by calling them or you can visit a facility.

Verify Job Status

This is easy. You can verify status using below command

Happy import/exporting!

comments powered by Disqus