Amazon Import Export is a service that makes it easier to export large amounts of data to
Glacier. You can ship your own portable storage device to AWS and after receiving, they will mount the device and copy to an s3 bucket or an EBS volume. Not only can you import data to AWS, you can also export data from AWS using this service. It used to be Amazon Import Export until 2015 re:Invent, but with the announcement of Snowball, shipping disks is now Amazon Import Export Disk.
We had to export about 10 terabytes of data to a couple of S3 buckets for one of our clients. We had tried
Aspera already and exported a subset of data, but since we had a deadline to meet, we figured it would be faster to ship a disk from the US data center and load data directly to s3 buckets. Since I didn't find many articles about Import/Export in detail, I thought I would write one.
Choose a compatible disk
It is very important to choose a compatible external storage device to ship.
S3 export supports
NTFS, EXT2, EXT3, EXT4, exFAT, and
FAT32 filesystem types and up to 16 TB. A full list of supported file systems, maximum capacity, device type, adapter power requirements are available here.
Create an Import Job
- download the tool from here
- Extract the contents from the downloaded zip file
- You will find a file called
AWSCredentials.propertiesfile. You need secret and access keys of an IAM user that has enough access to create and view Import/Export jobs. Since we are importing data to an S3 bucket, the user should also have read and write access on the target S3 bucket(s).
- Fill the
SecretKeyon the above file and save it.
Create An Import Manifest File
Depending on the kind of import job, we need to create a manifest file in order to create the import job. This will have details of an s3 bucket for logging information, notification email, return addresses and import job destination. Web services folder we have extracted in the above step comes with example manifest files for EBS/S3/Glacier import/exports. My manifest file
ImportThis.txt for S3 Import would look like this
manifestVersion:2.0 generator:Text editor bucket:myappdatabucket logBucket:myappdatabucket-log eraseDevice:no notificationEmail:email@example.com;firstname.lastname@example.org acl:private serviceLevel:standard returnAddress: name:Bruce Wayne company:Wayne Enterprises street1:1007,Wayne Manor, Mountain Drive city:Gotham country:USA phoneNumber:201-xxx-xxxx
Pay attention to
eraseDevice option. This is a directive for AWS to whether or not the device will be erased before shipping back. The default is to erase data unless you explicitly state that you don't want it erased.
Creating the job using web service tool
Open a command prompt and cd to the directory you have extracted above. and run the following command.
java -jar lib/AWSImportExportWebServiceTool-1.0.jar CreateJob Import ImportThis.txt
This will create a job using the details provided inside the manifest file and also create a file named
SIGNATURE in your working directory. If you check the file, you will see something like this -
version:2.0 signingMethod:HmacSHA1 jobId:7Y24H signature:PyRzwSFPKdVINuT5rl+hgOJyT8Y=
Save this signature file because we will need to copy it to the root of the device before shipping it.
Get shipping address
Though you created the job successfully, you still don't know where to ship to. In order to get the shipping address, use
GetShippingLable command to generated a pre-paid shipping label.
Run the following command
java -jar lib\AWSImportExportWebServiceTool-1.0.jar GetShippingLabel
It will prompt you a series of questions like JobID that you created on the above step, the address specified in the manifest file etc. Since I have specified an s3 bucket for logging, my shipping label was saved as a pdf file on it. It looks like this.
You are now nearly done. On to the final step now.
Shipping the device
- Copy the
SIGNATUREfile to the root of the storage device.
- Download the packaging slip, print it and fill the values:
Date: mm\dd\yyyy AWS Account Email: The email you signed up with, root account Contact: Bruce Wayne Phone\Email: 425-xxx-xxxx\email@example.com JobID: 7Y24H (you should use this same jobid) DeviceIdentifier\SerialNumber: Someidentifier\YourDeviceSerial (If your device has a serial number)
- Use the prepaid shipping label you got above, print it and securely attach it to your storage device. It has the destination address.
- Ship the device and if the storage device has cables and power supply, ship them too via UPS. We have to do this ourselves. You can arrange for a pickup from UPS by calling them or you can visit a facility.
Verify Job Status
This is easy. You can verify status using below command