Export data to Amazon S3 with data pipelinePublic Preview
Automate recurring data exports from Stripe to your AWS S3 Storage bucket with Data Pipeline.
AWS S3 Storage destination
Data pipeline can deliver copies of all your Stripe data as Parquet files into your AWS S3 Storage bucket. It includes a directory of files for each table, delivered and updated every 6 hours.
Prerequisites
Before starting the integration, make sure you have an active AWS account and permission to:
- Create an AWS S3 bucket.
- Create an IAM role enabling Stripe to create objects in the provisioned bucket.
- Access the Stripe Dashboard with an admin or developer role.
Prepare a New Bucket
- Prepare a bucket for Stripe data by visiting your Amazon S3 console.
- If you don’t currently have an S3 bucket, you can follow the AWS guidelines for creating your first bucket. We recommend a name including “stripe,” such as “<name>-stripe-data.”
- Remember the bucket name and region because you need them for future steps.
- Keep your AWS Console open to configure an IAM role in future steps.
Start the onboarding process
- Visit the Data Pipeline Dashboard.
- Click Get started.
- Select the AWS S3 logo and click Next.
- On this permissions step you will see code blocks to be used while creating IAM role and trust policy.
Create a new IAM Role for Stripe to access and write files to bucket
- In your Amazon console, navigate to the IAM console.
- In the navigation pane of the console, click Policies > Create policy.
- To create your permission policy, select JSON, copy the JSON code in your Amazon console, and replace the entire existing policy text provided under “Permission Policy” in the Stripe Dashboard “Permissions” step.
- In the Resource section of the Policy editor code block, replace
BUCKET_
with your intended bucket name prepared in section 2.RESOURCE - Click Next. Under Policy details, add a policy name, along with any tags (optional), then click Create policy.
- Return to the navigation pane of the console, then click Roles > Create role.
- Choose the
Custom
trust policy role type, copy and paste the provided code block given under Trust policy in the data pipeline onboarding “Permissions” step, and click Next. - To select your permission policy, locate the newly created permission policy in the list. Enable the checkbox to select the policy, scroll down, and click Next.
- To create a role name, copy and paste the provided code block given under Role in the Stripe Dashboard “Permissions” step, and click Create role…
Establishing your AWS S3 connection
- Return to the Stripe Data Pipeline onboarding process.
- Enter the AWS Account ID, bucket name and region generated in the previous step.
- Select your data encryption option. If you chose to use a customer managed key, upload your public key. Check the step to generate encryptions keys to see how to create one.
- Click Next. Clicking Next sends test data to the bucket you provided, but not production data.
- When you confirm test data delivery, go to your S3 bucket.
- Open the bucket, navigate to the
penny_
directory, and open the acct_ prefixed sub-directory to locate the deliveredtest account_
test file.validation. csv - Download the
account_
file.validation. csv - Upload this test file in your data pipeline onboarding step.
- Click Confirm value.
- When you confirm the test value, click Subscribe. This subscribes you to the product and schedules the initial full load of data for delivery to your AWS S3 bucket, a process that can take 6-12 hours.