124. S3 Performance
AWS Bites
In this episode, we discuss some tips and tricks for optimizing performance when working with Amazon S3 at scale. We start by giving an overview of how S3 works, highlighting the distributed nature of the service and how data is stored redundantly across multiple availability zones for durability. We then dive into specific tips like using multipart uploads and downloads, spreading the load across key namespaces, enabling transfer acceleration, and using S3 byte-range fetches. Overall, we aim to provide developers building S3-intensive applications with practical guidance to squeeze the most performance out of the service.
💰 SPONSORS 💰
AWS Bites is brought to you by fourTheorem an AWS consulting partner with tons of experience with S3. If you need someone to work with to optimize your S3-based workloads, check out at fourtheorem.com !
🔖 Chapters:
00:00 Intro 01:29 Some S3 usage data and how S3 works under the hood 05:25 S3 Performance tip #1 - Split into chunks & leverage concurrency 08:37 S3 Performance tip #2 - Spread the load across prefixes 12:09 S3 Performance tip #3 - Use multiple connections 14:11 S3 Performance tip #4 - Use the "edge" 17:41 S3 Performance tip #5 - Select only the data you need 19:29 Closing notes
In this episode, we mentioned the following resources:
- Our previous episode on S3 best practices: https://awsbites.com/83-bucket-list-of-s3-best-practices
- “Deep dive on Amazon S3” (re:Invent talk from 2021): https://www.youtube.com/watch?v=FJJxcwSfWYg
- More recent re:Invent talk on Amazon S3 (with updated data) - We discovered this one just after the recording: https://youtu.be/sYDJYqvNeXU
- Multi-part upload user guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- Code examples from the SDK (high-level and low-level APIs): https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html
- Node.js official helper library (@aws-sdk/lib-storage): https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-lib-storage/
- Example on how you can implement byte-range fetches: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_Scenario_UsingLargeFiles_section.html
- s3-getobject-accelerator library: https://github.com/widdix/s3-getobject-accelerator
- aws-c-s3 library: https://github.com/awslabs/aws-c-s3
- S3 storage lens: https://aws.amazon.com/s3/storage-lens/
- Docs on S3 Transfer Accelerator: https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html
- Performance Guidelines for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html
- Performance Design Patterns for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html
You can listen to AWS Bites wherever you get your podcasts:
- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017
- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q
- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==
- Breaker: https://www.breaker.audio/aws-bites
- RSS: https://anchor.fm/s/6a3312a0/podcast/rss
Do you have any AWS questions you would like us to address? Leave a comment here or connect with us on X, formerly Twitter:
#aws #cloud #s3 #storage #performance #amazon ... https://www.youtube.com/watch?v=KajcOqehb2k
155497839 Bytes