🐾S3 Object Lambda: Transform data without extra infrastructure🐾
🤓 Do you have an application which reads data from S3, but needs to transform it before providing it to the end user? Have you heard about S3 Object Lambda?
When AWS first introduced S3 Object Lambda in 2021, it flew under the radar for many developers — I rarely meet someone who knows about this feature. Yet it solves a common problem: transforming data on-the-fly without maintaining additional processing infrastructure.
Use cases
S3 Object Lambda is a perfect match for:
Format conversion on-demand: Store data in one format (JSON, CSV, Parquet) and transform it to whatever format your application needs. For example, User A needs JSON for their modern web app, while User B needs CSV for their legacy system - both access the same file through the Object Lambda Access Point and get the format they need automatically.
Dynamic image processing: Store original high-resolution images once, then dynamically resize/compress them. For example, User A on a mobile device automatically gets a compressed 640px wide version, while User B on a desktop receives the full 1920px version - both requesting the same image through the same Access Point.
Content-based filtering: Implement content-based filtering without duplicating storage. For example, User A requests a document, your Lambda might redact sensitive fields based on their permissions, but if User B requests the same document, they might see the complete version - all using the same Access Point and Lambda function.
At the same time, it isn't suited for:
Latency-sensitive operations: The Lambda invocation adds some processing time that might affect time-critical applications.
Very complex transformations or processing of large files: Lambda functions have memory and execution time limits that could be exceeded with extremely large files or complex processing needs.
Write transformations: S3 Object Lambda only works with GET requests (reading data), not PUT/POST operations. If you need to transform data before writing it to S3, you'll need a different solution.
S3 Object Lambda vs API Gateway + Lambda
Many teams implement data transformation using API Gateway and Lambda functions, but this approach has several drawbacks compared to S3 Object Lambda:
Higher cost: API Gateway charges for each request, adding significant expense for high-volume scenarios. S3 Object Lambda avoids these API Gateway costs.
More complex: With API Gateway + Lambda, you must build and maintain REST APIs, handle authentication, manage request/response formatting, and implement error handling. S3 Object Lambda handles all this automatically.
Client modifications: Applications must be updated to use your custom API instead of standard S3 SDKs. With S3 Object Lambda, applications continue using familiar S3 operations.
Authentication overhead: You need to implement your own auth layer or integrate with API Gateway auth. S3 Object Lambda leverages S3's existing IAM permissions model that you're likely already using.
A typical API Gateway + Lambda solution could cost 2-3x more than S3 Object Lambda when considering all components and operational overhead.
How It Works
The architecture is pretty simple, when a client requests an object through the Access Point, AWS automatically:
Invokes your Lambda function
Passes the original S3 object to your function
Returns your function's transformed output to the client
Thank you for reading, let’s chat 💬
💬 Have you heard about S3 Object Lambda?
💬 What are other use cases you use S3 Object Lambda for?
💬 Are you interested in more content related to Lambda function?
I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻
