❓Do you use incremental crawls for your data lakes already? If no - you definitely should check it as it improves speed and reduces cost of crawls. ✅ Glue Incremental crawls are using S3 event notifications pulled from SQS queue to identify folders that were added and crawl only them. If queue is empty you would not pay for Crawler run.
Share this post
🐾Optimisation for Glue Crawler - Incremental…
Share this post
❓Do you use incremental crawls for your data lakes already? If no - you definitely should check it as it improves speed and reduces cost of crawls. ✅ Glue Incremental crawls are using S3 event notifications pulled from SQS queue to identify folders that were added and crawl only them. If queue is empty you would not pay for Crawler run.