Deprecation of Lake Formation’s Ruled Tables Function

After cautious consideration, we’ve made the choice to finish help for Ruled Tables, efficient December 31, 2024, to give attention to open supply transactional desk codecs resembling Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. This determination stems from buyer desire for these open supply options, which supply ACID-compliant transactions, compaction, time journey, and different options beforehand offered by Ruled Tables. Amazon Net Companies (AWS) has enhanced our help for these codecs throughout varied analytics providers, together with Amazon Athena, Amazon EMR (Elastic MapReduce), AWS Glue, and Amazon Redshift, with options that embrace computerized compaction help for Apache Iceberg, retention and snapshot expiration and orphan file deletion for Apache Iceberg, enhancements to AWS Glue Knowledge Catalog CreateTable API to create Apache Iceberg tables, and AWS Glue Crawlers schema detection help throughout Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. Prospects can now use these open supply codecs to attain ACID-compliant transactions with Amazon Easy Storage Service (Amazon S3) backed knowledge, benefiting from their wealthy options and huge compatibility.

After December 31, 2024, prospects will not be capable to create Ruled Tables transactions (lakeformation:StartTransaction), write to Ruled Tables (lakeformation:UpdateTableObjects), or question your Ruled Tables utilizing Amazon Athena. Prospects will nonetheless be capable to entry their desk state data by calling the lakeformation:GetTableObjects and transaction data by calling lakeformation:ListTransactions till February 17, 2025. After February 17, 2025, all Ruled Desk APIs will begin to fail. Ruled Tables metadata will live on inside the AWS Glue Knowledge Catalog, and the Ruled Tables knowledge will stay in your S3 buckets. No different desk varieties can be affected by this transformation, together with Hive (Apache Parquet, CSV, ORC, and so forth), Iceberg, Hudi, and Delta Lake tables.

Migrating your Ruled Tables

Prospects can migrate their tables from Ruled Tables to one of many open supply codecs by copying their ruled desk knowledge on to Apache Iceberg utilizing Amazon Athena. Emigrate knowledge to Iceberg, you need to use the Amazon Athena CREATE TABLE AS (CTAS) assertion, as proven within the following code instance.

CREATE TABLE my_iceberg_table WITH ( 
    table_type="ICEBERG",
    is_external = false,
    location = 's3://mybucket/myicebergdata/'
) AS 
SELECT * FROM my_governed_table

You’ll be able to specify extra table-level properties, that are listed within the Amazon Athena Consumer Information. If you happen to specify partitions or buckets as a part of the Apache Iceberg desk definition, then you could run into the 100 partition per bucket limitation. On this case, consult with Use CTAS and INSERT INTO to work across the 100 partition restrict.

If you happen to require any help migrating your tables, or have any questions, attain out to us at governed-tables-support@amazon.com.

In regards to the writer

Mert Hocanin is a Principal Large Knowledge Architect with AWS Lake Formation.