AWS re:Invent 2024, the flagship annual convention, came about December 2–6, 2024, in Las Vegas, bringing collectively 1000’s of cloud fanatics, innovators, and business leaders from across the globe. This premier occasion showcased groundbreaking developments, keynotes from AWS management, hands-on technical classes, and thrilling product launches.
Analytics remained one of many key focus areas this yr, with important updates and improvements aimed toward serving to companies harness their information extra effectively and speed up insights. From enhancing information lakes to empowering AI-driven analytics, AWS unveiled new instruments and providers which can be set to form the way forward for information and analytics.
On this submit, we stroll you thru the highest analytics bulletins from re:Invent 2024 and discover how these improvements will help you unlock the complete potential of your information.
Amazon SageMaker
Introducing the following technology of Amazon SageMaker
AWS declares the following technology of Amazon SageMaker, a unified platform for information, analytics, and AI. This launch brings collectively extensively adopted AWS machine studying (ML) and analytics capabilities and gives an built-in expertise for analytics and AI with unified entry to information and built-in governance.
The subsequent technology of SageMaker additionally introduces new capabilities, together with Amazon SageMaker Unified Studio (preview), Amazon SageMaker Lakehouse, and Amazon SageMaker Information and AI Governance. Amazon SageMaker Unified Studio brings collectively performance and instruments from the vary of standalone studios, question editors, and visible instruments obtainable at the moment in Amazon EMR, AWS Glue, Amazon Redshift, Amazon Bedrock, and the prevailing Amazon SageMaker Studio. Amazon SageMaker Lakehouse gives an open information structure that reduces information silos and unifies information throughout Amazon Easy Storage Service (Amazon S3) information lakes, Redshift information warehouses, and third-party and federated information sources. Amazon SageMaker Information and AI Governance, together with Amazon SageMaker Catalog constructed on Amazon DataZone, empowers you to securely uncover, govern, and collaborate on information and AI workflows.
Amazon DynamoDB zero-ETL integration with Amazon SageMaker Lakehouse
Amazon DynamoDB zero-ETL integration with SageMaker Lakehouse automates the extraction and loading of knowledge from a DynamoDB desk into SageMaker Lakehouse, an open and safe lakehouse. Utilizing the no-code interface, you’ll be able to keep an up-to-date reproduction of your DynamoDB information within the information lake by shortly establishing your integration to deal with the entire strategy of replicating information and updating data. This zero-ETL integration reduces the complexity and operational burden of knowledge replication to allow you to deal with deriving insights out of your information. You possibly can create and handle integrations utilizing the AWS Administration Console, the AWS Command Line Interface (AWS CLI), or the SageMaker Lakehouse APIs.
Amazon S3 Tables
Amazon S3 Tables – Absolutely managed Apache Iceberg tables optimized for analytics workloads
Amazon S3 Tables ship the primary cloud object retailer with built-in Apache Iceberg help, and essentially the most easy approach to retailer tabular information at scale. S3 Tables are particularly optimized for analytics workloads, leading to as much as 3 occasions quicker question throughput and as much as 10 occasions larger transactions per second in comparison with self-managed tables. S3 Tables are designed to carry out continuous desk upkeep to routinely optimize question effectivity and storage value over time, at the same time as your information lake scales and evolves. S3 Tables integration with the AWS Glue Information Catalog is in preview, permitting you to stream, question, and visualize information—together with Amazon S3 Metadata tables—utilizing AWS analytics providers equivalent to Amazon Information Firehose, Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight.
Amazon S3 Metadata (Preview) – Best and quickest approach to handle your metadata
Amazon S3 Metadata is the best and quickest method that can assist you immediately uncover and perceive your S3 information with automated, queried metadata that updates in close to actual time. S3 Metadata helps object metadata, which incorporates system-defined particulars like measurement and the supply of the item, and {custom} metadata, which lets you use tags to annotate your objects with info like product SKU, transaction ID, or content material ranking, for instance.
S3 Metadata is designed to routinely seize metadata from objects as they’re uploaded right into a bucket, and to make that metadata queryable in a read-only desk. These metadata tables are saved in S3 Tables, the brand new S3 storage providing optimized for tabular information. Moreover, S3 Metadata integrates with Amazon Bedrock, permitting for the annotation of AI-generated movies with metadata that specifies its AI origin, creation timestamp, and the particular mannequin used for its technology.
AWS Glue
With AWS Glue 5.0, you get improved efficiency, enhanced safety, help for SageMaker Unified Studio and SageMaker Lakehouse, and extra. AWS Glue 5.0 allows you to develop, run, and scale your information integration workloads and get insights quicker.
AWS Glue 5.0 upgrades the engines to Apache Spark 3.5.2, Python 3.11, and Java 17, with new efficiency and safety enhancements. It additionally updates open desk format help to Apache Hudi 0.15.0, Apache Iceberg 1.6.1, and Delta Lake 3.2.0. AWS Glue 5.0 provides Spark native fine-grained entry management with AWS Lake Formation so you’ll be able to apply table-, column-, row-, and cell-level permissions on S3 information lakes. Lastly, AWS Glue 5.0 provides help for SageMaker Lakehouse to unify all of your information throughout S3 information lakes and Redshift information warehouses.
Amazon S3 Entry Grants now combine with AWS Glue
Amazon S3 Entry Grants now combine with AWS Glue for analytics, ML, and utility improvement workloads in AWS. S3 Entry Grants map identities out of your identification supplier (IdP), equivalent to Entra ID and Okta or AWS Id and Entry Administration (IAM) principals, to datasets saved in Amazon S3. This integration offers you the power to handle Amazon S3 permissions for end-users working jobs with AWS Glue 5.0 or later, with out the necessity to write and keep bucket insurance policies or particular person IAM roles. When end-users within the acceptable person teams entry Amazon S3 utilizing AWS Glue ETL for Apache Spark, they are going to then routinely have the mandatory permissions to learn and write information.
AWS Glue Information catalog now automates producing statistics for brand spanking new tables
The AWS Glue Information Catalog now automates producing statistics for brand spanking new tables. These statistics are built-in with a cost-based optimizer (CBO) from Amazon Redshift and Athena, leading to improved question efficiency and potential value financial savings. Beforehand, creating statistics for Iceberg tables within the Information Catalog required you to repeatedly monitor and replace configurations to your tables. Now, the Information Catalog allows you to generate statistics routinely for brand spanking new tables with one-time catalog configuration. Amazon Redshift and Athena use the up to date statistics to optimize queries, utilizing optimizations equivalent to optimum be a part of order or cost-based aggregation pushdown. The Information Catalog console gives you visibility into the up to date statistics and statistics technology runs.
AWS expands information connectivity for Amazon SageMaker Lakehouse and AWS Glue
SageMaker Lakehouse declares unified information connectivity capabilities to streamline the creation, administration, and utilization of connections to information sources throughout databases, information lakes, and enterprise purposes. SageMaker Lakehouse unified information connectivity gives a connection configuration template, help for traditional authentication strategies like primary authentication and OAuth 2.0, connection testing, metadata retrieval, and information preview. You possibly can create SageMaker Lakehouse connections by means of SageMaker Unified Studio (preview), the AWS Glue console, or a custom-built utility utilizing APIs underneath AWS Glue.
With the power to browse metadata, you’ll be able to perceive the construction and schema of the info supply and establish related tables and fields. SageMaker Lakehouse unified connectivity is offered the place SageMaker Lakehouse or AWS Glue is offered.
Saying generative AI troubleshooting for Apache Spark in AWS Glue (Preview)
AWS Glue declares generative AI troubleshooting for Apache Spark, a brand new functionality that helps information engineers and scientists shortly establish and resolve points of their Spark jobs. Spark Troubleshooting makes use of ML and generative AI applied sciences to offer automated root trigger evaluation for Spark job points, together with actionable suggestions to repair recognized points. With Spark troubleshooting, you’ll be able to provoke automated evaluation of failed jobs with a single click on on the AWS Glue console. Powered by Amazon Bedrock, Spark troubleshooting reduces debugging time from days to minutes.
The generative AI troubleshooting for Apache Spark preview is offered for jobs working on AWS Glue 4.0.
Amazon EMR
Introducing Superior Scaling in Amazon EMR Managed Scaling
We’re excited to announce Superior Scaling, a brand new functionality in Amazon EMR Managed Scaling that gives you elevated flexibility to regulate the efficiency and useful resource utilization of your Amazon EMR on EC2 clusters. With Superior Scaling, you’ll be able to configure the specified useful resource utilization or efficiency ranges to your cluster, and Amazon EMR Managed Scaling will use your intent to intelligently scale the cluster and optimize cluster compute sources.
Superior Scaling is offered with Amazon EMR launch 7.0 and later and is offered in all AWS Areas the place Amazon EMR Managed Scaling is offered.
Amazon Athena
Amazon SageMaker Lakehouse built-in entry controls now obtainable in Amazon Athena federated queries
SageMaker now helps connectivity, discovery, querying, and imposing fine-grained information entry controls on federated sources when querying information with Athena. Athena is a question service that makes it easy to investigate your information lake and federated information sources equivalent to Amazon Redshift, DynamoDB, or Snowflake utilizing SQL with out extract, remodel, and cargo (ETL) scripts. Now, information employees can connect with and unify these information sources inside SageMaker Lakehouse. Federated supply metadata is unified in SageMaker Lakehouse, the place you apply fine-grained insurance policies in a single place, serving to to streamline analytics workflows and safe your information.
Amazon Managed Service for Apache Flink
AWS introduced help for a brand new Apache Flink connector for Amazon Managed Service for Prometheus. The brand new connector, contributed by AWS for the Flink open supply venture, provides Amazon Managed Service for Prometheus as a brand new vacation spot for Flink. You need to use the brand new connector to ship processed information to an Amazon Managed Service for Prometheus vacation spot beginning with Flink model 1.19. With Amazon Managed Service for Apache Flink, you’ll be able to remodel and analyze information in actual time. There are not any servers and clusters to handle, and there’s no compute and storage infrastructure to arrange.
Amazon Managed Service for Apache Flink now delivers to Amazon SQS queues
AWS introduced help for a brand new Flink connector for Amazon Easy Queue Service (Amazon SQS). The brand new connector, contributed by AWS for the Flink open supply venture, provides Amazon SQS as a brand new vacation spot for Apache Flink. You need to use the brand new connector to ship processed information from Amazon Managed Service for Apache Flink to SQS messages with Flink, a preferred framework and engine for processing and analyzing streaming information.
Amazon Managed Service for Apache Flink now presents a brand new Flink connector for Amazon Kinesis Information Streams. This open supply connector, contributed by AWS, helps Flink 2.0 and gives a number of enhancements. It permits in-order reads throughout stream scale-up or scale-down, helps Flink’s native watermarking, and improves observability by means of unified connector metrics. Moreover, the connector makes use of the AWS SDK for Java 2.x, which helps enhanced efficiency and safety features, and native retry technique. You need to use the brand new connector to learn information from a Kinesis information stream beginning with Flink model 1.19.
Amazon Redshift
Amazon SageMaker Lakehouse and Amazon Redshift help for zero-ETL integrations from eight purposes
SageMaker Lakehouse and Amazon Redshift now help zero-ETL integrations from purposes, automating the extraction and loading of knowledge from eight purposes, together with Salesforce, SAP, ServiceNow, and Zendesk. As an open, unified, and safe lakehouse to your analytics and AI initiatives, SageMaker Lakehouse enhances these integrations to streamline your information administration processes. These zero-ETL integrations are absolutely managed by AWS and decrease the necessity to construct ETL information pipelines. Optimize your information ingestion processes and focus as a substitute on evaluation and gaining insights.
AWS declares the overall availability of Amazon Redshift multi-data warehouse writes by means of information sharing. Now you can begin writing to Redshift databases from a number of Redshift information warehouses in only a few clicks. With Redshift multi-data warehouse writes by means of information sharing, you’ll be able to maintain ETL jobs extra predictable by splitting workloads between a number of warehouses, serving to you meet your workload efficiency necessities with much less effort and time. Your information is instantly obtainable throughout AWS accounts and Areas after it’s dedicated, enabling higher collaboration throughout your group.
Saying Amazon Redshift Serverless with AI-driven scaling and optimization
Amazon Redshift Serverless introduces the following technology of AI-driven scaling and optimization in cloud information warehousing. Redshift Serverless makes use of AI strategies to routinely scale with workload modifications throughout all key dimensions—equivalent to information quantity modifications, variety of concurrent customers, and question complexity—to fulfill and keep your price-performance targets. Amazon inside checks show that this optimization can present you as much as 10 occasions higher worth efficiency for variable workloads, with out handbook intervention.
Redshift Serverless with AI-driven scaling and optimization is offered in all AWS Areas the place Redshift Serverless is offered.
Amazon Redshift now helps incremental refresh of materialized views on information lake tables. This functionality helps you enhance question efficiency to your information lake queries in an economical and environment friendly method. By enabling incremental refresh for materialized views, you’ll be able to keep up-to-date information in a extra environment friendly and reasonably priced method.
Assist for incremental refresh for materialized views on information lake tables is now obtainable in all business Areas. To get began and be taught extra, go to Materialized views on exterior information lake tables in Amazon Redshift Spectrum.
AWS declares Amazon Redshift integration with Amazon Bedrock for generative AI
AWS declares the mixing of Amazon Redshift with Amazon Bedrock, a totally managed service providing high-performing basis fashions (FMs) making it less complicated and quicker so that you can construct generative AI purposes. This integration allows you to use giant language fashions (LLMs) from easy SQL instructions alongside your information in Amazon Redshift.
The Amazon Redshift integration with Amazon Bedrock is now typically obtainable in all Areas the place Amazon Bedrock and Amazon Redshift ML are supported. To get began, see Amazon Redshift ML integration with Amazon Bedrock.
Saying basic availability of auto-copy for Amazon Redshift
Amazon Redshift declares the overall availability of auto-copy, which simplifies information ingestion from Amazon S3 into Amazon Redshift. This new characteristic allows you to arrange steady file ingestion out of your S3 prefix and routinely load new information to tables in your Redshift information warehouse with out the necessity for added instruments or {custom} options.
Amazon Redshift auto-copy from Amazon S3 is now typically obtainable for each Redshift Serverless and Amazon Redshift RA3 Provisioned information warehouses in all AWS business Areas.
Amazon DataZone
AWS declares basic availability of Information Lineage in Amazon DataZone and the following technology of SageMaker, a functionality that routinely captures lineage from AWS Glue and Amazon Redshift to visualise lineage occasions from supply to consumption. Being OpenLineage suitable, this characteristic permits information producers to reinforce the automated lineage with lineage occasions captured from OpenLineage-enabled methods or by means of an API, to offer a complete information motion view to information customers. This characteristic automates lineage seize of schema and transformations of knowledge property and columns from AWS Glue, Amazon Redshift, and Spark executions in instruments to keep up consistency and cut back errors. Moreover, the info lineage characteristic variations lineage with every occasion, enabling you to visualise lineage at any time limit or evaluate transformations throughout an asset’s or job’s historical past.
Amazon DataZone now enhances information entry governance with enforced metadata guidelines
Amazon DataZone now helps enforced metadata guidelines for information entry workflows, offering organizations with enhanced capabilities to strengthen governance and compliance with their group wants. This new characteristic permits area homeowners to outline and implement necessary metadata necessities, ensuring information customers present important info when requesting entry to information property in Amazon DataZone. By streamlining metadata governance, this functionality helps organizations meet compliance requirements, keep audit readiness, and simplify entry workflows for better effectivity and management.
Amazon DataZone expands information entry with instruments like Tableau, Energy BI, and extra
Amazon DataZone now helps authentication with the Athena JDBC driver, enabling information customers to question their venture’s subscribed information lake property in Amazon DataZone utilizing in style enterprise intelligence (BI) and analytics instruments equivalent to Tableau, Domino, Energy BI, Microsoft Excel, SQL Workbench, and extra. Information analysts and scientists can seamlessly entry and analyze ruled information in Amazon DataZone utilizing a normal JDBC reference to their most popular instruments.
This characteristic is now obtainable in all of the AWS business Areas the place Amazon DataZone is supported. Try Increasing information evaluation and visualization choices: Amazon DataZone now integrates with Tableau, Energy BI, and extra and Connecting Amazon DataZone with exterior purposes by way of JDBC connectivity to be taught extra about the best way to join Amazon DataZone to exterior analytics instruments by way of JDBC.
Amazon QuickSight
Saying situations evaluation functionality of Amazon Q in QuickSight (preview)
A brand new state of affairs evaluation functionality of Amazon Q in QuickSight is now obtainable in preview. This new functionality gives an AI-assisted information evaluation expertise that helps you make higher choices, quicker. Amazon Q in QuickSight simplifies in-depth evaluation with step-by-step steering, saving hours of handbook information manipulation and unlocking data-driven decision-making throughout your group. You possibly can ask a query or state your aim in pure language and Amazon Q in QuickSight guides you thru each step of superior information evaluation—suggesting analytical approaches, routinely analyzing information, surfacing related insights, and summarizing findings with advised actions.
Amazon QuickSight now helps prompted reviews and reader scheduling for pixel-perfect reviews
We’re enabling QuickSight readers to generate filtered views of pixel-perfect reviews and create schedules to ship reviews by means of electronic mail. Readers can create as much as 5 schedules per dashboard for themselves. Beforehand, solely dashboard homeowners may create schedules and solely on the default (creator revealed) view of the dashboard. Now, if an creator has added controls to the pixel-perfect report, schedules could be created or up to date to respect picks on the filter management.
Prompted reviews and reader scheduling are actually obtainable in all supported QuickSight Areas—see Amazon QuickSight endpoints and quotas for QuickSight Regional endpoints.
Amazon Q in QuickSight unifies insights from structured and unstructured information
Amazon Q in QuickSight gives you with unified insights from structured and unstructured information sources by means of integration with Amazon Q Enterprise. With information tales in Amazon Q in QuickSight, you’ll be able to add paperwork, or connect with unstructured information sources from Amazon Q Enterprise, to create richer narratives or shows explaining your information with further context. This integration permits organizations to harness insights from all their information with out the necessity for handbook collation, resulting in extra knowledgeable decision-making, time financial savings, and a big aggressive edge.
Amazon Q Enterprise now gives insights out of your databases and information warehouses (preview)
AWS declares the general public preview of the mixing between Amazon Q Enterprise and QuickSight, delivering a transformative functionality that unifies solutions from structured information sources (databases, warehouses) and unstructured information (paperwork, wikis, emails) in a single utility.
With the QuickSight integration, now you can hyperlink your structured sources to Amazon Q Enterprise by means of the in depth set of knowledge supply connectors obtainable in QuickSight. This integration unifies insights throughout information sources, serving to organizations make extra knowledgeable choices whereas lowering the time and complexity historically required to assemble insights.
Amazon OpenSearch Service
Amazon OpenSearch Service zero-ETL integration with Amazon Safety Lake
Amazon OpenSearch Service now presents a zero-ETL integration with Amazon Safety Lake, enabling you to question and analyze safety information in-place immediately by means of OpenSearch. This integration lets you effectively discover voluminous information sources that have been beforehand cost-prohibitive to investigate, serving to you streamline safety investigations and procure complete visibility of your safety panorama.
Amazon OpenSearch Ingestion now helps writing safety information to Amazon Safety Lake
Amazon OpenSearch Ingestion now lets you write information into Amazon Safety Lake in actual time, permitting you to ingest safety information from each AWS and {custom} sources and uncover helpful insights into potential safety points in close to actual time. With this characteristic, now you can use OpenSearch Ingestion to ingest and remodel safety information from in style third-party sources like Palo Alto, CrowdStrike, and SentinelOne into OCSF format earlier than writing the info into Amazon Safety Lake. After the info is written to Amazon Safety Lake, it’s obtainable within the AWS Glue Information Catalog and Lake Formation tables for the respective supply.
AWS Clear Rooms
AWS Clear Rooms now helps a number of clouds and information sources
AWS Clear Rooms declares help for collaboration with datasets from a number of clouds and information sources. This launch permits corporations and their companions to collaborate with information saved in Snowflake and Athena, with out having to maneuver or share their underlying information amongst collaborators.
Conclusion
re:Invent 2024 showcased how AWS continues to push the boundaries of knowledge and analytics, delivering instruments and providers that empower organizations to derive quicker, smarter, and extra actionable insights. From developments in information lakes, information warehouses, and streaming options to the mixing of generative AI capabilities, these bulletins are designed to rework the best way companies work together with their information.
As we glance forward, it’s clear that AWS is dedicated to serving to organizations keep forward in an more and more data-driven world. Whether or not you’re modernizing your analytics stack or exploring new prospects with AI and ML, the improvements from re:Invent 2024 present the constructing blocks to unlock worth out of your information.
Keep tuned for extra deep dives into these bulletins, and don’t hesitate to discover how these instruments can speed up your journey towards data-driven success!
Concerning the Authors
Sakti Mishra serves as Principal Information and AI Options Architect at AWS, the place he helps prospects modernize their information structure and outline end-to end-data methods, together with information safety, accessibility, governance, and extra. He’s additionally the creator of Simplify Huge Information Analytics with Amazon EMR and AWS Licensed Information Engineer Research Information books. Outdoors of labor, Sakti enjoys studying new applied sciences, watching motion pictures, and visiting locations with household. He could be reached by way of LinkedIn.
Navnit Shukla serves as an AWS Specialist Options Architect with a deal with analytics. He possesses a powerful enthusiasm for helping purchasers in discovering helpful insights from their information. By way of his experience, he constructs progressive options that empower companies to reach at knowledgeable, data-driven decisions. Notably, Navnit Shukla is the completed creator of the e book titled “Information Wrangling on AWS.” He could be reached by way of LinkedIn.
