Advanced Video Understanding and Semantic Search with TwelveLabs Technology

Overview

Using sophisticated multimodal foundation technology, TwelveLabs models can process a video’s audio, text, and even the temporal relationships between frames and its visual components. This multifaceted approach produces a nuanced, contextual understanding that can recognise objects, actions, background noise, and the larger narrative context of any given video. Two flagship models are available on the platform: Pegasus 1.2, a sophisticated video-language model that turns raw video into structured data by producing titles, summaries, hashtags, and detailed reports, making previously opaque video content readable and actionable, and Marengo 2.7, which excels at quickly producing context-aware embeddings for semantic search, scene classification, and pattern discovery without the need for predefined tags.

Introduction

The digital landscape is seeing a remarkable increase in video content creation. Platforms are producing over 720,000 hours of video every minute worldwide. Video makes up nearly 80% of all internet traffic and is the fastest-growing part of enterprise data. Yet, it remains one of the hardest types of content for organizations to analyze, search, and gain insights from. Traditional video management systems treat video files as unclear binary objects. They mainly rely on metadata, file names, and manual tagging. These methods only touch the surface of the rich information within the videos.

This change will transform how they operate and serve customers in a world increasingly focused on video.

Key Capabilities and Benefits

With these models, organizations can:

• Search for natural language in video: For example, they can query, “show me every bicycle crash in the race,” and find the exact scenes.
• Classify scenes and create summaries: They can go beyond simple tags to provide useful metadata and insights.
• Extract insights on a large scale: They can review thousands of hours of video to uncover hidden patterns, themes in customer feedback, or trends in product usage.
• Ensure security and scalability suitable for enterprises: They can achieve this with the reliability and governance controls expected on AWS.
• Analyze multiple types of information and time patterns: They can look at visual elements, sound, spoken dialogue, and time-based patterns together for thorough indexing.

How It Works and Typical Workflow

There is no need for specialised infrastructure or internal machine learning knowledge because the TwelveLabs models are fully managed and serverless within Amazon Bedrock.

Typical flow of usage:

1. Use Amazon Bedrock to access models: Choose TwelveLabs from the Bedrock model marketplace in the AWS Console, then submit a request for region-specific access to either Pegasus 1.2 or Marengo 2.7.
2. Connect or upload video assets: Save your video to Amazon S3 or another compatible location.
3. Establish workflows:

• Semantic search: Look for actions, events, scenes, or objects in videos using natural language.
• Classification and summarisation: Produce reports, highlights, hashtags, and chapter markers automatically.
• Get results and incorporate: Utilise the resulting timestamps, metadata, or summaries in applications for business operations, compliance, analytics, or content production.

Example Code: Using TwelveLabs in Amazon Bedrock

Here’s a typical example using the AWS SDK for Python (boto3). This shows how you might use Marengo for video semantic search.

Use Cases in the Real World

• Media & Entertainment: Instantly pull every commentary, celebration, or play in sports archives for repackaging.
• Broadcasting: Rapidly locate moments of interest in breaking news footage.
• Public Safety: Accelerate review of incident or surveillance video by searching for “car leaving the scene” or “missing person appearance”.
• Corporate Training & Compliance: Summarize and classify onboarding or safety training video hours with custom, explainable markers and highlights.

Conclusion

With TwelveLabs’ video understanding models on Amazon Bedrock, advanced video search, content summarization, and analytics become accessible to any organization, regardless of technical skill. This unlocks vast business value from video assets that previously sat idle, with AWS’s scale, security, and reliability. The partnership brings a new era of human-like video understanding for enterprises everywhere.

Drop a query if you have any questions regarding Amazon Bedrock and we will get back to you quickly.

About CloudThat