Multimodal Starter kit for storytelling: Transform silent footage to narratives
Eleven Labs x VideoDB: Adding AI Generated voiceovers to silent footage  ·  VideoDB Documentation
Case: Automatically Creating a David Attenborough-style Voiceover for Silent Footage of the Underwater World⁠⁠OverviewVoiceovers are the secret sauce that turns silent footage into captivating stories. They add depth, emotion, and excitement, elevating the viewing experience to new heights. With VideoDB's cutting-edge technology, creating dynamic voiceovers for your videos is easier than ever before.Creating AI content and merging outputs from different tools can be tough and time-consuming. Doing it manually is even harder! What if you could do it all- in just a few lines of code? VideoDB's cool tech helps you generate intelligent AI content and combine outputs from various tools quickly and automatically. No more long and tedious processes. Just simple, efficient, and awesome results! Let’s try adding an automatically generated voiceover to  in the style and voice of Sir David Attenborough. We shall do this using the powerful tech provided by Open AI, elevenlabs and VideoDB⁠⁠⁠Setup📦  Installing packages 🔑 API KeysBefore proceeding, ensure access to , , and  API key. If not, sign up for API access on the respective platforms.Get your API key from . ( Free for first 50 uploads, No credit card required ) 🎉os.environ["OPENAI_API_KEY"] = ""os.environ["ELEVEN_LABS_API_KEY"] = ""os.environ["VIDEO_DB_API_KEY"] = ""🎙️  ElevenLab's Voice ID You will also need ElevenLab's VoiceID of a Voice that you want to use.For this demo, we will be using  ElevenLabs has a large variety of voices to choose from (browse them ). Once finalized, copy the Voice ID from ElevenLabs and link it here.voiceover_artist_id = "VOICEOVER_ARTIST_ID"Implementation🌐 Step 1: Connect to VideoDBConnect to VideoDB using your API key to establish a session for uploading and manipulating video files.from videodb import connect# Connect to VideoDB using your API key🎥 Step 2: Upload Video# Upload a video by URL (replace the url with your video)video = conn.upload(url='https://youtu.be/RcRjY5kzia8') 🔍 Step 3: Analyze Scenes and Generate Scene DescriptionsStart by analyzing the scenes within your Video using VideoDB's scene indexing capabilities. This will provide context for generating the script prompt.Let's view the description of first scene from the videoscenes = video.get_scenes()print(f"{scenes[0]['start']} - {scenes[0]['end']}")print(scenes[0]["response"])The image presents a close-up, textured pattern reminiscent of organic forms. Dominated by a cool color palette of blue and turquoise hues, it gives the impression of looking at a magnified cluster of cells, crystalline structures, or perhaps a zoomed-in portion of aquatic flora. The shapes appear irregular yet symmetrical, like petals or leaves clustered tightly together. Light and shadow play across the surfaces, creating a dynamic interplay that suggests depth and complexity. The overall effect is one of natural beauty, with a soothing, almost hypnotic visual rhythm. The image is abstract enough to allow for multiple interpretations, depending on the viewer's perspective. 🎤 Step 4: Generate Voiceover Script with LLMCombine scene descriptions with the script prompt, instructing LLM to create a voiceover script in David Attenborough's style.This script prompt can be refined and tweaked to generate the most suitable output. Check out  to explore more use cases.script_prompt = "Here's the data from a scene index for a video about the underwater world. Study this and then generate a synced script based on the description below. Make sure the script is in the language, voice and style of Sir David Attenborough \n \n"full_prompt = script_prompt# run this prompt for each scene  full_prompt += f"- {scene}\n"openai_res = client.chat.completions.create(    messages=[{"role": "system", "content": full_prompt}],voiceover_script = openai_res.choices[0].message.content# If you have ElevenLab's paid plan remove the :2500 limit on voiceover_script = voiceover_script[:2500]🎤 Step 5: Generate Voiceover Audio with ElevenLabsUtilize the generated script to synthesize AI-generated voiceover narration in David Attenborough's voice using ElevenLabs API.# Call ElevenLabs API to generate voiceoverurl = f"https://api.elevenlabs.io/v1/text-to-speech/{voiceover_artist_id}"    "xi-api-key": os.environ.get("ELEVEN_LABS_API_KEY"),    "Content-Type": "application/json"    "model_id": "eleven_monolingual_v1",    "text": voiceover_script,