Reverse-Engineering Video Editing Styles into Programmatic Templates via AI

Speaker 0: For is concerning about, is concerning, reverse engineering, videos. So for example, I just downloaded a random 1 minute video from YouTube that is dynamic. It has different cuts. It has animations. Some worry, it has, text. Speaker 0: So my goal was to use such ready rendered videos and try to, reverse engineer it in a way that I can, get all these key frames where the video cuts, so between scenes Andrii also get, the audio transcribed it, use, these subtitles. And, actually, I wanted to get us to to to get as much insights from the video as I can. So for this, I built a pipeline that works in the following way. So, in the first stage, I send this video to the model. I used actually a predefined, ready library, which is called py detect scene. Speaker 0: Others have put it uses OpenCV and NumPy. It just identifies huge difference in pixels between frames. So when when it's IDENTIFICATION moments, it just creates a timestamp and stores it. So and that's a result, I can see, where the scenes are being split. And also, I'm using this, split. Speaker 0: And from each clip, I want to also know what is this scene about to extract some information for him from there. And I just choose 1 frame from this Andrii use OCR to scrape the text from it. And also, I I send additional these 2 MLM, the Gemini lightweight model to describe this frame and also I Try I use also a whisper open source model from open AI to transcribe this This audio so I have already more information so Okay, it currently runs to create a report, but actually I can show you already created from previous run. I can see is summary from this pipeline. I can see scenes. Speaker 0: I can see the time codes. Also, I have stored a frame. This video is dynamic, so it contains 50 scenes. That's why it loads for longer. And also, I have subtitles they can use for my analysis. Speaker 0: I have also labels, some elements that I can see in Ajit video. I can even I can see even brands and a lot of of information. I don't actually, sure that I need all this information in my case, but I I wanted to grab as much insight as I needed, as possible. And also, I used another library that that is called Open Timeline IO, that is, it is, was built by Pixar Studio. It is meant it was developed to define video sequence, for for example, where the cuts where the, like, the cuts of clips and also layers. Speaker 0: If you if somebody work with video editing, you may know that there are some layers. Andrii also, we have video and audio tracks. So this is, can be defined using this library, and I use this library to output, my this project in Final Cut Pro format, and this format can be used to open for for example, I use DaVinci Resolve for it with the editing, and I can open this file. And here I can see the same video, but it, already has cuts when the scene is changing. Sometimes it can, have mistakes because of these flashes. Speaker 0: But in most cases, it works pretty good. I can, actually use this, project to modify this video and rerender it without doing all these cuts manually. And also, it splits audio. I use another library that is called Dimax. It is from Meta. Speaker 0: It, splits, the audio into 2 tracks. 1 which is, extracts only voices from the video, but another any other sounds that are being used in background. So, yeah, that's actually a lot of data that I got from this workflow. And the other reason that that, the another problem that I wanted to solve using this workflow is also, window validation because I am building a project to create a window using user queries, and I need to have a pipeline to evaluate if this video is actually good to gain some metrics and to do to make this, this, platform better to improve it. So I actually can use this insight in my evaluation pipeline to actually generate using predefined user queries. Speaker 0: I can generate videos and then reverse engineer them to gain insights and compare against this expected expected result that we want to get. So that's that's actually what was this project about. It it it is meant to be just for a video validation and trying to to use a rendered video and make Ajit, like, edit it more easily if you already kept this decomposition of parts. Okay. So maybe you have some questions. Speaker 1: Any questions for Andre, guys? Yeah. Can can see, architecture by analyze, like, Speaker 0: so do you mean, like, if if if if it will work with real time data. Right? Exactly. Actually, I didn't think about it, but I think that this can be we need actually to to run this pipeline using a small parts of a small chunks of the video, but, actually, it is a really interesting case. I would like to consider to try to try it. Speaker 1: I I would need to document that. Maybe I'll update it. Speaker 2: Yeah. Thank you. Any other questions? Speaker 0: Question from the Can you detect, like, something else instead of cuts, like key moments, for example, in a in a video? Key moments? Yeah. Like, like, consider you have, like, a CCTV recording, and you want to find, like, all of the frames or their people in the video. Yeah. Speaker 0: Actually, yeah. That's that's a really good case. I I forget to mention this because from the old insights that I have I have even, like, okay, it's Andrii, National Geographic's brands. But also I got faces of Ajit it it could identify different faces in different parts. Like, for example, from fifth to sixth, second, it it saw the first person, the second, and, also, it can describe these people. Speaker 0: It uses LLM and OCR to to describe what is seen on these frames. So is it actually what you were talking about? Yeah. Yeah. Basically. Speaker 0: Yeah. This end actually can be used also for indexing. For example, if you have a large library of videos Andrii you run each video through this pipeline, you can make a really advanced search finding, for example, you want to find a moment where you where there is some blonde man doing something. And it can point not only to the video but also to the, exact moment time stamp. Mhmm. Speaker 2: Yeah. Probably related to that question of you're saying about having, multiple layers in the video. So with that, you could actually just you could take these and overlay them as different layers in the video that could be turned off. 1 of these thought was very interesting there is you have sentiment analysis. I don't know. Speaker 2: I saw it as a topic, but I don't know. Was it in there? So you could Topics? Yeah. I think it was it was maybe something sent Speaker 0: or Yeah. Yeah. Yeah. Descriptions, like, the it it these scores actually were given by Alan. Speaker 2: I don't know how to find it. I just saw it when I was you're scrolling past it. I I don't think it was a topic, but I think it was on the major areas. Although there's something to be able to, like, see if you actually understand not even the people that are Speaker 1: in it, but, like, how, you know, perception over time stacking on top Speaker 2: of a live stream. Interesting to see that you actually do analysis of it live stream as it's happening, and understand metadata around it, as an additional, like, layer that was overlaid onto it. Speaker 0: Yeah. Actually, since, the Google dropped their second version of embeddex model because it it's now multimodal, and I tried also to use this for some semantic indexing Andrii search. And it also can can be used in such pipelines instead of extracting frames and describe in these frames and then embedding them, it can be optimized to the just sending, the parts of the video directly to the embedding model, and it can be faster maybe. But I didn't evaluate the cost of this and the processing time. Speaker 2: That's my second question. Yeah. How much it costs to do that demo? Speaker 0: How much cost? It's it's, Speaker 2: guessing OpenWhisperer is local. Right? You can probably use some local and some that's Yeah. Speaker 0: I tried to do it as cheap as possible. That's why I used different local models that are lightweight. But the only reason the only thing that I use from external services is Gemini or LAN just for text analysis. Okay. Speaker 2: So it's it's Then you have this all open on your machine. Yeah. Oh, very good. I missed that. Right? Speaker 1: Alright. Thank you, Andre. Yeah. Thank you. It's always exciting what you can do with multimodal stuff. Speaker 1: Right? So next up, we have Julian, and he's gonna orchestrate his demos. Note. Yep. Speaker 3: I was asking him to share my screen so I didn't know if it worked or not. Speaker 0: Oh, did you? Can you not know that? Alright. Because I'm already going here. So Speaker 3: wait. If I disable video at the moment? Speaker 1: I can assure you this is the most interesting part tonight. Every time the technology stops working, you know, we all go work in tech, and this is the best we can come up. Only that was like a tool that would buncher screen. Like, just project your mind. Speaker 2: Sky. Now some cruise it. Speaker 1: I don't know. On the previous conversation that we've had and the demo, so how many of you have actually tried to play around with Stitch and any of these like video generation or like image generation models? Models. I'm just curious to know from the group. Just you, sir. Speaker 2: Yeah. Run I used Runaway a long, like, long time ago. But, yeah, we just we tried a few just we don't work in a random, but it was quite mild. I think what I find most interesting about multimodal is that we can add either just related to my work. I had to look at look at all these different multimodal models. Speaker 2: And I had an initial concept onto it as, like, audio and this video and this video to video, text to video, and you listen them all off. When you actually look into some of the edge case providers that are out there, there's some really bizarre, like, virtual try on and stuff that onto it. That's where it actually got really interesting. It wasn't even video, models. It was that's where I I find there's a whole other landscape out there of of unusual use cases. Speaker 1: For for video models. Right? Speaker 2: Yes. They're kinda conscious video models, but, yeah, like, you wanna see what you're look like wearing some clothes or something. Like, all of these exist in in these other they're not like in the mainstream models, but they they they're they're out there. Yeah. So it's worthwhile looking at the smaller providers that you submit those yourself. Speaker 2: I think the wrong way, I saw it probably the 1 I'm thinking of actually. We'll have it back to Julian Speaker 1: and we'll entertain you guys if we have issues again. Speaker 3: We'll see, we'll see. Yeah, so this is a project that we started in the hackathon a while ago. This gentleman over there was part of my team that's why I just shared him. Me? Speaker 1: Oh, or Speaker 0: He was obviously involved on TapCloud. Speaker 3: This was a while ago. Speaker 1: This is like Speaker 3: It's been a while ago but this is this is what happens when you when you write a random Hackathon project into a field Andrii then, you just keep extending it. Anyways, so, what we did and I just, expanded on this a bit. So our idea was, hey. How can we how can we make this whole workflow a little more smooth, especially a little more low code in the sense that how how can what would be the most natural natural way for us currently to work? And I guess most of us kind of end up in some agile situations where basically we have cards that we move between doing done, I mean there are thousands of project management systems but I feel agile is something that most companies have also been pushing for Andrii it's also like a kind of somewhat natural way to perform these projects. Speaker 3: So we were thinking, okay, well, how much of this, how much of these tasks could we actually do, ourselves? And how much of these could we give to an AI maybe and see, okay, maybe there are tasks that are better fitted for an AI, maybe there are tasks that, humans should actually do because they require some kind of common sense maybe that that machines don't have or they, we need to give them some specific background information or we have some constraints that we don't really want to just give. So we were thinking, okay, we can do probably most of the project management with an AI. But, then whenever humans are required, we can just jump in, do a task, or just, perform that on the fly. So basically, let's organize my marketing campaign for, for, I don't know, an AI builder community. Speaker 3: And basically, we first ask if Ajit works. Yes. So we basically just ask an agent to create us a set of tasks. So in this case, it came up came up with, these 7 tasks. So, identify key marketing channels. Speaker 3: So, for example, 1 of the tasks would be, okay, where do we actually wanna do that? Andrii then, basically, for each of these tasks, we can either decide to okay. Maybe maybe we should work on this ourselves or we should give this to an AI, and we can work on this. So very low key, just run an agent on the on the card, identify characteristics of this, start working with this, then, it produces us a whole bunch of stuff, move this to done. Yeah. Speaker 3: And then we could basically, work ourselves through this. Maybe we figure out that, the key marketing channels are already well known to us. Maybe we know we only have our Instagram page, so we could just go into, go into this, the folder that we're currently working in Andrii then, yeah, just do our own research here or even just add it into into the card directly. And then Speaker 1: I'm normally not that big. I can Speaker 3: move this card around, say done, and so on and so forth. Basically, I feel what what this mostly tries to give us is sort of a little bit of a different approach, like how we how we talk to agents. So when we were planning this, what we were thinking about was how can we make this process a little more smooth? How what's like a little more natural way of communicating between agents Andrii humans. And so what we were building was basically having this back end in the middle Andrii then sort of have these 2 representation. Speaker 3: 1 representation for the Julian, that's what you just saw, that's the front end. And then, on the other hand, have a second way into this back end basically Andrii build, build like an MCP around the back end as well at the same time that both sort of have so both MCP and front end don't really have anything to do with each other, but they both interact on the same, on the same back end. And through that, we sort of have a very natural way. We sort of have the best way to represent cards to humans. So we Speaker 1: have a very natural way of showing information to humans. Speaker 3: At the same to, to agents through these MCPs. Yeah. And then this is pretty much all it is. We have so if we have a database of cards, we, we have sort of we have a database of cards. We define some card schema that we then actually send in the initial prompt. Speaker 3: We send this to the agent. We tell the agent, okay. Look. This is what cards look like. This is what our problem looks like. Speaker 3: Please create some cards. Works most of the time, if you're not going for the smallest, cheapest model, but I mostly run this with Gemini 2.5 lite and it actually works relatively well. So, in most cases we actually get, get these cards in a pretty, pretty decent shape. The back end is running on top of ChromaDB, sort of just keeping track of the cards, providing these APIs that we so we basically do the separation on so we have the same APIs for the front end and for the MCP. Andrii then just Speaker 1: on Speaker 3: this step, the front end is calling, obviously. So the front end is calling the same APIs as the as the MCP. They just looked a little more fancy. Yeah. That's pretty much all I have to share about this. Speaker 3: Yeah. Thanks. If you have any questions, feel free. Speaker 4: How many see that now? Yeah. I mean, it's just, when I look at this, it looks a little bit, with some common things between. So the other 1 is so you can create a project or you can give it an existing project, and then it can say, you know, it would it can have a it has a concept of hire. So it can say, you know, this is the CEO I can hire. Speaker 4: Mhmm. You know, and it can automatically as well, assign each task to, an agent, which is, like, an interpreter. And then it can automatically so you and then you can say, you know, as you can do some Speaker 3: Really nice. Do you need to, like, define up front which roles you want to have? Is it figuring that out Speaker 4: on the fly? Or You don't do anything. You only give it a project. So if you have a gift gift opportunity, Speaker 1: Okay. Cool. We have another question at the back. It's a good question. Yeah. Speaker 1: Question. Yeah. Okay. Okay. 1 question for me. Speaker 1: And you had a question, did you? Yeah. I saw a question. Sorry. Yes. Speaker 1: Okay. So you're now slowly to GitHub database there, are you? You have an Speaker 3: yeah. No. It's it's just a local Speaker 1: So talk to us some more about a task that combines human input and an agent input. Sorry? Sorry? Talk to us some more about this that requires both human and Speaker 3: agent input. I mean, so basically, we generate these MD so each agent generates these MD files in the background, that are then given also as context, to to the agent in the on the way back. So basically, how I was imagining this, it's not really here yet, but, we could put basically we generate, for example, what we had before the, the target audience, so we could iterate on that as well. Speaker 1: Okay. So 1 of your cards is develop community value proposition, I have an agent to work on that. Do you then have a space for human input on that particular card? Speaker 3: Not at the front end yet. It's markdown files in the back. Okay. Not yet. They usually come out in an order that you need to work them 1 by 1. Speaker 3: They basically have all their, so we we have a we have a concept of order in there, but yeah I think would be really nice to have like this personal assistant that also tells you do this first Andrii unblock here, I think the same concept from the Speaker 1: paperclip. I think we have questions, but we have constrained on time. We can bring Julian after and definitely ask you a question. So, you've got a question that you could answer that. Right. Speaker 1: But, Speaker 3: I sit right there. That works. Speaker 1: We have what Julian is building. Anyone else use Jira or, like, any other task management software? Jira as well? Speaker 3: Anything else? Any any Notion? Notion. Anything else? Any other news? Speaker 3: I believe in a Speaker 2: trailer course. No. No. Speaker 1: So monday.com. Okay. Lot of people myhateforyou..com. Alright. So we've got Rishigesh next. Speaker 1: Yeah. That's it. So we've got Rishigesh next. And, yeah, he's gonna walk us through his what's your demo on again? Hi. Speaker 1: It's called. It's like a memory system that I've built for my agent. Okay. So you can Okay. Speaker 4: So you Speaker 1: can plug this in Speaker 3: or do that? Is Speaker 1: that good? Yeah. That should be fine. Okay. Okay. Speaker 1: So, yeah, okay. I would like to talk about this thing called. That's something that I've valued for my agents. I have haven't started sharing anything. Meaningfully, I'd like to do that with. Speaker 1: So I I remember when, like, Uber came to my hometown. Like, it was really exciting. You could just get