Solution

The developed software package consists of two subsystems.

“Autoslide”

It automatically combines video and presentation of the lecturer in the process of video post-processing. Both files are uploaded to the system. The neural network determines which slide is shown at a particular time point and automatically replaces it with a clear, bright, properly sized slide from the original presentation. This frees the editor from the long and routine work of manual inserting the necessary materials.

The system also recognizes the figure of the lecturer in the video stream; selects it and places the image on top of the inserted slide, ensuring the realism of the output video stream.

“Auto Cameraman”

It allows shooting video lectures without the participation of a videographer. Ultra-high-resolution cameras installed in the auditorium remove the overall plan. During video processing, the system independently determines the active zone and frames the image, relying on the experience of professional cameramen.

Advantages:

• Automates the cameramen and installers’ work, which routinely takes up to 80% of the time and the entire cost of production.
• The production of video lectures is cheaper: routinely, up to 90% of the budget is spent on shooting and editing.
• Accelerates the production of video lectures by eliminating manual editing, which takes up to 80% of the preparation time.
• Improves the quality of the final video product: slides are included in the video stream more correctly, the absence of visual interference improves the perception of the material.
• The affordable software is an economical budget alternative to integrated option for video conferences from world-famous manufacturers, as it does not require the purchase and installation of expensive specialized equipment.

Details

Two software tools have been developed, combined in one web interface, which allows using them both individually and together, and includes sequential processing of video lectures in each of the subsystems.

For the Autoslide subsystem to work, you need to download the source video file and presentation materials, customize settings depending on the conditions in which the video lecture was shot and the properties of the presentation materials. At the beginning of processing, presentation materials are detected throughout the entire video, which helps to stabilize the image. Next, for each sequence of frames with the same image of the presentation area, the corresponding slide is selected from the presentation materials based on the key point method; then the replacement is performed followed by color correction. At the end of the processing, the lecturer's mask is highlighted, and using it, the lecturer is placed on top of the overlay presentation; this helps to avoid unrealistic moments in the video associated with placing the presentation on top of the lecturer. You can download the file either in the video format or in the project format compatible with Adobe Premier Pro for the subsequent manual processing.

For the Auto-Cameraman subsystem to work, it is required to download the original video file previously shot from a super-high-resolution camera statically installed in the auditorium. At the beginning of processing, the lecturer and information display areas (chalk or marker boards, projectors, flip charts) are detected: specifically those that are used as part of a particular lecture to display information. Further, based on the digitized experience of the cameramen, the video stream is cropped on the basis of static positions and transitions. In the result of the Auto Cameraman subsystem work, non-informative elements of the scene are removed.

In solving these problems, neural network technologies were used. The set of initial data transmitted by the customer was marked up and used for training the neural network. Thus, a high level of realism is achieved when simulating the work of cameramen and editors.

Project Stages:

1. Creation of the Auto slide subsystem.
2. Creation of the Auto Cameraman subsystem.

"Lectorium": providing initial data (including synthetic data for hypothesis testing), consultations on the work of cameramen and editors, and interim and final testing of the developed software. Laboratory PSPOD: development of algorithms and software for the intellectual analysis of video lectures.

The scientific and technical novelty of the development is automation of the activities of cameramen and installers by non-algorithmic methods. The application of this approach is more suitable for low- formalizable, partly creative work on developing video lectures.

The 99% in-studio recognition accuracy of key elements in the frame has been achieved; in the classroom conditions, it is about 95%. Processing speed amounts to some 10 frames per second.

Currently, the Auto Cameraman subsystem is being finalized in the part related to the expansion of its functionality. In addition, there are plans to create new intelligent products in the field of distance education.

Technical advantages:

1. The possibility of applying video lectures at the post-processing stage
2. Increased realism through the use of neural network technologies
3. Configurability of the applied modules.

Technologies

Intellectual Property

Publications