Supporting Creative Design for 3D Cinematic Composition with Generative Artificial Intelligence

Abstract

Cinematic composition plays a pivotal role in creating compelling visual storytelling and evoking emotional impact in filmmaking. With the emergence of game-engine-based filmmaking, directors can now explore and evaluate cinematic designs within a real-time 3D virtual environment. However, achieving creative outcomes remains a challenging task, as filmmakers must navigate vast 3D spaces and carefully manipulate virtual cameras to achieve their vision.Existing automated systems often fall short, producing either context-insensitive results or overly conventional designs that fail to inspire creativity.

In response to these challenges, my PhD study introduces Cinemassist, a generative AI- based creativity support tool (CST) designed to facilitate creative cinematic composition in game-engine-based filmmaking (He et al., 2024a). Informed by a formative study conducted with professional filmmakers, the system provides a diverse range of real-time cinematic composition suggestions that transcend conventional paradigms. These suggestions operate at two levels of storytelling granularity: the frame level (the smallest unit of film production and display) and the scene level (a continuous sequence of frames depicting a single story event within a film). By offering design possibilities across these levels, Cinemassist empowers filmmakers to explore and experiment with innovative cinematic compositions. Figure 1 illustrates the user interface of the CST system.

Despite the potential of assistive tools, present rule-based and generative AI–based applications were often implemented as black boxes and struggled to adapt to the unique needs of creative cinematic design tasks. As such, this research explored the technical affordances of existing AI techniques—as design materials—and developed a technically viable solution, ensuring the technical feasibility of the system’s functionality and interaction design. Particularly, a machine learning model was constructed by integrating multiple AI techniques appropriately (He et al., 2024b). The model was trained on a diverse dataset of real film footage spanning various genres and emotional contexts. This approach enables the system to generate camera poses and sequences conditioned on animations and input semantics, affording the core functionality of the Cinemassist system.

A summative user study evaluated the effectiveness of the CST system and its impact on filmmakers’ creative thinking. This led to the development of the novel AISEE (pronounced “I see”) model, shown in Figure 2. The AISEE model introduces a novel creative thinking workflow assisted by AI that emphasizes user performance, system feedback, and previously unidentified needs in AI-assisted filmmaking. It also offers three key design implications for generative AI-based CST systems: (1) addressing the domain knowledge disparity between users of different expertise levels through coach mode and assistant mode, (2) enabling communication of vague design requirements through multi-modal inputs, and (3) supporting divergent exploration by allowing users to refine AI suggestions within a real-time spatial adjustment space.