Topic: A Guide to Creating 2D Animation Using Unreal Cartoon Rendering

Speaker: Lee Jae-hong, Technical Artist at Neople

Category: Visual Art & Sound

Recommended Audience: Technical Artists, 3D Modelers, and Animators

Tags: #CartoonRendering #WorkflowAutomation

[🚨 Session Topic] This session explores the implementation of Unreal Engine-based Japanese animation-style cartoon shaders, drawing on three years of experience producing in-game cutscenes and promotional videos for 'Dungeon & Fighter.' The goal is to eliminate the uncanny feeling typical of 3D and perfectly replicate the aesthetic of 2D cel animation. I will share the process of building a shader synchronization pipeline that aligns visuals in real-time across various DCC tools—such as 3ds Max, Maya, and Substance Painter—and the engine, aimed at reducing artist fatigue and increasing the accuracy of final output predictions.

Toon shading is the most widely used technique in 'subculture' games, a genre that has evolved from a niche category into a mainstream phenomenon. Because these games must balance 3D graphics with the aesthetic appeal of 2D animation, a significant amount of effort and consideration is naturally invested in toon shading.

For developers looking to create games based on toon shading, Neople Technical Artist Lee Jae-hong shared his expertise on the challenges faced during development and key considerations for success.

Why does my toon shader look awkward?

Lee explained that in the early stages, he worked hard to identify why his toon shaders couldn't shake off the inherent awkwardness of 3D, pinpointing 'excessive texture presence' and 'dull shadows' as the culprits. His analysis was that as the shader's dark areas were layered over textures like muscle definition, it created the impression of gradients being 'painted' onto a 3D model. Furthermore, basic shaders simply multiplied achromatic shadows into dark areas, resulting in muddy colors.

He stated, "Simply dividing light and dark areas is not enough to erase the feeling of a gradient painted onto a 3D model." To solve this, he analyzed the artwork of high-quality 2D animations like 'Violet Evergarden' as a reference.

The resulting solution focused on limiting texture detail, creating natural dark areas and shadows, and using lines and diverse colors to increase visual density.

He explained that by limiting texture detail, they minimized gradients and simplified shading into two or three tones, ensuring that colors in dark areas and shadows blended naturally rather than appearing disconnected. Additionally, instead of simple black shadows, they applied clear, vibrant, shader-specific colors tailored to each part (face, hair, clothing, etc.) according to the artist's intent, while increasing detail by adding complex lines to areas like hair and collars.

Techniques for Perfect Control of Light, Lines, and Shadows

Lee divided the work into three areas—light, lines, and shadows—and began advanced control efforts. To control light, he implemented texture separation and utilized world normal maps. To prevent the unnatural separation of light and dark areas, he generated 'fake lights' that only receive directional information from the base lighting. At the point of light calculation, he added black-and-white Ambient Occlusion (AO) textures before applying color in the final stage. This, he explained, allows real-time lighting and textures to blend naturally.

Furthermore, by completely separating base textures from shade textures, he ensured that shadow areas displayed unique colors specified by the artist rather than achromatic ones. To prevent the face's shading from looking distorted at certain angles, he applied 'hand-crafted world normal maps' that allow for sculpting shading regardless of mesh geometry, creating the attractive shading characteristic of 2D.

Lee also discussed his know-how in maintaining sharp line rendering and the rediscovery of the G-Buffer. During the production of the 'Dungeon & Fighter Mobile' promotional video, low depth buffer precision caused lines to break. He resolved this by adding logic to calibrate scene depth values based on distance using parameters, ensuring sharp lines regardless of size or distance.

He also mentioned repurposing the 'Specular' channel of the G-Buffer—which is typically unused in cartoon rendering within Physically Based Rendering (PBR) systems—as a custom ID map. By assigning a specific specular value only to the bangs mesh, the post-process could detect it and draw eyebrow lines only in that area. Furthermore, by extracting a hair boundary mask using a Sobel filter, he slightly erased eyebrow lines where they met the bangs, achieving extreme detail that makes the hair appear thin and translucent.

Discussing shadow control, Lee explained the use of post-processing and the metallic channel. Using fake lights often results in a lack of shadows between objects, making characters look detached from the background. To solve this, he calculated the difference between the final render and the base color before lighting, extracting only the 'shadow areas' darkened by light during the post-process stage. He added depth by overlaying shade colors stored in the G-Buffer's Subsurface channel.

He also noted that he used the 'Metallic' channel as a custom ID to enforce the 'detail shadows' characteristic of 2D artwork. By assigning metallic values (e.g., red for hair, yellow for face), he set a forced rule in the post-process: "If the current pixel is yellow and the pixel directly above is red, draw a shadow." This enabled precise, localized 2D shadow effects completely independent of light direction.

Minimizing the Realm of Imagination: Pipeline Innovation

Lee shared how the Neople team mitigated the stress artists face when they have to guess the final look because it can only be verified within the engine. They completely overhauled the pipeline to allow artists to preview the final look—similar to the engine's output—in real-time within their primary DCC software.

Key improvements included upgrading the 3ds Max environment for modelers, optimizing the Maya workflow for animators, and porting Substance Painter for texture artists. For 3ds Max, they analyzed the core logic of the Unreal Material Editor (light, AO, base color, rim, outline) and converted it into a shared HLSL shader ported to Max. This allowed modelers to sculpt forms while seeing the effects of light and shadow in real-time in the Max viewport.

To improve the Maya workflow for animators, the team moved away from the previous, inefficient cycle of working in Maya, exporting to Unreal to check for broken lines, and returning to Maya to fix them. Instead, they ported the HLSL shaders to Maya syntax—matching the setup in 3ds Max—and provided a 'mini-tool' that allows animators to directly adjust depth intensity and line thickness. The TA noted, "While we couldn't perfectly replicate normal lines due to differences in rendering methods, the depth lines alone significantly boosted work efficiency."

For the Substance Painter port, the team recognized that Substance Painter, optimized for PBR rendering, was highly inconvenient for toon shader work, so they converted the HLSL shaders to GLSL and ported them to the software. This enabled artists to paint while checking black-and-white masks for line removal in real-time, and allowed for intuitive control of toon-style rim lighting by utilizing the metallic channel. Furthermore, they built a system that automatically packs and exports complex, fragmented texture sets according to engine rules.

Concluding the presentation, Lee shared the Neople team's future vision. The goal is to replicate the feel of hand-drawn 2D animation as closely as possible. He indicated plans to provide an environment where artists can easily draw complex face normal maps and to fully upgrade background shaders to match the 2D animation style as promotional video scales increase.