Blockout to Concept Art using AI

COFA Reflection:

COFA video gave me a detailed insight on how to get research started. Several speakers highlighted the importance of using diverse sources of information and finding different voices to find their material. Trial and error is a significant part of the process and having a linear approach may not be a good idea. A thought that always runs in my mind is “there is nothing new to say or research”  but with this video guidance I can get a better understanding on the importance of conducting research. It is more about understanding the knowledge gap in the existing field and trying to fill the void with newer research and findings. 

Thoughts on research idea:

As per my current experience and knowledge below is the idea i’ve built and plan to take a deeper dive,

“One of the most common challenges I notice is that many 3D artists and students often struggle with creating concept art. Drawing is not always their strongest skill set, and I want to help bridge that gap.

By exploring a combination of 3D and AI tools, i want to expand the horizon on having more stronger control on the creative process. The users should be able to custom make their concept art and should be able to change the elements on the go without struggling too much on the prompt engineering and focus more on the creative aspect.

This approach may offer a smart and practical way to use generative AI with user-assisted creative input, helping students overcome drawing limitations while still developing strong visual ideas.

I was planning to record the above process as a tutorial and upload it on youtube and chase this with a Reflective writing on my personal/professional blog. “

Observation:  One of the most common challenges I notice is that many 3D artists and students often struggle with creating concept art. Drawing is not always their strongest skill set, and I want to help bridge that gap.

Question: What will happen if I combine 3D blockout techniques & Generative AI  technology to help create quick concept arts?

Hypothesis: It may help artists have stronger control over their art, rather than relying much on strong drawing, 3d and concept art skills.

Experiment:  Create a demo video showcasing 3D blockout techniques combined with AI technology that allows artists to have a stronger control over their own concept art. 

Analyse: If the workflow is strong enough for the artist to have a good control on their AI art generation

Conclude: Lighting and mood of the image to be controlled easily 

Literature Review:

Several Gen AI Models have been made ranging from Dall-E, NovelAI for character generation. A large amount of focus is being placed on LTGM by Google, Microsoft, OpenAI etc.  A massive impact to visual artists who deal with visual sources as the main component of their work. Industries that will be impacted are Graphic Design, Digital Art, new media art etc. While all the developments were ongoing the line between exploration and exploitation started to surface. The collaboration between AI and artists were taken positively by helping in the ideation process, on the other hand implications these AI models can bring up were being surfaced. In my personal perspective, working in such a demanding industry where “time is money” and projects that need faster turnaround time, these models act as a support tool. (Ko et al., 2023) 

While working on a project with a client, I was tasked to come up with a brand new set design for their upcoming road show. I had a strong design idea in mind and I wanted to base the whole design on the Origami theme. The theme choice was to support their new marketing campaign. Since my drawing skills were weak and I was left with no other choice but to hire a 2D artist to help put my ideas into a visual aid. Having another person on board came with its own Pros and Cons. The positive was I had a good concept art created but on the other hand, I had to invest a lot of time to brief the artist, several iterations, several brainstorming sessions, and several different renditions. Finally I had a concept art that was sent for client approval.  

Being an academic for nearly 14 years, I've seen a paradigm shift in the mindset of students. Coming from a generation where Google & Wikipedia gave us the answers, to the current generation where every source of information is derived from ChatGPT. The term “Google It” has been replaced with “Check on ChatGPT”. We can all strongly agree that AI is here to stay and will not leave. AI will not take our jobs, the only people to lose their jobs are the ones who don’t use AI in their workflow.  Blaming our flaws are no more an excuse but using these tools to fill the void is essential for the success of the project and themselves. 

The authors interviewed 28 participants from different aspects of creative industries and they were asked several questions. The one that sparked my interest is to report the possible limitations of LTGMs (Ko et al., 2023). As per their research a major issue that they noticed was, many artists contemplated for several minutes to write down a text prompt. They found it difficult to narrate a design that requires an entire story line into a few sentences/words. The design process for generating images can be difficult for beginners and mostly it becomes trial and error (Liu & Chilton, 2022, p. 2).   Although text-to-image tools offer unlimited opportunities for visual exploration, users may struggle to achieve precise outcomes. This limitation is particularly relevant to beginners, as repeated prompt refinements can divert a user's attention from creative decision making. It is important for the designers to have creative control on their work and use prompt engineering as a tool to assist the creative process.

Generative AI can make visual creation more accessible, interacting effectively with these systems may introduce a different technical barrier (Oppenlaender et al., 2025, p. 10207) . With so many different AI models available, prompt engineering has become a new type of skill that must be acquired. Oppenlaender also highlights that the prompt engineering has diverted users from the creative control. Although text-to-image models can rapidly generate visually detailed images, text prompts may not provide artists with sufficient control over spatial and compositional decisions. Zhang et al. (2023) acknowledge that “precisely expressing complex layouts, poses, shapes and forms can be difficult via text prompts alone” (p. 3836). This limitation is particularly relevant to concept-art development, where camera placement, perspective, silhouette and spatial relationships must be deliberately controlled. 

Revised research question:

How can 3D blockouts and generative AI be combined to help 3D artists create  concept art without relying on drawing skills and complex prompt engineering?

Paper link:

Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works 

Design Guidelines for Prompt Engineering Text-to-Image Generative Models 

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering 

Adding Conditional Control to Text-to-Image Diffusion Models 

References

Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023). Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works. Proceedings of the 28Th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3581641.3584078 

Liu, V., & Chilton, L. B. (2022). Design guidelines for prompt engineering text-to-image generative models. 

Oppenlaender, J., Linder, R., & Silvennoinen, J. (2025). Prompting AI art: An investigation into the creative skill of prompt engineering. International Journal of Human–Computer Interaction, 41(16), 10207–10229. https://doi.org/10.1080/10447318.2024.2431761

Zhang, L., Rao, A., & Agrawala, M. (n.d.). Adding Conditional Control to Text-to-Image Diffusion Models [Review of Adding Conditional Control to Text-to-Image Diffusion Models]. Retrieved July 13, 2026, from https://arxiv.org/pdf/2302.05543

Link to the Outcome

3D Blockout to AI Concept Art - tyDiffusion and 3ds Max workflow

Background Research

Before beginning the practical work, I reviewed academic literature related to generative AI, prompt engineering and controlled image generation. I also looked at the official documentation and existing demonstrations of tyDiffusion, Stable Diffusion and ControlNet.

(Ibele, 2026)

This research helped me understand that text prompts are not the only way to control an AI-generated image. Information such as depth, shapes, silhouettes and reference images can also guide the generation process. Based on this, I decided to compare a simple text-based generation with an outcome guided by a 3D blockout.

Planning the Creative Experiment

For the practical experiment, I decided to create an abandoned city environment inspired by the visual atmosphere of The Last of Us. 

(Naughty Dog, 2013) 

This type of environment was suitable because it contains recognisable elements such as damaged buildings, abandoned vehicles, roadblocks and vegetation taking over the city.

I began with a very rough sketch to plan the general composition. The purpose of the sketch was not to create professional concept art. It was only used to establish the possible locations of the buildings, road, vehicles and other objects.

Figure 1. The initial rough sketch used to plan the basic composition and placement of objects.

Creating the 3D Blockout

The scene was created in 3ds Max. I began by setting the project to a metric unit system and added a human reference with a height of approximately 183 centimetres. This helped me judge the scale and proportion of the environment.

Basic boxes were then used to represent buildings on both sides of the street. These objects were duplicated, moved and scaled to create variation. A simple plane was edited to form the road and pavement.

The project used a fixed camera view, so I focused mainly on the objects that were visible through the camera. Areas outside the camera were not developed in detail because they would not influence the final image. This helped keep the experiment small and allowed me to concentrate on composition rather than detailed modelling.

Figure 2. The 3D blockout created using simple geometry to establish scale, depth and composition.

Testing AI Generation

After completing the basic blockout, I opened tyDiffusion inside 3ds Max. I first tested the scene using the simple prompt “New York City.” The AI produced a visually interesting city image, but it did not properly follow the structure of my 3D scene.

I then activated Depth ControlNet. This allowed the AI to read the depth and basic arrangement of the 3D objects. The ControlNet weight was adjusted to test how closely the generated image followed the blockout. A higher weight kept the result closer to the 3D composition, while a lower weight gave the AI more freedom to change the design.

This comparison was important because it showed the difference between depending mainly on a text prompt and using the 3D blockout as a visual guide.

Figure 3. The AI-generated outcome after Depth ControlNet was used to follow the structure of the 3D blockout.

Refining the Scene

The scene was developed through several small stages. Additional assets such as vehicles, bins, roadblocks and trees were added using Chaos Cosmos. These objects were selected because they supported the idea of an abandoned city and created more interesting silhouettes for the AI to follow.

The written prompt was also gradually refined. Descriptions such as abandoned city, old rusted vehicles, falling leaves, excessive vines and mist were added. Negative prompts were used to reduce unwanted results such as blurry, noisy, low-quality or watermarked images.

The results were generated several times after each major change. This allowed me to observe how additional geometry and small prompt changes affected the final image. Adding trees and vegetation to the buildings created one of the clearest improvements because it helped the AI understand that nature was taking over the environment.

Figure 4. The refined generation after vehicles, roadblocks and vegetation were added to improve the environment and its silhouettes.

An IP-Adapter was used during the final stage. A visual reference was provided to guide the colour, lighting and atmosphere of the generated concept. The reference was used for visual inspiration during this educational experiment and was not presented as original project artwork.

Different styles and generation settings were also tested. This produced several variations while still maintaining the main composition created in 3ds Max.

Figure 5. The final concept-art direction after the IP-Adapter was used to influence the lighting, colour and atmosphere.

Collection and Evaluation of Evidence

The research did not use surveys, interviews or numerical data. Evidence was collected through screen recordings, generated images and visual comparisons between different stages of the workflow.

The outcomes were evaluated using the following points:

• How closely the generated image followed the original 3D composition.

• Whether the final image communicated the intended abandoned-city concept.

• How much written prompting was required.

• Whether adding more 3D information improved the outcome.

• What visual errors or inconsistencies remained.

• Whether the workflow could be useful for a 3D artist with limited drawing skills.

The evaluation was mainly based on visual observation and personal reflection. My experience as a 3D artist helped me judge the composition, scale and usefulness of the results. However, I also understand that this makes the evaluation personal and that another artist may respond differently to the same workflow.

Limitations

This was a small experiment involving one artist, one environment and one software workflow. It did not include participant testing or compare the experiences of different artists. The results may also change depending on the AI model, software settings, hardware and reference images being used.

AI-generated images also contained several visual errors and inconsistent details. The process did not produce completely finished concept art without further work. A 2D artist or 3D artist would still need to correct these issues and ensure that the final design meets the requirements of the project or client.

Because of these limitations, the findings should not be treated as a final answer for every artist or production. They only demonstrate the potential of the workflow within this specific experiment.

Creative Output

The final creative output was a short video documenting the complete process. It showed the development from the initial sketch and 3D blockout to the final AI-generated concept art.

The project helped me gain a better understanding of how research methodology can be applied to my own creative practice. It also showed that generative AI works best when it is guided by the artist’s existing knowledge and decision-making. In this case, the 3D blockout provided control over the composition, while AI helped develop the lighting, atmosphere, textures and visual details.

The workflow did not remove the need for artistic skills. Instead, it provided another way for a 3D artist to communicate and explore an idea before moving into full production.

References

Ibele, T. (2026). tyFlow (Version 1.114) [Computer software]. https://pro.tyflow.com/

Naughty Dog. (2013). The last of us (PlayStation 3 version) [Video game]. Sony Computer Entertainment.