Should I consider a startup in 3D content creation, Metaverse?
The startups and ideas ecosystem in the 3d content generation, artists assistance tools landscape.
Introduction
First off the base: drawing is hard. Drawing in 2 dimensions is what we are used to do growing up. Pencil at hand, trying to sketch a dog — our dog. The first results? Oh well…
Fast forward some years, you wonder how some artists can draw a dog with just 2–3 lines…
Even as an adult, some of us can do this with ease, but most others… well… not so.
Fast forward a few more years, and you are considering a startup in the space of 3D design tools, 3D content generation, Metaverse. What the heck is that?
Growing up I remember seeing the first 3D video game on Commodore 64.
It was very crude, but I remember what really captured my 12 years-old attention: someone told me that the computer was calculating all of these 3D shapes in real time, modifying them at every frame. At an early age I was somehow attracted to computers. When my father convinced my grandma to use her monthly pension to buy a C-64 for me, it opened a new world. I remember I was telling my friends, soccer enthusiast and avid players of the C-64 game “Emlyn Hughes International Soccer”, that one day even their dear soccer game was going to be made in 3D! They laughed and told me I was crazy — how can that crappy looking wireframes morph into a beautiful soccer game? Well fast forward a few years and we got “Actua Soccer” — the first 3D soccer game…
With the Sony Playstation and better graphic processors, 3D became the norm in video games and in the 2000s progressed to full 3D movies. Why? Because at some point is it harder to draw everything in 2D than it is to create 3D shapes and render them, so you can see them in every perspective without having to manually re-draw them.
The Metaverse days
Today much has changed from my early days of 3D of the 1980s. 3D content is everywhere and it can even be used for full hollywood movies. In video-games it is so realistic that you get motion sickness! In fact Virtual Reality (VR) has also opened the door to more desire to create 3D scenes and environments, sculpting and arranging items as if we were in real life.
The desire of people to stay indoor and refuge in synthetic worlds is growing. Pandemics or not, it was already growing.
And it is only going to get more pervasive
What has not changed much is that creating this content is still hard, despite the enormous amount of code and tools we have today.
Today, if a casual user wants to build a 3D scene in 30 minutes, well they can’t!
Drawing in 3D
Despite the great success of 3D, how many of you have ever actually tried to make a 3D drawing with a computer? Yes there are many tools available, like Blender, Cinema 4D, 3ds MAX, Sketchup, Maya, Mixamo, Houdini… and many more — but they all have a very steep learning curve, and unless you use them on a daily basis you will never be able to master even the simplest combination of capabilities and shortcuts they offer.
Drawing in 3D is hard
very hard… beyond comprehension if you have not tried. You basically have to specify the position of each and every 3D point in your meshes. Try to draw a real car for example one day, maybe using Blender.
Of course you do not have to draw everything. If you are happy with simple modifications, you can take already drawn assets from one of many 3D asset store, like https://free3d.com. Here you can find many model from simple to more complex, from free to $ to $$$.
But can you make a game or a movie this way? Maybe not — because all assets will mismatch in style and characteristics, making the scene not very visually compelling. Some modification will be needed. Some may be easy, like changing the color of a flat material, but if you want to provide complex textures and nicely touch-up colors in many parts of a mesh you will soon find yourself having to learn one the big name tools.
If you are an artist or a person that works with these tools on a daily basis, things are of course easier, you can remember what to do. You are paid (or strive to) to learn these tools and get better at it. There are pain points of course. You still have to draw all the 3D points in a mesh, have to provide colors and textures to every part of your asset, and maybe you have to animate it also. Gets difficult very quickly.
Designing a 3D game, scene, level
A 3D item will just sit pretty on your computer if you do not use it. Maybe you want to make a game or a illustrate a scene or make a short powerpoint-like movie animation.
At the basic scene / level 3D design involves these steps:
- storytelling and scripting
- design of 3D meshes, items, landscapes, skybox, etc
- embellish meshes and assets with materials and textures
- scene and levels composition
- animation of some meshes
- coding of interactions, game logic
Even if you just wanted to illustrate a dream you had last night, while the story is clear in your mind, finding the rights 3D assets and animating is not yet open to the public.
Game and scene logic (when I open the cupboard, a ghost appeared), can be implemented in Scratch-like visual programming that is easy to learn.
In general, ask anyone in the field and they will tell you that the most time and resources spent in 3D design is in selecting and curating individual assets, characters, scene items, lightning, animations. And most of these steps today are the domain of experts only.
Would you want to create a startup in the 3D space / Metaverse?
The issue remains: 3D design is hard and not open to the public yet. Young adults and kids would like to build games in Roblox, for example, but can’t because it is hard to design and hard to code. A tool that allows for easy scene creation and for simple dynamics is definitely needed!
If you are considering a startup in this space, I recommend you consider these areas:
- 3D tools that are easy-to-use: tools like TinkerCAD are easy to use and should be the standard entry point to any 3D design tool aimed at the general public
- Create 3D scenes out of existing assets: there are many websites colleting community-created 3D assets. These make the life of casual user easier, but do not provide a consistent look and feel for implementing an entire scene. These assets also mismatch in size, texture definition, polygons count. There is a need for a centralized database of 3D assets like Kenney.
- Create 3D assets automatically: the Holy Grail is to be able to design an asset with just text: “a blue coffee mug that look like Lego”. A bit of a CLIP / DALL-E for 3D. We may be getting closer to this today, but the way is still long before we can provide consistent style, and fine-detail programmability that artists would want. Yet it may suffice for some casual users. One of the issue is that possibly specifying each 3D point with a neural network is overkill ad ill-posed, as there are much fewer control parameters in assets that one would want to vary.
- Extend procedural 3D assets: the other way, that is between using fixed assets and neural network created assets, is to use procedural assets that are hierachical. These can be design with less parameters and even graphically (Geometry Nodes and Textures in Blender) and offer a hierarchy as a bonus. They may be just the perfect way to set up a large database.
- Placing items in a scene: once you have all your assets for your scene, the next step is where to place them. Specify rules, script arrangements, or use a neural network placer?
- Logic and animation: the ultimate step in 3D (and more) scene design is to be able to specify scene logic, interactions and animations purely with natural language. Today all this is done with computer code, but graphical and visual programming with blocks really helps (Scratch, Dreams, Garage Game Builder). Animations are more complicated and will require tools that can transfer them from a video to a character, or from a character to another character with different bone structure.
Want more?… the future is all yours.
Who are you and your customer?
The first question if you area considering a startup in this space is who are you and who is your customer (as usual…).
You: let’s say your goal is to make 3D design easier and less painful. Your experience in your personal 3D struggles and the pain-points of the industry can guide you to the tools that are most needed.
And who is your customer or target audience:
Artists— You will need to develop professional tools or most likely extensions to existing tools in existing ecosystems. At first, you may only target one tool ecosystem because of compatibility, licensing and deals. You may be more restricted to what artists want, which is usually a projection of the existing tools and the rigid scheme they end up imposing. Artists are usually loking for tools that remove monotonous tasks, give them ways to enhance their creativity, construct higher level abstractions and integrate in their existing framework.
Casual Users — Visual quality may be more forgiving, but simplicity of use will win — see: Canva. Casual users want to quickly learn to design ideas, are not so concerned with fancy graphics, but want to learn fast and realize their creation with little effort and a easy learning curve. You can develop new tools that are free from established ecosystems, and that may enhance collaboration and sharing of resources more widely than professional tools, with lower costs and wider distribution of platforms and user-base. Most importantly, you can open up a larger user base, like young teen or pre-teen artist that want to make their own 3D game or movie, or just illustrate ideas.
Today opening 3D tools to casual users has the potential to be the most lucrative and exciting option. The reason is Canva. The question is: will there ever be as many people that want to design 3D scenes as the ones that want to design a birthday card?
What is out there?
The 3D design and artists tools environment is large and wide, and it is always expanding. Here we list some of the existing tools today.
Professional tools: professional tools offer all the weapons for the basic 3D steps of design listed above. Blender is largely favorite of the masses because it is free and open source. It is a powerful mesh design tool that can also do materials and animations. Professional tools have one or more specialty. An overview is here: https://all3dp.com/1/best-free-3d-modeling-software-3d-cad-3d-design-software/
Then there are game creation tools, movie creation tools, etc. In terms of video games Unreal and Unity dominate. They take the assets and animation and compose game logic, levels and all the user interfaces. Most large video-games today are made with these tools.
Panda3d, threeJS, BabylonJS are game-developement tools designed for python and web browsers. They can develop entire video games using scripting languages. They are generally aimed at smaller video games and for learning to code and design games and game logic.
More casual tools:
For game logic, I recommend looking into https://scratch.mit.edu which allows simple 2D games and animations with no code.
For easy 3D drawings, I recommend looking at https://www.tinkercad.com.
New tools / startups:
https://recroom.com is designed to be an interactive tool for VR and social interactions
https://www.shapesxr.com is a tool to build 3D scenes in VR with limited machine learning usage
https://www.prometheanai.com scene composition with voice, no-code, similar to Anything.world
https://anything.world scene composition with voice, no-code, assets built-in, animation-ready, entire scenes ready (park, beach), can add items, items know by default where they should land
https://dotbigbang.com is an easy to design 3d games tool with limited machine learning usage
https://decentraland.org 3d design tool on web, with shared assets and community
https://www.shapesxr.com 3d design in VR (like Dreams on PS4)
https://readyplayer.me transfer your face to an avatar
https://www.aquifermotion.com makes animated 3D video for marketing, transfer motion and face tracking to 3D characters
Game making ecosystems
Unity and Unreal are the top game making frameworks, including all steps required to make a video-game or a digital 3D production. Needless to say they are a professional tool for professional coders and require entire university courses to master. Not useful for casual users!
On the other had casual users have some other simpler options if they want to make a game:
Roblox — need programming experience, young users cannot easily make new games
Minecraft — block-based, restricted to own creative game types
Dreams, PS4 — can build small games with no-code, but stuck in platform
Garage game Builder, Nintendo — can build small games with no-code, but stuck in platform
more…
References
Here are some reference I found useful to write this and survey the field:
AI research tools
GAN for graphics:
Generative Adversarial Networks - The Story So Far
Generative adversarial networks (GANs) have been the go-to state of the art algorithm to image generation in the last…
blog.floydhub.com
InfinityGAN: Towards Infinite-Pixel Image Synthesis
We present a novel framework, InfinityGAN, for arbitrary-sized image generation. The task is associated with several…
hubert0527.github.io
2d images to 3d meshes:
GitHub - shunsukesaito/PIFu: This repository contains the code for the paper “PIFu: Pixel-Aligned…
News: [2020/05/04] Added EGL rendering option for training data generation. Now you can create your own training data…
github.com
GitHub - lingtengqiu/Open-PIFuhd: This is an implementation of PIFuhd based on Pytorch
This is a unofficial implementation of PIFuhd PIFuHD: Multi-Level Pixel-Aligned Implicit Function forHigh-Resolution 3D…
github.com
Is It Possible to Combine Deep Learning And Reinforcement Learning For 3D Modeling
Being very interested in 3D animations I always try to find ways that might help to automate or at least cut some edges…
medium.com
3D-R2N2: Recurrent Reconstruction Neural Network
This repository contains the source codes for the paper Choy et al., 3D-R2N2: A Unified Approach for Single and…
3d-r2n2.stanford.edu
NERF
Tree generation
SpeedTree - 3D Vegetation Modeling and Middleware
New features and licensing options are available now!
store.speedtree.com
Neural materials:
https://arxiv.org/pdf/2104.02789.pdf
Scene and placement:
Sceneformer: Indoor Scene Generation with Transformers
Sceneformer: Indoor Scene Generation with Transformers 3DV 2021 Oral We address the task of indoor scene generation by…
xinpeng-wang.github.io
Images and caption
DALL·E 2
Thanks to those who helped with and provided feedback on this release: Sandhini Agarwal, Sam Altman, Chester Cho, Peter…
openai.com
Ai-based video generation
Synthesia - AI Video Generation Platform
Create AI videos by simply typing in text. Make engaging videos for e-learning, customer onboarding, etc. No need for…
www.synthesia.io
No-code tools:
https://developers.google.com/blockly
Rete.js
Edit description
rete.js.org
Nodes - a new way to create with code | https://nodes.io
What if programming was about ideas, not semicolons? Compose, abstract, generalise. Start from top down or bottom up…
nodes.io
https://github.com/jagenjo/litegraph.js
Making games
if you ever want to learn video game development this is one nice resource to start:
Scratchapixel
@ 2009-2022 Scratchapixel (All rights reserved) This website and its content is copyright of Scratchapixel. The source…
www.scratchapixel.com
https://scratch.mit.edu — learning to code for games and simple animations and interactiosn in 2D
https://www.tinkercad.com — easy 3D drawing for 3D printers etc.
About the author
I have almost 20 years of experience in neural networks in both hardware and software (a rare combination). See about me here: Medium, webpage, Scholar, LinkedIn, and more…
If you found this article useful, please consider a donation to support more tutorials and blogs. Any contribution can make a difference!