Snippet Wednesday - Images in a GeometryView

Hi everyone,

For this weeks snippet Wednesday (yes, it’s Thursday, sorry about that), I thought I’d share a nice possibility to add images to our GeometryView.

The Goal
Add images (such as maps) to my GeometryView. For instance; here I combined the map of Rotterdam around the VIKTOR office with 3D data!

The Solution

VIKTOR geometries are based upon the trimesh library. So as a plan of attack, let’s create a trimesh plane ourselves, add a texture and then add it to the VIKTOR geometry group.

Creating the trimesh plane
First well create the trimesh plane, and add a texture to it.

import numpy as np
import trimesh

...

@GeometryView("3D", duration_guess=1)
def image_in_geometryresult(self, params, **kwargs):
    # Create a simple plane with a given size
    width = 10
    height = 10
    plane_vertices = np.array([
        [-width / 2, -height / 2, 0],  # bottom-left
        [width / 2, -height / 2, 0],  # bottom-right
        [width / 2, height / 2, 0],  # top-right
        [-width / 2, height / 2, 0]  # top-left
    ])

    # Define face indices (two triangles for a quad)
    plane_faces = np.array([
        [0, 1, 2],  # first triangle
        [0, 2, 3]  # second triangle
    ])

    # Define texture coordinates (UV mapping)
    texture_coords = np.array([
        [0, 0],  # bottom-left
        [1, 0],  # bottom-right
        [1, 1],  # top-right
        [0, 1]  # top-left
    ])

    # Create a Trimesh object for the plane
    plane_mesh = trimesh.Trimesh(vertices=plane_vertices, faces=plane_faces)
    im = Image.open("map.png")
    color_visuals = trimesh.visual.TextureVisuals(uv=texture_coords, image=im)
    plane_mesh.visual = color_visuals

    # Assign the texture coordinates (store as a vertex attribute)
    plane_mesh.visual.uv = texture_coords

    # Create a scene with the mesh
    scene = trimesh.Scene([plane_mesh])

Creating a GeometryResult
We now have a trimesh.Scene. We can directly load the scene into a GeometryResult as following:

from trimesh.exchange.gltf import export_glb

   ...
   glb_data = export_glb(scene)
   return GeometryResult(File().from_data(glb_data))

Combining the textured plane with VIKTOR Geometry
Say we have some ViktorGeometry, and want to combine it with our previously created Scene. Luckily all VIKTOR geometry elements have a (hidden) function called _to_trimesh(). This converts the VIKTOR geometry to a trimesh scene, which we can then combine together. The code for that looks like this:

    ...
    # Our previously defined plane-mesh-scene
    scene = trimesh.Scene([plane_mesh])

    # Create a new scene from a SquareBeam
    box_scene = SquareBeam(1,1,1)._to_trimesh()
    for geometry_name, geometry in scene.geometry.items():
        # Add the plane_mesh to our viktor scene
        box_scene.add_geometry(geometry)

    # Export the scene as GLTF
    glb_data = export_glb(box_scene)
    return GeometryResult(File().from_data(glb_data))

This should now create a mesh with the ‘map.png’ image, and a Cube on top of it! To create the image on the top, we have combined OpenStreetMaps with 3DBag to create this nice image!

Hopefully this inspires you to know that there is always a way to hack yourself around the Viktor geometry toolbox, and create the beautiful visuals you need!! :framed_picture:

Full example code

Complete Code
import trimesh
import numpy as np
from trimesh.exchange.gltf import export_glb

@GeometryView("Testing an image in 3D", duration_guess=1)
def image_in_geometry_result(self, params, **kwargs):
    # Create a simple plane (e.g., 10x10 in size)
    width = 10
    height = 10
    plane_vertices = np.array([
        [-width / 2, -height / 2, 0],  # bottom-left
        [width / 2, -height / 2, 0],  # bottom-right
        [width / 2, height / 2, 0],  # top-right
        [-width / 2, height / 2, 0]  # top-left
    ])

    # Define face indices (two triangles for a quad)
    plane_faces = np.array([
        [0, 1, 2],  # first triangle
        [0, 2, 3]  # second triangle
    ])

    # Define texture coordinates (UV mapping)
    texture_coords = np.array([
        [0, 0],  # bottom-left
        [1, 0],  # bottom-right
        [1, 1],  # top-right
        [0, 1]  # top-left
    ])

    # Create a Trimesh object for the plane
    plane_mesh = trimesh.Trimesh(vertices=plane_vertices, faces=plane_faces)
    im = Image.open("map.png")
    color_visuals = TextureVisuals(uv=texture_coords, image=im)
    plane_mesh.visual = color_visuals

    # Assign the texture coordinates (store as a vertex attribute)
    plane_mesh.visual.uv = texture_coords

    # Create a scene with the mesh
    scene = trimesh.Scene([plane_mesh])

    # Create a scene from a SquareBeam
    box_scene = SquareBeam(1,1,1)._to_trimesh()
    for geometry_name, geometry in scene.geometry.items():
        # Add the geometry
        box_scene.add_geometry(geometry)

    # Export the scene as GLTF
    glb_data = export_glb(box_scene)
    return GeometryResult(File().from_data(glb_data))

1 Like

Hi Rick,

Thanks for sharing this possibility, it is one of the foundations of a new app that I’m proposing currently.

I’ve had to change some bits here and there and I got mine to work like yours:

Now I’d like to add to that the possibility to use a user specified image using FileField

class Parametrization(vkt.Parametrization):
...
    tab_files.image_file_upload = FileField(
         ui_name="Upload an image file",
         file_types=['.jpg', '.jpeg', '.png']

And work with this in the controller. However, the current flow doesn’t seem to recognize an uploaded file as a png / jpg / jpeg I think, because with the following code:

class Controller(vkt.Controller):
    parametrization = Parametrization

    @GeometryView("3D model with reference", duration_guess=1, x_axis_to_right=True)
    def image_in_geometry_result(self, params, **kwargs):
...
        image_file = params.tab_files.image_file_upload.file
        color_visuals = trimesh.visual.texture.TextureVisuals(uv=texture_coords, image=image_file)

I get no error message, just a grey plane where an image should be.

For the static image I found it is important to open the image with:

        im = Image.open(image_file)

and then:

        color_visuals = trimesh.visual.texture.TextureVisuals(uv=texture_coords, image=im)

But this gives me an error:

2024-10-30 15:32:08.180 INFO    : Job (uid: 1044768) received - EntityType: Controller - call: image_in_geometry_result
2024-10-30 15:32:08.379 ERROR   : Exception is raised
Traceback (most recent call last):
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\PIL\Image.py", line 3475, in open
    fp.seek(0)
    ^^^^^^^
AttributeError: 'File' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "viktor_connector\\connector.pyx", line 298, in connector.Job.execute
  File "viktor\\core.pyx", line 2055, in viktor.core._handle_job
  File "viktor\\core.pyx", line 2038, in viktor.core._handle_job._handle_view
  File "viktor\\views.pyx", line 2518, in viktor.views.View._wrapper
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\app.py", line 336, in image_in_geometry_result
    im = Image.open(image_file)
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\PIL\Image.py", line 3477, in open
    fp = io.BytesIO(fp.read())
                    ^^^^^^^
AttributeError: 'File' object has no attribute 'read'

I’m 100% sure I’m missing something basic here but I couldn’t figure out what exactly it is and where to find a reference in the docs.

Could you assist with this? Thanks!!

@mslootweg thanks for pointing me to this code snippet!

Hi @Michael684,

This should be the way to load the image

with params.filefield.file.open_binary() as fp:
    image = PIL.Image.open(fp)

No luck unfortunately, see error message at the bottom.

The weird thing is that if I switch this around with a static image, that looks to be exactly the same, it works just fine

with params.tab_files.image_file_upload.file.open_binary() as fp:
    im = PIL.Image.open(fp)
print("FileField file (doesn't work): ")
print(im)

im2 = Image.open("3DViewLayer.png")
print("static image file (work fine):")
print(im2)

With this I get in the terminal:

FileField file (doesn't work): 
<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=2480x3307 at 0x232A0DB2510>
static image file (work fine):
<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=2480x3307 at 0x232A0DE0CD0>

Error message:

2024-10-31 07:20:17.466 INFO    : Job (uid: 1046050) received - EntityType: Controller - call: image_in_geometry_result
2024-10-31 07:20:18.069 ERROR   : Exception is raised
Traceback (most recent call last):
  File "viktor_connector\\connector.pyx", line 298, in connector.Job.execute
  File "viktor\\core.pyx", line 2055, in viktor.core._handle_job
  File "viktor\\core.pyx", line 2038, in viktor.core._handle_job._handle_view
  File "viktor\\views.pyx", line 2518, in viktor.views.View._wrapper
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\app.py", line 355, in image_in_geometry_result
    glb_data = export_glb(box_scene)
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\trimesh\exchange\gltf.py", line 204, in export_glb
    tree, buffer_items = _create_gltf_structure(
                         ~~~~~~~~~~~~~~~~~~~~~~^
        scene=scene,
        ^^^^^^^^^^^^
    ...<3 lines>...
        extension_webp=extension_webp,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\trimesh\exchange\gltf.py", line 714, in _create_gltf_structure
    _append_mesh(
    ~~~~~~~~~~~~^
        mesh=geometry,
        ^^^^^^^^^^^^^^
    ...<6 lines>...
        extension_webp=extension_webp,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\trimesh\exchange\gltf.py", line 900, in _append_mesh
    current_material = _append_material(
        mat=mesh.visual.material,
    ...<3 lines>...
        extension_webp=extension_webp,
    )
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\trimesh\exchange\gltf.py", line 1971, in _append_material
    hashed = hash(mat)
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\trimesh\visual\material.py", line 162, in __hash__
    hashed = hash(self.image.tobytes())
                  ~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\PIL\Image.py", line 805, in tobytes
    self.load()
    ~~~~~~~~~^^
  File "C:\Users\m.vantelgen\viktor-apps\poc-floor-editor\venv\Lib\site-packages\PIL\ImageFile.py", line 267, in load
    seek(offset)
    ~~~~^^^^^^^^
  File "viktor\\core.pyx", line 811, in viktor.core._BinaryURLFile.seek
  File "viktor\\core.pyx", line 717, in viktor.core._ResponseStream.seek
  File "viktor\\core.pyx", line 726, in viktor.core._ResponseStream._load_until
  File "viktor\\core.pyx", line 515, in viktor.core._File.tell
ValueError: I/O operation on closed file

Hi @Michael684,

export_glb apparently also loads in the image. So the file should be still opened, or saved somewhere in memory.

So this works:

        ....
        # Create a Trimesh object for the plane
        plane_mesh = trimesh.Trimesh(vertices=plane_vertices, faces=plane_faces)
        with params.image_file_upload.file.open_binary() as fp:
            im = Image.open(fp)
            color_visuals = TextureVisuals(uv=texture_coords, image=im)
            plane_mesh.visual = color_visuals

            # Assign the texture coordinates (store as a vertex attribute)
            plane_mesh.visual.uv = texture_coords

            # Create a scene with the mesh
            scene = trimesh.Scene([plane_mesh])

            # Create a scene from a SquareBeam
            box_scene = SquareBeam(1, 1, 1)._to_trimesh()
            for geometry_name, geometry in scene.geometry.items():
                # Add the geometry
                box_scene.add_geometry(geometry)

            # Export the scene as GLTF
            glb_data = export_glb(box_scene)
            return GeometryResult(File().from_data(glb_data))

1 Like

That solved the issue. Thanks!