How I created an interactive bubble system in Unity

In this article, I would like to present one of the challenges I faced while implementing a specific functionality in developing an Augmented Reality game app and the solution I devised to overcome them.

App Outline

The requirement is to develop and integrate an Augmented Reality game module into an existing native mobile app for Android and iOS platforms. The app should provide users with a library of 3D quadruped characters with unique features. The user can augment one animal character at a time to their environment and make it perform certain actions. More like a Tamagotchi concept in 3D.

Technology Stack

The stack I proposed is to use Unity as the game engine and AR Foundation by Unity as the augmented reality framework which incorporates core features from ARKit and ARCore enabling cross-platform development. The project will then be exported as a library module to integrate with the native mobile apps.

• Unity
• AR Foundation
  • ARKit (iOS)
  • ARCore (Android)

Functionality Challenge

In this app, there is a functionality requirement where the animated character model's surface should be populated with bubbles over specific regions and the user should be able to wipe the bubbles off by touch and swipe.

Demonstration on a sample model from Sketchfab, not from the original project.

If the model was static (without skeletal animations), this could be achieved by estimating approximate vertex positions and populating the bubble textures as billboard quads. However, since the mesh is animated, the populated quads should stick to the mesh surface, only on specific regions, and track relative vertex positions for each frame.

The following approaches are the Unity-provided systems to implement particle systems, but they don't pack the required features to fully implement the functionality at the time of development, and here is why.

• Legacy Particle System - This allows particles to be emitted from the mesh surface but difficult to make the particles stick to specific regions of the mesh surface and track relative vertex positions for each frame. Also found it difficult to access individual particle positions to clear based on touch position.

• Unity VFX Graph - It is possible to make the populated particles track to the mesh surface using VFX Graph but seems like only a subset of high-end mobile devices supports VFX Graph and compute shaders.

Implemented Solution

The solution I came up with is to -

1. Take a snapshot of the skinned mesh using Mesh.BakeMesh() and read the position for each vertex.
2. Use GPU instancing to draw the bubble texture quads based on updated vertex positions.
3. Use VertexColors to define bubble concentration regions, like an influence map.
4. Utilize multi-threading to execute computationally intensive and improve performance.
5. Instead of placing the bubble quads per vertex, I used barycentric coordinates to get a random point on each triangle to display the bubbles with randomness.

Capture Vertex Positions

I started with a custom class named SkinnedVertexStreamer, a component to capture the snapshot of the skinned mesh upon request and throw the response as an event once computed. This component depends on a gameObject with the SkinnedMeshRenderer.

csharp
1/// Component to stream vertex data from a SkinnedMeshRenderer component
2[RequireComponent(typeof(SkinnedMeshRenderer))]
3public class SkinnedVertexStreamer : MonoBehaviour
4{
5    ... More Lines ...
6
7    /// Recomputes the geometry data and throws the callback 
8    /// on completion (Invoke from LateUpdate for continuous update)    
9    public void RequestFrameUpdate()
10    {
11        //Get a new snap shot of the mesh as baked data
12        _MeshRenderer.BakeMesh(mBakedMesh, true);
13
14        //Get the updated vertices
15        mBakedMesh.GetVertices(BakedVertices);
16        //Get the updated normals (Uncomment if required)
17        //mBakedMesh.GetNormals(BakedNormals);
18
19        //If required to bake transform
20        if (_TransformBaked)
21        {
22            //Iterate through each vertices
23            for (int i = 0; i < VertexCount; i++)
24            {
25                //Transform position with respect to scale
26                BakedVertices[i] = transform.TransformPoint(BakedVertices[i] * _BakeScale);
27                //Transform normal (Uncomment if required)
28                //BakedNormals[i] = transform.TransformDirection(BakedNormals[i]).normalized;
29            }
30        }
31
32        //Invoke callback
33        OnVertexStreamUpdated?.Invoke(this);
34    }
35
36    ... More Lines ...
37}
38

With this, I could pass the updated vertex positions to the bubble instances. But the requirement also mentions showing the bubbles on specific regions.

Controlling influence using VertexColors

To limit the bubble instances to specific regions on the mesh surface, the character models were Vertex Painted in Blender to define the regions where the bubbles should appear.

VertexColor added using Blender

The regions painted with white are where the bubble quads will appear and the implementation will ignore the vertices painted black. The modified model files were then brought back to Unity with VertexColor attributes.

Because of the above issue, I decided to cache the VertexColor information in a required manner within a text asset. For that, I wrote a simple utility extending the Unity Editor class.

Custom wrote Editor Utility

The utility will take a skinned mesh as the 'Source Mesh' and generate a JSON with the number of painted faces as vertexColoredFaces. The binaryVertexColors is an array representing which all vertices have white color. If the value at a particular index has a numeric value of 1, then that vertex is painted white and zero if no color.

json
1{
2  "vertexColoredFaces":341,
3  "binaryVertexColors":[....,0,0,1,1,1,1,0,1,1,1,1,0,0,1,0,1,1,0,0,....]
4}
5

Required VertexColor info cached as JSON

GPU Instancing

To render multiple copies of the bubble texture efficiently, A class named BubbleInstance is created, representing a single instance of the bubble and using GPU Instancing to render them in batches.

csharp
1/// Class defining a single bubble GPU draw instance
2public class BubbleInstance
3{
4    public Vector3 Position { get; set; }
5    public Barycentric3 Barycentric { get; private set; }
6    public float Scale { get; set; }
7    public float OriginalScale { get; set; }
8
9    private readonly Transform mParent;
10
11    //Constructor
12    public BubbleInstance(float scale, Transform parent)
13    {
14        //Set a random Barycentric3            
15        Barycentric = Barycentric3.Random();
16        //Set original scale value
17        OriginalScale = scale;
18        //Set current scale as original scale
19        Scale = OriginalScale;
20        //Set reference to transform
21        mParent = parent;
22    }
23
24    //Property - Transform Matrix
25    public Matrix4x4 TransformMatrix
26    { get => Matrix4x4.TRS(Position, Quaternion.identity, mParent.lossyScale.magnitude * Scale * Vector3.one); }
27
28    //Sets the position vector appling the barycentric value
29    public void SetPositionBary(Vector3 v0, Vector3 v1, Vector3 v2)
30    {
31      Position = v0 * Barycentric.X + v1 * Barycentric.Y + v2 * Barycentric.Z;
32    }
33
34    //Restore the render data (Sets scale of this instance back to default)
35    public void Restore()
36    {
37        //Restore scale to original
38        Scale = OriginalScale;
39    }
40}
41

The class ultimately holds a transform matrix constructed from the provided barycentric position value, the parent's local scale. The rotation is set to zero since the billboard quads will be oriented towards the MainCamera through a shader.

Demonstration

To demonstrate the above implementation, let's take a look at a custom class component named BubbleSystemDemo class. Upon requesting the SkinnedVertexStreamer to process the vertex positions for the next frame, the callback is received with the updated vertex positions.

csharp
1public class BubbleSystemDemo : MonoBehaviour
2{
3  #region Inspector Fields
4  ...
5  [SerializeField] private SkinnedVertexStreamer vertexStreamer;
6  ...
7  #endregion
8
9  ... More Lines ...
10
11  private void LateUpdate()
12  {
13    //Request to get updated vertex positions
14    vertexStreamer.RequestFrameUpdate();
15  }
16
17  ... More Lines ...
18}
19

Since the mesh skinning runs on CPU for mobile devices, reading or taking snapshots of the vertex data for every frame was costing a bit of performance. So I leveraged multi-threading using C# Job System and Burst compiler for handling the vertex position computing.

The following struct represents a C# Job to process positions of the vertices with a valid vertex color and discard the ones with an invalid color.

csharp
1public class BubbleSystemDemo : MonoBehaviour
2{
3  ... More Lines ...
4
5    //Compiles immediately, not in background over time
6    [BurstCompile(CompileSynchronously = true)]
7    private struct TransformJob : IJob
8    {
9        [ReadOnly] public int totalFaces;
10        [ReadOnly] public NativeArray<int> triangles;
11        [ReadOnly] public NativeArray<Vector3> vertices;
12        [ReadOnly] public NativeArray<int> cachedVertexColIndices;
13
14        [WriteOnly] public NativeList<Vector3> transformPos;
15
16        public void Execute()
17        {
18            //Iterates through each triangle
19            for (int i = 0; i < totalFaces; i++)
20            {
21            //Get the indices forming this triangle
22            var i0 = i * 3 + 0;
23            var i1 = i * 3 + 1;
24            var i2 = i * 3 + 2;
25
26            //Get the indices of the vertices forming this triangle
27            var vi0 = triangles[i0];
28            var vi1 = triangles[i1];
29            var vi2 = triangles[i2];
30
31            //Get the vertex positions of this triangle
32            var v0 = vertices[vi0];
33            var v1 = vertices[vi1];
34            var v2 = vertices[vi2];
35
36            //Treat as valid if the vertex at target index has a valid color
37            bool v0Valid = cachedVertexColIndices[vi0] > 0;
38            bool v1Valid = cachedVertexColIndices[vi1] > 0;
39            bool v2Valid = cachedVertexColIndices[vi2] > 0;
40
41            //Discard if none of the vertex of this triangle has color
42            if (!(v0Valid && v1Valid && v2Valid))
43            continue;
44
45            // Else keep the vertex positions
46            transformPos.Add(v0);
47            transformPos.Add(v1);
48            transformPos.Add(v2);
49        }
50    }
51
52  ... More Lines ...
53}
54

Using the SkinnedVertexStreamer callback, the TransformJob is executed to get the updated positions and pass them to the transform matrix of the bubble instances to render using GPU instancing.

csharp
1public class BubbleSystemDemo : MonoBehaviour
2{
3    ... More Lines ...
4
5    /// On new vertex stream data update receieved from SkinnedVertexStreamer
6    private void OnVertexDataUpdate(SkinnedVertexStreamer streamer)
7    {
8        //Ignore if render batches are not initialzied
9        if (mBubbleBatches == null || mBubbleBatches.Count == 0)
10          return;
11
12        //Clear transformed vertex position list
13        transformVertices.Clear();
14
15        //Structure triangle and vertex information to native data types
16        NativeArray<int> _triangles = new(streamer.Triangles, Allocator.Persistent);
17        NativeArray<Vector3> _vertices = new(streamer.BakedVertices.ToArray(), Allocator.Persistent);
18
19        //Create the TransformJob (custom C# Jobsystem)
20        TransformJob job = new()
21        {
22            totalFaces = (streamer.Triangles.Length / 3),
23            triangles = _triangles,
24            vertices = _vertices,
25            cachedVertexColIndices = binaryVertexColors,
26
27            transformPos = transformVertices
28        };
29
30        //Shedule and complete the transform job
31        job.Schedule().Complete();
32
33        //Indices for batches and individual instanecs
34        int batchIndex = 0, instanceIndex = 0;
35
36        //Check if mesh has valid triangle count
37        if (transformVertices.Length % 3 == 0)
38        {
39            //Iterate for each triangle
40            for (int i = 0; i < transformVertices.Length; i += 3)
41            {
42                //Set to target current batch
43                var cBatch = mBubbleBatches[batchIndex];
44                //Set to target instance in current batch
45                var cInstance = cBatch.instanceBatch[instanceIndex];
46
47                //Get the position of the triangle vertices
48                var p0 = transformVertices[i];
49                var p1 = transformVertices[i + 1];
50                var p2 = transformVertices[i + 2];
51
52                //Pass position to the render instance to apply with pre-cached barycentric values
53                cInstance.SetPositionBary(p0, p1, p2);
54
55                //Iterate until max of this batch instances
56                if (instanceIndex < cBatch.instanceBatch.Count - 1)
57                    instanceIndex++;
58                else
59                {
60                    //Progress to next batch
61                    batchIndex++;
62                    instanceIndex = 0;
63                }
64            }
65        }
66
67        //Dispose the temporary triangle indices array and vertex position array for the frame
68        if (_triangles.IsCreated)
69          _triangles.Dispose();
70        if (_vertices.IsCreated)
71          _vertices.Dispose();
72    }
73
74    ... More Lines ...
75}
76

Finally, a custom shader is written to render the bubbles in Unlit mode with each quad facing toward the camera.

Clearing bubbles with touch-swipe

I'm not elaborating on how I did the feature to clear the bubble instances by touch-swipe. But here is a gist of how it is implemented -

• The character model is set up with multiple colliders tracking the skeleton and confining it to its volume.
• By performing Raycast from touch point towards world-space and scaling down the bubble instances which are closer to the ray hit point by a certain threshold.

Hope this info is helpful to someone. Cheers!🍻