Sceneform AR physics experiment

Jul 4, 2020

One of the Sceneform example applications demonstrates the image recognition capability (which I use in my AR Map application to activate the augmented classroom billboards) by augmenting a green maze structure onto a sheet of paper from a notebook and placing an augmented red ball into the maze as well. After that, the user can tilt the notebook along with the augmented maze to cause the ball to travel inside the maze structure. This is the augmented reality equivalent of toys like these.

That code example uses the JBullet physics engine for the physical simulation portion of the augmented reality: gravity, collision detection of the ball and the maze wall pieces, friction, restitution. The ball has to be affected by gravity so that it moves inside the maze, and the maze wall structure has to be constructed along with a bottom plane that prevents the ball from falling out of the scene. The maze has to follow the orientation of the notebook.

I was thinking of a good scenario for an AR physics experiment and I remembered a real-time CUDA GPGPU demo many years ago where the user threw metal balls against a wooden KEVA plank tower structure. It was something like this; incidentally, in this YouTube video the physics engine is also Bullet. Another example of complex KEVA plank tower ball throw simulation could be this.

Here is the resulting app’s android store listing: https://play.google.com/store/apps/details?id=dev.csaba.arphysics.

Here is a demo video showing the ARPhysics app in action:

This browser does not support the YouTube video player. Watch on YouTube

Demo video of the Sceneform ARPhysics application.

Bullet is a native C library, while JBullet (cz.advel.jbullet:jbullet:20101010) is a Java implementation, which can certainly have performance consequences. As you can see, the current tower structure is very simple because I’m just exploring the limitations. Regardless, I made the height of the tower (number of plank levels) configurable along with many physical properties such as density, restitution, friction, and gravity. I knew I’d run into interesting things, and the project didn’t disappoint me. I already wrote about how I hit a brick wall when I tried to rekindle with existing Sceneform projects, which set me back a week. Once I jumped through that hoop and reached the point of physics simulation, several issues emerged:

  1. Originally, I enabled the simulation right after the tower structure was placed on a surface. However, the structure immediately "exploded". I felt that I probably placed the planks too close to each other and they repelled each other as if they were magnets of the same polarity. So first I artificially spaced them out so they’d have to fall a little bit. At that point, I discovered that in an equilibrium state they were visibly hovering above each other, not touching. This is because there’s an artificial margin around the box collision shapes: the engine uses that to test collision situations, and it’s a very important parameter. The default is 0.03, which in Bullet / JBullet is, I think, supposed to be in centimeters (?). However, since the Sceneform and ARCore standard dimension is in meters, the planks were hovering with a gap of 1–2 centimeters above each other. So first I decreased the margin to 0.0025 (2.5 millimeters), and then I added logic to the PhysicsController layer (the layer between JBullet and my AR Scene) to artificially remove that margin from the AR scene boxes unbeknownst to ARCore. This way the physics engine is happy and the AR Scene won’t see any gaps (even if they are as little as 2.5 mm) either. These techniques decreased the initial pop of the structure, but didn’t eliminate it completely. The sphere collision shape doesn’t require this treatment.
  2. Besides the initial bounce, there is another annoying phenomenon: if I turn on the simulation right after placing the tower, the whole structure is jittery, the planks are restless, and the tower slowly slides apart. This is possibly a side effect of rounding errors and imperfections in the simulation, and the engine needs to use countermeasures to soothe the elements of the simulation. The engine can use heuristics to deactivate items to avoid this. Also, there’s a parallel concept of "sleeping" items. One can specify the threshold levels for sleeping: ballRB.setSleepingThresholds(0.8f, 1.0f);. When an item goes into a sleeping state, it can still wake up when another body kinetically pushes or interacts with it.
  3. The simulation itself seems to be very limited: the planks don’t really rotate. It’s either the limitation of the cz.advel.jbullet:jbullet:20101010 Java port or I need to enable something in the engine (use a different algorithm?).

I have several future plans lined up:

  1. Use the Sceneform-Bullet physics engine. This is also a Java derivative of Bullet, but it embeds Bullet in its C implementation as an NDK native library. This requires compilation of the native part, and not so long ago the NDK dropped GCC from the SDK, which is a huge headache for my long-time VR project and I predict will cause problems when compiling Sceneform-Bullet as well for sure.
  2. Another engine I would like to integrate as an option is JMonkey: it does have a Bullet core, but it also has its own engine implementation which I’m very curious about. It seems that integrating either JMonkey or Sceneform-Bullet will require major refactoring of how the physics layer interacts with the AR Scene layer. Currently, the AR Scene layer hands over AR Nodes to the physics layer, and lets the physics layer move and rotate the AR objects according to the physics simulation. This is a tighter coupling than a full-fledged, well-defined API.
  3. I also want to integrate other scenarios. For example, I could randomly scatter a bunch of planks bounded by a rectangular 1-meter-by-1-meter box. Then place an extra giant "hockey puck" which would be a Sceneform TransformableNode. The user would be able to grab that hockey puck and yank it around, stirring the objects within the simulation box. On the Bullet physics end, the puck would be a so-called "Kinematic object", so the physics engine would not be able to move it, but the controller layer would move it based on the user’s gestures. The goal of the concept would be something like what is seen here (source code here).
Comments loading...