Let’s Take a Brick

We all know what LEGO is—it is a timeless, fun and educational toy for a kid, one he or she played with as a child, and now probably steps on when walking around the house as a parent.

But the charm of LEGO knows no bounds. In fact, more and more adults nowadays show an increased interest in tinkering with these brightly hued bricks, and to Prof. Philip Fu, LEGO is not just a child’s play.

‘Actually, LEGO is a profound subject that should not be trifled with,’ said Professor Fu of the Department of Computer Science and Engineering.

As a LEGO enthusiast, Professor Fu has tried hard to tap into the hidden potential of these tiny little bricks and broaden their applications. ‘The LEGO Technic system, introduced as an expert series for building advanced 3D models in 1977, allows users to build frame-based architectural structures. Some enthusiasts have even made use of such system to create a wide variety of customizable robots. I, too want to make something out of LEGO Technic.’

There is one salient obstacle, though. Professor Fu said, ‘Advanced architecture often involves complex assembly mechanisms, like bicycles and Ferris wheels which require ample space for rotation. Therefore, when you start building something from scratch, it takes a lot of time and effort to build a model with structural integrity.’

After years of committed research, Professor Fu and his team, in collaboration with Simon Fraser University, finally came up with an effective solution—thanks to their sophisticated computational algorithm, the world’s first-of-its-kind, many hand-drawn architectural sketches can now be optimally modelled through automation.

‘Our algorithm can come up with a simple yet structurally-sound model in only a few seconds, along with videos and images teaching you how to assemble the model with the LEGO bricks,’ Professor Fu proudly proclaimed.

Here is how the algorithm works: users will first sketch their designs in Graphical User Interface, a computational tool, then AI—artificial intelligence in short—of the algorithm will produce an initial layout and subsequently fine-tune the result. Finally, an assembly instruction will be generated to indicate to users the possible ways for construction.

‘AI unquestionably plays a vital part in our algorithm,’ said Professor Fu. He explained that one of the biggest features in the project relates to the internal visualization of user’s sketches. ‘As we all know, computers do not understand human language other than zeroes and ones. By the same token, input sketches have to be converted into something that is comprehensible to the AI.’

In hopes of overcoming the aforementioned compatibility issues, Professor Fu and his team have devised a computational model for the algorithm to work on the sketches. ‘We have studied the brick properties from existing LEGO models and contemplated all kinds of connection mechanisms. These data are then written as programming codes and fed into the model. As a result, the algorithm can construct a guiding layout graph each time a user draws a sketch.’

The most challenging part, as the professor recalled, occurs at the fine-tuning stage. ‘A structure can have hundreds and thousands of possible combinations of structural orientations and diversities. On top of that, using LEGO to craft 3D designs has to make use of pins and connectors to allow joint rotations. All these add further difficulty in ensuring architectural stability, while at the same time remaining faithful to the input design.’

The solution? Mathematics. ‘Through many rounds of trial-and-error, the AI in the algorithm will have a general idea on the probability of success on objective factors such as “structural integrity”, “model simplicity” and “adherence to sketch”. The calibration and ultimately the assembly instruction are the ones which have the highest probability of success amongst all those factors.’

To make it easier for understanding, Professor Fu gave the following analogy: imagine you are shopping in a mall hoping to buy a certain kind of product with the lowest price. After browsing through dozens and dozens of shops, you will roughly know the price of that product and probably also the kinds of shops that offer the best value for money. Similarly, by running simulations continuously through the algorithm, the AI can estimate when to use a specific brick for certain orientation or result, therefore rendering accuracy to the fine-tuning of the layout.

With Professor Fu’s AI-powered algorithm, a rich variety of LEGO models can now be automatically generated in the blink of an eye. ‘You can basically use our algorithm to make all kinds of things. Want to craft something fine and minute like a kite or a picker? Check. Something big and sturdy like a bridge or a castle? Also check.’ He also remarked that for those massive models that are made up of thousands of bricks, the entire process, ranging from sketching to final assembly, can be completed in only a few minutes.

Funded by the Hong Kong Research Grants Council and featured in SIGGRAPH Asia Technical Papers, Professor Fu’s AI-powered algorithm can be extended not only to all kinds of interlocking blocks, but also intricate architectures and mechanical designs as well.

As for future plans, the computer engineer wishes to expand the existing algorithm in a number of ways. ‘For one, machine learning is a direction we are heading towards. Together with the help from mechanical and automation engineering, we aim to develop our algorithm such that it can learn from existing structures and assemble new designs on its own.’

By ronaldluk@cuhkcontents

LEGO is a trademark of the LEGO Group, which does not sponsor or endorse this site.

This article was originally published on CUHK Homepage in Aug 2020.

Tags
Philip Fu LEGO Artificial Intelligence robot Department of Computer Science and Engineering