How is Computer vision helping aid the design project ?  

The idea of using computer vision was to integrate it in a step where the core of the project lay in and where no other  solution could replicate it as accurately as a computer model could, using machine learning and CV. The idea was  to integrate Computer vision in procurement of circular wood where the computer would help identify defects in the  wood and demarcate out the unusable parts that would hinder with the structural integrity of the usable wood to cre

ate beams. The project would be split into two primary tasks of defect identification and coordinate generation of the  usable wood.

Imperfections and failure points are identified and catalogued.

Bounding boxes around the points drawn.

Imperfections and failure points are identified and catalogued.

Bounding boxes around the points drawn.

Pipeline of the task from the input to the output

A very linear pipeline of object identification, defect detection and boundary extraction would be followed to make use  of the output data to further use them in the design process. The inputs would have to be images of circular wooden  boards on a conveyor belt scanned using a line camera and synchronised using a pulse encoder. Once the CV is  able to extract the size of the board, a trained model would then identify the defects demarcating them with boundary  boxes. These would then be converted to text/coordinate data that would then be used within Grasshopper in Rhino,  to generate a convenient pattern for cutting the usable wood.

What data is needed ? How is it annotated and are there datasets available ?

The primary data needed are of a pretrained model that are annotated or that already have semantic maps for. A reli able dataset consisting of 4000 large scale image dataset of wood surface defects was found that could be used. The  drawback being it belonged to a singular species of wood which reduced species entropy when used in a practical  sense for scanning and defect identification. The dataset would ideally require more heterogeneity to become practical  which could be done through self annotating examples and then augmenting them.

Software - what softwares are being used and are there off the shelf models that can be used ?

An early thought of using UNet or faster RCNN for the purpose was identified which was quickly replaced by the yolov5  model written in the Pytorch framework; That can perform much faster compared to the preceding models, especially  within a facility of moving parts on a conveyor belt. The off-the-shelf yolov5 is pre-trained on the COCO dataset and  trains quickly and is able to generate inferences and results efficiently. The coordinate output data would then be  transferred into grasshopper for further developing them into a design and using human-robot interaction to cut out the  wooden beams from the circular wood.