1 of 51

Lane Detection �Progress Report

Daniel Feng

2 of 51

YOLOP

Wu, Liao, Zhang, etc. YOLOP

3 of 51

LaneNet (Enet based)

Xu, lanenet-lane-detection-pytorch

4 of 51

Robust Lane Detection From Continuous Driving Scenes

Q. Zou, H. Jiang, Q. Dai, Y. Yue, L. Chen and Q. Wang, "Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks” (GitHub)

5 of 51

Robust Lane Detection From Continuous Driving Scenes

Q. Zou, H. Jiang, Q. Dai, Y. Yue, L. Chen and Q. Wang, "Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks” (GitHub)

6 of 51

Bad result from downsampling�Sliding window?

Fixed!

7 of 51

Fail to detect horizontal stop line

8 of 51

Fail to detect horizontal stop line

9 of 51

Model memorize lane direction�Rotation? Performance is not ideal

10 of 51

Specifically, faraway stop line ignored�model remember the thickness?

11 of 51

Results after implementing the code

12 of 51

Apply Sequential RANSAC

Dasgupta, Sequential RANSAC approach to find all straight lines in an image

​

13 of 51

Results

14 of 51

Video Results

After RANSAC

Segmentation result

15 of 51

Task: Speed up calculation�(current 1.25s per frame)

16 of 51

Attempt 1:�KLT Tracking (sparse optical flow)

17 of 51

Attempt 2:�Downsampling image

18 of 51

Achilles heel:�Model inference speed

  • Change model back to previous model: HybridNets (1.07s per frame)

19 of 51

ONNX optimization

  • Significant improvements: 1.78s per frame

20 of 51

OpenVino optimization

  • Even better inference speed: ~0.13-0.16s per frame

21 of 51

Integrate new model into post-processing code

22 of 51

Refine detection based on LiDAR reflective value

23 of 51

Refine detection based on LiDAR reflective value

24 of 51

Project LiDAR point cloud onto image plane

25 of 51

Project LiDAR point cloud onto image plane

26 of 51

Project LiDAR point cloud onto image plane

27 of 51

Project LiDAR point cloud onto image plane

28 of 51

Project LiDAR point cloud onto image plane

29 of 51

Project LiDAR point cloud onto image plane

30 of 51

LiDAR approach conclusion

  • Noisy Data -> can’t guarantee an improvement on image-based detection
  • Terrible performance under insufficient lighting
  • Might slow down algorithm

​

31 of 51

Remove ground points? (LiDAR)

  • Still the signal is not strong enough to be used for detection alone
  • However, is it good for a verification method once we detect a lane at that position?

32 of 51

Integration into the Pipeline

  • Fill in LaneMarkingLnDist

​

  • Method:
    • Calculate lane position on image
    • Find LiDAR pointcloud to image pixel correspondence (projection)
    • Take the bottom point, mid-point of the line on image
    • Find the closest corresponding LiDAR point
    • Fetch the y value of corresponding LiDAR points and average them

​

33 of 51

Visual Guide

Pt_3d[y]

Pt_3d[y]

Average

34 of 51

New Task:

  • Lane Type
  • Lane Color

35 of 51

Line Type:�Road shoulder detection

  • Utilizing the road segmentation

​

36 of 51

Find Edge points

37 of 51

Get a consistent line using RANSAC

38 of 51

Result

39 of 51

Lane type & color: first attempt

  • Find 2d pixel points along the detected lane, compute average hue & variance on value

​

  • Failed: Too Ideal! The detected line usually does not perfectly align with the actual lane!

​

40 of 51

Second attempt: implementing a “Scan”

41 of 51

How does it identify lane type & color?

  • To check if the horizontal stripe (a single frame from scanning) contains a lane, take the brightest pixel and calculate its deviation from the mean, if it deviates enough it likely belongs to a lane, now set the lane value for this stripe as True.

​

42 of 51

But… How about the noise?

  • Belief: it will not affect all the strips

​

  • Lane_Stripe: [True True True True True… False True… True]
            • This is a solid lane!
  • Lane_Stripe: [True... True False False… False True… True]
            • This is a dashed lane!

​

43 of 51

Results

  • Red: Lane Present
  • Yellow: Lane Absent

44 of 51

45 of 51

46 of 51

Find the longest lane segment

  • Stitch up some gaps and calculate the longest lane segment, if it’s longer than a certain percentage of the total length, then it’s solid.

47 of 51

How does it identify lane type & color?

  • To check if To check if the horizontal stripe contains a yellow lane given it contains a lane, take the brightest pixel (again!) to see if it belongs to the color range of yellow lane (a very broad range!) If so, set the yellow value for this stripe as True

​

48 of 51

Result

  • Yellow = yellow absent�Red = yellow present

49 of 51

How do we make decision

  • Calculate percentage of points/stripes that is considered yellow
  • The difference is surprisingly big:
    • For non-yellow lane, it’s almost always 0.0
    • For yellow lane, it’s always >0.7

50 of 51

Combined result on current task

51 of 51