What is the preferred window size (N) for each case (Go-back-N and Selective-Repeat) for a case where propagation delay is D seconds and link rate is R bps?

In Go-back-N, the window size has to be smaller than the range of sequence numbers used.

In Selective-Repeat, the window size has to be smaller (or equal than) half of the range of sequence number used.

Provided that you have the freedom, you want the window size N to be:

N = 2 * D * R  (bytes) ^[1]

This ensures you can reach the condition of sending non-stop. A lower value will reduce the throughput. This window size is measured in bits. To convert it into packets just divide it by the packet size (P).

N = ( 2 * D * R  ) / P (packets)

Why a too large window is a bad idea?

A too-large value will not improve performance but it will increase the memory requirements.

What is the relationship with the maximum size of the transmission window (N) and the number of bits used for the sequence numbers on both, Go-back-N and Selective Repeat?

If b bits are used for sequence numbers, then the range will be {0 .. 2b-1} so the maximum window size in Go-Back-N is N=2b-1

For Selective-Repeat, N=2b-1

What is the interest of the delay-bandwidth product when choosing the window size?

When considering delay as the total round-trip delay of a link, including twice the propagation delay plus any other delay source as queueing or processing delays, the delay times bandwidth product converts this time into bits.

This is the total number of bits that can be transmitted during the delay time. As mentioned in the first question, we want to be able to have a transmission window large enough to fill this pipe.