mjhjmtu

Bitcoin uses TCP for P2P but why is UDP not used? Modern day internet has relatively low packet loss rate so UDP is reliable. Even if some packets are dropped the peers can always request for the blocks. With TCP, the network is rather staticly constructed whereas UDP you can construct a random graph each time a block is propagated as its connectionless.

NANO uses UDP and I think it’s a good approach. I’m just curious as to if I’m missing something. Bootstrapping can obviously be done using TCP

The Bitcoin system doesn't have just one network protocol: Any way of obtaining the blocks is equally valid-- blocks over freenet, over satellite broadcast, over the P2P network, all work just as well and are used in practice. UDP is used with bitcoin too, by the Fibre protocol.

As far the common Bitcoin P2P protocol goes UDP is not an especially good fit for most operations that it performs. Bitcoin needs to reliably send messages far larger than an IP packet, like transactions and blocks. Although the internet is "reliable" packet loss rates of 1-3% are common. This means that any application needing to communicate large messages with UDP must implement packetization, retransmission, reordering, etc-- the same things that TCP already implements for us. Many applications that have 'rolled their own TCP' in userspace have ended up with exploitable bugs in them, so it isn't something that should be done without good cause.

UDP also has the issue of NAT traversal: Getting bidirectional communication across a NAT with UDP is not a simple matter. Crossing anything more complicated than a full cone nat requires considerable amounts of special code but without it there will be a many hosts that just can't talk to other peers with UDP.

You mention connectionlessness allowing you relay to a random host in the network, but in Bitcoin P2P we make use of the consistent relationship between nodes to make the network more reliable and efficient. Nodes have an idea what their peers already know and can avoid sending them redundant data. Nodes also know which peers have been fastest in sending them blocks in the past and handle them specially. Likewise, even if the protocol is connectionless there are costs to processing messages from peers by prioritizing handling messages from existing peers, Bitcoin reduces the impact of some kinds of DOS attack. Handling larger than one-packet messages, retransmissions, etc. also mean that even with a connectionless transport there still needs to be some kind of persistent state.

But an if implementation did want to randomly connect to send a message it could, at an extra cost is a couple round trips for the handshake. Or not even that: there are only about 10,000 reachable nodes in the P2P network, there wouldn't be much of a challenge in holding open a mostly idle TCP connection to each and ever one of them if all the state retained were just the TCP state. So for that point the use of UDP would at best just be an optimization for something that could already be done but isn't done. I think it would be more interesting to demonstrate the usefulness of having all those connections first, before worrying about optimizing it.

So why might UDP transport be useful to the Bitcoin P2P protocol?

• To get lowest latency block transfer possible there must be no round trips even in the face of packet loss which precludes TCP and which is why Fibre uses UDP. But to get no round trips the protocol must be able to handle loss without retransmission and to get low latency a block must be decodable with a minimum amount of data received. This requires very sophisticated error correction techniques which are an evolving area and haven't become mature enough yet to consider mainstreaming them. The latency benefits of fibre also exist so long as a small number of hosts are using it, since it does the heavy lifting of taking blocks all around the world. Round trips don't cause much harm on low latency links. And this latency concern only applies to block relay.




• Unlike TCP, UDP requires some amount of NAT traversal handling to simply get bidirectional communication working. But combined with full nat traversal handling UDP is often able to establish communication between hosts which are both behind different NATs. This could be helpful for the Bitcoin P2P network since most hosts are unreachable behind a nat. So, ironically, one of UDP's challenges is also one of its uses. However, to connect hosts which are both behind a net the assistance of a third party non-natted host is required along with even more NAT traversal code. Considering the complexity of traversal improving Bitcoin's support for TCP port mapping (e.g. implementing NAT-PMP) would probably be a better return on investment right now.




• Use of UDP would allow for "worst than best effort" traffic handling. For low priority "bulk traffic" like syncing the blockchain it would be nice if the traffic carefully shaped itself to avoid interfering with other traffic on the network. Alternative congestion control algorithims like LEDBAT make it possible to transfer low priority data with minimal impact. But since TCP stacks don't yet commonly support these congestion control approaches, applications that want them currently need to implement their own transports. For Bitcoin it would be ideal if we could just flip a socket option and turn LEDBAT on and off on existing connections (e.g. when the peer is requesting historical blocks or in the future when something like Fibre is being used by the node to relay new blocks) but that isn't an option yet.

[These last two reasons are the reasons that Bittorrent usually uses UDP]

Simply, NANO uses UDP for blocks which are usually (certainly mostly, but always??) lower than the theoretical size limit of a UDP packet, 65 KB. In Bitcoin's case, transactions ≠ blocks. It's surely inefficient to transfer long data (Bitcoin blocks can be up to 4 megabytes - theoretically) using UDP, and that's also how other services choose between them. Besides, even if blocks were a few kilobytes, packet drop will cause miners to lose A LOT, as a few seconds delay is significant.

If the data is small, you can just send one packet (which is usually much lower than 65 KB). Otherwise, using TCP, handshake, keepalive things are not expensive.

Trivia: In Satoshi's client, there were no additional error correction codes in the protocol. Satoshi trusted TCP error correction.

UDP is not very useful in Bitcoin. It is largely a unidirectional communication structure, whereas Bitcoin relies on bidirectional communication. When a node connects to another node, there is saved state for those nodes (nodes keep track of things that they have sent to other nodes and of things they have received from other nodes) and there is back and forth communication (e.g. send inv, receive getdata response, and send the data). UDP is not useful for that, but TCP is.

Furthermore, UDP exposes the nodes to the unreliability of the networks. The packet loss rate is not 0 (and cannot be 0), so using UDP results in a lot of extra data being sent due to packet loss. This is because blocks and transactions cannot be missing any bytes whatsoever. Otherwise they will be invalid. With TCP, if a packet is dropped, TCP will handle the resending of the data. But with UDP that needs to be handled at the application layer and it just gets complicated since figuring out what has been dropped requires bidirectional communication which is not easily done with UDP.

So overall, UDP is not useful for Bitcoin. It does not support bidirectional communication and it does not guarantee delivery of packets. Both of those things are necessary for Bitcoin's P2P protocol to work, so UDP can't be used. TCP is does both of those things at the protocol level, so it is used.

There are several Reliable UDP implementations available by now, so requirements like packet loss recovery and statefulness can be addressed. One example is the UDT library developed by the University of Chicago: http://udt.sourceforge.net/

We've developed a protocol named SRT (Secure Reliable Transport), initially based on UDT and extended by adding bi-directional data transfer and encryption to the mix. Although it's optimized for real-time data like video, it's content independent by design: https://github.com/Haivision/srt

In video use cases, we are seeing dramatically increased bandwidth utilization numbers using UDP over TCP (e.g. RTMP), especially relevant for longer distance communication. I do believe it does make sense to evaluate UDP based alternatives for data transfer between nodes in order to increase network efficiency.