How could I rise to UDP speed ?

Joe von Habsburg

@JonB said in How could I rise to UDP speed ?:

You are aware, aren't you, that using UDP you may always lose data packets?

Yes I know why everyone repeat that ? I have to use UDP and I need do not miss. Yes I know again, It is impossbile. But I did not choose that (TCP/UDP)...

@JonB said in How could I rise to UDP speed ?:

Apart from the crazy time issue, you will never reliably receive all data sent by the other side

I add crc and some variables like index and count, you can see on my example.

@JonB said in How could I rise to UDP speed ?:

does whatever you do with the data not care about this?

If somewhere updated and somewhere do not, I use same array, code work ok, but when somewhere do not update in array yes I loss some future.

@Kent-Dorfman said in How could I rise to UDP speed ?:

you guys are still skirting around the main issues. OP says single UDP "packet" is 0.5-1MB in length. Of course that is impossible with UDP so we're not getting the strait story.

No I did not say like that. I take from client 0.5-1MB data yes but, it is splitted.
@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I take 1460B in 1 time. So I take bytes in 300-600 step.

@goldenhawking

Thank you for reply.

Thanks everyone for try help. I know, I want imposible thing, but that it is not my choose.
@goldenhawking @JonB @Kent-Dorfman @SimonSchroeder <3

Topic has been closed.

JonB

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

Apart from the crazy time issue, you will never reliably receive all data sent by the other side

I add crc and some variables like index and count, you can see on my example.

How does that help? If you understand UDP you know that makes no difference. You use UDP when you don't care about losing some data. So long as that is acceptable you are OK.

J.Hilk

There were many valid answers given to your topic and it's rather rude to the people who took the time and answered and rude to potential future visitors to simply delete this topic.

therefore I restored it.

You're essentially trying to put diesel into a petrol engine — UDP and TCP exist for fundamentally different reasons, and fighting that is working against the protocol, not with it.

UDP is designed to be fast and fire-and-forget. It deliberately has no delivery guarantee, no ordering, and no flow control. That's not a bug — it's the entire point.

What you're trying to achieve — receiving lossless megabytes of data reliably — is exactly what TCP was designed for. The moment you start thinking "I need to ask the sender to retransmit missing datagrams", you are reinventing TCP on top of UDP, and you will do it worse than the OS already does it for you.

SimonSchroeder

@J.Hilk said in How could I rise to UDP speed ?:

you are reinventing TCP on top of UDP, and you will do it worse than the OS already does it for you.

Well, it totally makes sense to reinvent TCP on top of UDP. The major problem of TCP is that it starts with really slow speeds and then gets faster until one of the two sides cannot handle the speed anymore. And then it drops back to the really slow initial speed and the whole game repeats. This is why QUIC was invented which is based on UDP. You can definitely do better than TCP (at least for speed).

J.Hilk

@SimonSchroeder Sadly I have 0 experience with QUIC, so I take your word for it ^^
A quick google search shows it was "invented" in 2012, which is already 14 years ago... dear god I'm getting old!

Joe von Habsburg

@SimonSchroeder said in How could I rise to UDP speed ?:

Well, it totally makes sense to reinvent TCP on top of UDP.

I do not want to like that but, I have to.. I have to use Udp, I have to use like tcp because, I have to know each datagram what... Because I receive list, every datagram carry some part of list, where is that start end end. So, I have to use like tcp...

@SimonSchroeder said in How could I rise to UDP speed ?:

QUIC

I have not no idea for that.

@J.Hilk said in How could I rise to UDP speed ?:

dear god I'm getting old!

me too brother

JonB

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I have to use Udp

OK, then why is it so hard for you to confirm (or deny) what we have been saying: is your "client" program OK with missing/losing some packets? If you want reliable --- no packets lost --- then you are going to have problems using UDP.

Pl45m4

@Joe-von-Habsburg

If your point / use case is data streaming / data distribution... maybe look into some middleware like MQTT or DDS...
AFAIK both are able to send via TCP and UDP, but most libraries/implementations have their own QoS, which will handle lost packages for you (and resend them) even if you chose UDP transport.

Joe von Habsburg

@JonB said in How could I rise to UDP speed ?:

why is it so hard for you to confirm (or deny)

I know man, you are right, just, I need take without losing. I wondered is there any way. If I give bad feeling to you, I am sorry.

@JonB said in How could I rise to UDP speed ?:

have problems using UDP.

yes

@Pl45m4 said in How could I rise to UDP speed ?:

ome middleware like MQTT or DDS...

Thank you for your reply.

JonB

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I know man, you are right, just, I need take without losing. I wondered is there any way. If I give bad feeling to you, I am sorry.

It's OK. Just according to me if you are only offered UDP (no TCP) you cannot guarantee to "take without losing". That is my understanding. So I do not think there is a way, if it is only offering UDP it is intended to allow for losing. I will say no more. Good luck.

Pl45m4

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I need take without losing.

Who said that? People demand a lot of bs, when they have no idea how things work. Using UDP without losing packages is like trying to cross a desert without touching any sand...
Either use TCP and accept some drawbacks in performance (due to handshake, flow control etc.) or add your own layer / protocol on top of UDP in your program where you number and count your packages etc etc.
(same is done when using RTPS Wire-Protocol in middleware like DDS. Sender and receiver have internal counters and packages are numbered and much more)

Joe von Habsburg

@Pl45m4 said in How could I rise to UDP speed ?:

Who said that? People demand a lot of bs, when they have no idea how things work. Using UDP without losing packages is like trying to cross a desert without touching any sand...

yes... I could not explain them.

Joe von Habsburg

Hello, I'm back again. I have a problem. I'm told that there shouldn't be any packet loss when connecting directly via cable. They say the packet loss is due to me not receiving it. Is this even possible?

JonB

@Joe-von-Habsburg
You could investigate what people have to say about this by a Google search like is udp still lossless when connected directly by cable:

No, UDP is not inherently lossless, even when connected directly by a cable (point-to-point).

While a direct cable connection removes the risks of network congestion from external routers and switches, it does not change the core design of the User Datagram Protocol (UDP). UDP is "fire-and-forget," meaning it provides no mechanisms for error correction, retransmission of lost packets, or flow control

Application-Level Issues: If the application on the receiving end is not reading from the socket fast enough, packets will be lost, regardless of how good the cable is.

Summary

A direct connection makes packet loss highly unlikely, but it does not guarantee it will be lossless. If your application absolutely requires all data to arrive in order, you must implement your own reliability layer on top of UDP or use TCP.

Per some people's comments you may find your situation is pretty reliable, though I would not necessarily count on 100.0%.

If you suspect your (Qt) application is failing to keep up when it should, it would not take you long to knock together a standalone, non-UI, non-Qt C++ application using sockets which is tiny and runs as fast as possible. You could then compare that against your Qt implementation --- even running at the same time as each other or whatever --- to see whether your Qt version keeps up.

Pl45m4

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I'm told that there shouldn't be any packet loss when connecting directly via cable. They say the packet loss is due to me not receiving it. Is this even possible?

To catch the lost packages, you need to place a funnel and a bucket below your cable so all the data that leaks out is caught in the bucket.

/s

For real:
You lose packages not only because of your transport medium (cable, air, etc...). Adapters, sockets, and other endpoint bottle necks also matter. There is a protocol that takes care of that, your choice (UDP) does not.

Joe von Habsburg

@JonB said in How could I rise to UDP speed ?:

If you suspect your (Qt) application is failing to keep up when it should, it would not take you long to knock together a standalone, non-UI, non-Qt C++ application using sockets which is tiny and runs as fast as possible. You could then compare that against your Qt implementation --- even running at the same time as each other or whatever --- to see whether your Qt version keeps up.

Hello, I tested with vanilla C++, again I lost some packages.

@Pl45m4 said in How could I rise to UDP speed ?:

data that leaks out is caught in the bucket.

:D

@JonB @Pl45m4 thank you for try help again.

I have a question. I ask because I do not know why. I can see the lost package in wireshark. why ?

J.Hilk

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

I have a question. I ask because I do not know why. I can see the lost package in wireshark. why ?

Did you ever play „Chinese whispers“ ?

If you're the 10th in line, the data is more likely to be scrambled, especially if the 9th person before you is 100% busy spinning in place.
If you listen in, via wire shark, when the data is first whispered, chances are higher you overhear everything

Wireshark taps the network stack before the socket receive buffer, there's a whole truck load of software before your program comes into play. OS Buffers, drivers etc

JonB

@Joe-von-Habsburg said in How could I rise to UDP speed ?:

Hello, I tested with vanilla C++, again I lost some packages.

I should actually have said you only need C not C++ for a sockets program, so you can keep it minimal. But this is probably not a C++ issue in itself.

I would actually briefly try your standalone test as a Python script, using whatever package they tell you to use for UDP. Python does sometimes have packages which do/know more than you might. If that keeps up while your C++/Qt efforts do not you have a problem which can be addressed; if it fails to keep up with Python and a suitable UDP package then you presumably have no hope.... You can explain your efforts to boss/stakeholder/client sender.

Joe von Habsburg

I have some good news for now. I think I've solved the problem on a large scale. I changed receive buffer size in socket option. It works for me.

// From
_udpSocket.setSocketOption(QAbstractSocket::ReceiveBufferSizeSocketOption, 16 * 1024 * 1024);

// To
_udpSocket.setSocketOption(QAbstractSocket::ReceiveBufferSizeSocketOption, 1024 * 1024 * 1024);

JonB

@Joe-von-Habsburg Ah ha! One of us should have mentioned this when you said about sender packet size :(