reinterpret_cast for received tcp packets
-
@Redman said in reinterpret_cast for received tcp packets:
No need for checking if the tcpSocker buffer is empty or if the available bytes in the buffer are only partially tansmitted etc.etc. Simply readAll bytes and cast it to the desired struct.
Are you doing this as the communication runs along during the program, or only at the end after you have closed the socket? Assuming it's as you go along, how do you know that
readAll()returns all bytes which might have been sent? It only guarantees to return 1 byte whenreadyRead()signal is fired, you have no guarantee any further bytes are there. The fact that it seems to work in some situation does not mean it won't go wrong in another. This has nothing to do with the casting.What do you use in Python to pack the data and send it, and how that compares to your
struct's size/order/padding? What endian-ness is the sender machine and order of bytes sent? -
wrote on 15 Jan 2025, 07:51 last edited by
@J-Hilk As far as I understood it either its me who has to adjust or the sender. Why would you consider this to be a generous definition of fixed?
-
@J-Hilk As far as I understood it either its me who has to adjust or the sender. Why would you consider this to be a generous definition of fixed?
@Redman said in reinterpret_cast for received tcp packets:
Why would you consider this to be a generous definition of fixed?
What about indianess?
-
@Redman said in reinterpret_cast for received tcp packets:
Why would you consider this to be a generous definition of fixed?
What about indianess?
wrote on 15 Jan 2025, 07:56 last edited by Redman@jsulm Since both participants are Windows, little-endian is to be assumed. We did in fact switch to big-endian for each part of the message. This resulted in the first 2 parts not being recognized as a 1 anymore.
The fact, that in the 3rd part a 1 was sent but was cast to a 65536 indicates the byte-alignment problem.
Now Im not sure why @J-Hilk said it was a generous definition of fixed. -
@J-Hilk As far as I understood it either its me who has to adjust or the sender. Why would you consider this to be a generous definition of fixed?
@Redman said in reinterpret_cast for received tcp packets:
Why would you consider this to be a generous definition of fixed?
what about different compiler? pragma is highly depending on compiler and compiler version
also casting a char* to a struct is still UB and just a dud waiting to explode
-
@Redman said in reinterpret_cast for received tcp packets:
Why would you consider this to be a generous definition of fixed?
what about different compiler? pragma is highly depending on compiler and compiler version
also casting a char* to a struct is still UB and just a dud waiting to explode
wrote on 15 Jan 2025, 08:07 last edited by@J-Hilk Since there are no plans of switching to a different Compiler than MSVC I did not consider that.
I would assume that#pragma pack(push, 1) #pragma pack(pop)would do the same through all MSVC versions.
Regarding that reinterpret_cast, I will, as you suggested, implement a ctor which takes a QByteArray.
In fact, I already tried that out during figuring out what the actual problem was. To me the ctor looked "ugly".
DoingbyteArray.sliced(i,n)to retrieve the corresponding bytes and casting them to the desired type might be a quick thing to implement for such a small struct. But for larger structs this gets ugly very fast.Since c++ doesn't offer reflection, can I somehow use Qt's meta-object system to achieve a dynamic construction of my object?
-
I also would not assume indianess based on operating system. Windows also runs on ARM and ARM can be both big or little.
-
I also would not assume indianess based on operating system. Windows also runs on ARM and ARM can be both big or little.
-
@J-Hilk Since there are no plans of switching to a different Compiler than MSVC I did not consider that.
I would assume that#pragma pack(push, 1) #pragma pack(pop)would do the same through all MSVC versions.
Regarding that reinterpret_cast, I will, as you suggested, implement a ctor which takes a QByteArray.
In fact, I already tried that out during figuring out what the actual problem was. To me the ctor looked "ugly".
DoingbyteArray.sliced(i,n)to retrieve the corresponding bytes and casting them to the desired type might be a quick thing to implement for such a small struct. But for larger structs this gets ugly very fast.Since c++ doesn't offer reflection, can I somehow use Qt's meta-object system to achieve a dynamic construction of my object?
@Redman I would suggest using QDataStream
-
Casting a struct to a received bunch of bytes is not correct. It may work now but fail as soon as a small piece changes. Therefore you have to describe the bytes on the paket (e.g. in an interface document) and decode them accordingly. Everything else is a hack.
-
@JonB I do this during the whole execution time of my application. The tcp packet size is only 22 bytes, so fairly small.
How would I go about making sure I get all the bytes out of the buffer?wrote on 15 Jan 2025, 08:45 last edited by JonB@Redman said in reinterpret_cast for received tcp packets:
@JonB I do this during the whole execution time of my application. The tcp packet size is only 22 bytes, so fairly small.
How would I go about making sure I get all the bytes out of the buffer?Assuming
recEchoDataReady()is called as a slot fromreadyRead()signal, what you are supposed to do is append the bytes to a "pending" buffer and then check whether the total number read into pending equals or exceeds the number you need/expect for yourstruct. If so remove those from pending. Not only might you be "short" (read only 1+ bytes when 22 expected), you might also "overflow" (read e.g. 32 when 22 wanted for one packet because it also finds some of the next packet), in which case that packet will "work" but the next packet will go wrong.If you don't want to do this, because you believe it is always 22, you might at least put in a
QASSERT(bytes_read == sizeof(Coordinate))to verify that never goes wrong.You say a lot about the
structsize/packing/padding/endianness etc. at the receiving C++ side but did not answer how you send the data from the Python side. If I understand right you have looked at "the wire" or the bytes you receive and say they don't look right as raw bytes? In which case check the sender side. I believe you are saying that it used to work when the sender was C++ but now goes wrong when the sender is Python, in which case the receiver side is irrelevant and the bytes sent must be different, if that is really the case.01:00:00:00:01:00:01:00:00:00:01:00:00:00:00:00:00:00:01:00:00:00Unfortunately after casting that ByteArray to my struct I get following values
1 1 65536 65536 -1870011232
While this seems to contradict the statement that it is the sender side which has changed/gone wrong, I would examine this more closely. Off the top of my head that data buffer would not produce the (packed)
structvalues you say it does.You also wrote at one point:
@Redman said in reinterpret_cast for received tcp packets:
Fixed the problem.
Does this mean you no longer have any problem sending/receiving the structure? You introduced the
#pragma pack(push, 1), did that fix, was that at C++ receiver side or what did you do at Python sender side? Your Cstructneeds to bepackedif you expect it to be 22 bytes size. -
Casting a struct to a received bunch of bytes is not correct. It may work now but fail as soon as a small piece changes. Therefore you have to describe the bytes on the paket (e.g. in an interface document) and decode them accordingly. Everything else is a hack.
wrote on 15 Jan 2025, 09:08 last edited by@Christian-Ehrlicher The interface is well documented and both sites agreed to it. So there will be no change to that without further discussion between me and our partner :)
-
@Christian-Ehrlicher The interface is well documented and both sites agreed to it. So there will be no change to that without further discussion between me and our partner :)
@Redman said in reinterpret_cast for received tcp packets:
The interface is well documented and both sites agreed to it. So there will be no change to that
-
@Christian-Ehrlicher The interface is well documented and both sites agreed to it. So there will be no change to that without further discussion between me and our partner :)
@Redman said in reinterpret_cast for received tcp packets:
The interface is well documented and both sites agreed to it.
Then you would not have asked any of the questions above nor any problems wrt connecting to another process.
-
@Redman said in reinterpret_cast for received tcp packets:
The interface is well documented and both sites agreed to it.
Then you would not have asked any of the questions above nor any problems wrt connecting to another process.
wrote on 15 Jan 2025, 13:41 last edited by@Christian-Ehrlicher And you live in a perfect world where no problems occur? :)
-
@Christian-Ehrlicher And you live in a perfect world where no problems occur? :)
@Redman said in reinterpret_cast for received tcp packets:
And you live in a perfect world where no problems occur?
No, but your questions/problem revealed a basic lack of a "well documented" interface.
-
Hello,
I am developing an app on a 64bit windows machine.
Im using Qt 6.8.1 and the c++20 standard.My app receives tcp packets from an app running on windows.
I try to reinterpret_cast the packets into following struct.struct Coordinate { quint32 id; quint16 mode; qint32 x; quint64 y; qint32 z; };like followed
void MyReceiver::recEchoDataReady() { QByteArray bytes = m_tcpServerClient->readAll(); Coordinate pt = *reinterpret_cast<Coordinate*>(bytes.data()); QMap<QString, QVariant> coord; coord.insert(QStringLiteral("ID"), pt.id); coord.insert(QStringLiteral("MODE"), pt.mode); coord.insert(QStringLiteral("X"), pt.x); coord.insert(QStringLiteral("Y"), pt.y); coord.insert(QStringLiteral("Z"), pt.z); emit echoDataReady(coord); }This approached worked flawlessly. We transmitted several millions of packets over the course of a couple of months without any problems. No need for checking if the tcpSocker buffer is empty or if the available bytes in the buffer are only partially tansmitted etc.etc. Simply readAll bytes and cast it to the desired struct.
Recently there were changes made on the sender site of this protocol. They switched from a c++ app to a python app.
Following you can see the hex representation of the bytes that arrived at my site. If you try to fit this into the struct you would think that the data within the struct would be 1 1 1 1 1.01:00:00:00:01:00:01:00:00:00:01:00:00:00:00:00:00:00:01:00:00:00Unfortunately after casting that ByteArray to my struct I get following values
1 1 65536 65536 -1870011232Now I'm not sure if there are any incompatibilities between c++ and python datatype.
I'm kinda stuck on this problem and have no clue why that is happening.If anyone has an idea on how to investigate this or provide a meaningfull explanation it would be greatly appreciated.
wrote on 17 Jan 2025, 07:51 last edited by@Redman said in reinterpret_cast for received tcp packets:
01:00:00:00:01:00:01:00:00:00:01:00:00:00:00:00:00:00:01:00:00:00
This is what you'd expect for a packed struct (as you have figures out). It worked before because both sides were using C++ and so both sides were not using packed structs (so much about the "well defined protocol").
In order to not have that problem @J-Hilk is right to write a constructor expecting a QByteArray. For sending you should also pack the members into a QByteArray yourself. This also makes you immune to the order of the members (which is only guaranteed by a very, very recent C++ standard). It is also common practice to order members from biggest to smallest, so you can adhere to alignment rules and still get the most compact representation. This would also solve your problem (on the C++ side; I don't know Python) that you don't have to explicitly pack the struct.
19/26