[capnproto] Deserializing Capnp Messages - Getting an exception

[n***@gmail.com]

HI,

Using capnproto-c++-0.6.1 I am trying to serialize a Capnp message to
bytes to send over a multicast socket. I am getting an error when trying to
deserialize.

Here is what I am trying to do.



auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl; //
flatArray size : 17


auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl; //
bytesToSend size : 136

// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);

// But the following throws an exception: capnp/serialize.c++:43: failed:
expected array.size() >= offset; Message ends prematurely in segment table.

char result[bytesToSend.size()]; // what i would like to write to my UDP
socket
auto words = kj::heapArray<capnp::word>(bytesToSend.size()/ sizeof(capnp::
word));
std::cout << "words size : " << words.size() << std::endl; // words size :
17 (as expected)

memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!




Note - this is similar to this previous topic :
https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the
comments there didn't get me going.

Thank you very much for your time.

['Kenton Varda' via Cap'n Proto]

Hi Noah,

I think you're missing a line like:

memcpy(result, bytesToSend.begin(), bytesToSend.size());

So `result` never gets initialized, so at the end you're trying to parse
uninitialized data. :)

-Kenton

Post by n***@gmail.com
HI,
Using capnproto-c++-0.6.1 I am trying to serialize a Capnp message to
bytes to send over a multicast socket. I am getting an error when trying to
deserialize.
Here is what I am trying to do.
auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl; //
flatArray size : 17
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl; //
bytesToSend size : 136
// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);
expected array.size() >= offset; Message ends prematurely in segment table.
char result[bytesToSend.size()]; // what i would like to write to my UDP
socket
word));
std::cout << "words size : " << words.size() << std::endl; // words size
: 17 (as expected)
memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!
https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the
comments there didn't get me going.
Thank you very much for your time.
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[Noah Keen]

Hi Kenton.

You are of course absolutely correct. If I memcpy bytesToSend into result
the example works without exception. Much thanks for catching that one. I
have a followup question though.

If I use my code above to serialize my message, my client (in pycapn) fails
to deserialize my message. It throws an error - No Root Node Found.

// My client fails to deserialize this packed on receipt
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
sendto(sock, bytesToSend.begin(), bytesToSend.size(), 0,
(struct sockaddr *)&destinationAddress, sizeof(struct
sockaddr_in));

This could be an issue with my client, but with some trial and error I
found if I use the following code to serialize my message my client works
great:

kj::byte result[capnp::messageToFlatArray(message).asBytes().size
()];
kj::ArrayPtr<kj::byte> bufferPtr = kj::arrayPtr(result, sizeof(
result));
kj::ArrayOutputStream arrayOutputStream(bufferPtr);
capnp::writeMessage(arrayOutputStream, message);

sendto(sock, arrayOutputStream.getArray().begin(), arrayOutputStream
.getArray().size(), 0,
(struct sockaddr *)&destinationAddress, sizeof(struct
sockaddr_in));


Looking at raw bytes that get written I have confirmed the two approaches
differ at the packet level. The second, working approach, has 0x10 in the
4th byte, but that's the only difference I see.

My question is, what is capnp::writeMessage(arrayOutputStream, message) doing
that my first approach is not? Why can't I just get the bytes and send?

Is my second approach the correct way? Is there a better technique? It
seems a little clunky to me - especially the first line where i use

kj::byte result[capnp::messageToFlatArray(message).asBytes().size
()];

Thanks again - I appreciate your time.

Noah

Post by n***@gmail.com
HI,
Using capnproto-c++-0.6.1 I am trying to serialize a Capnp message to
bytes to send over a multicast socket. I am getting an error when trying to
deserialize.
Here is what I am trying to do.
auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl; //
flatArray size : 17
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl; //
bytesToSend size : 136
// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);
expected array.size() >= offset; Message ends prematurely in segment table.
char result[bytesToSend.size()]; // what i would like to write to my UDP
socket
word));
std::cout << "words size : " << words.size() << std::endl; // words size
: 17 (as expected)
memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!
https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the
comments there didn't get me going.
Thank you very much for your time.

['Kenton Varda' via Cap'n Proto]

Hi Noah,

Post by Noah Keen
Hi Kenton.
You are of course absolutely correct. If I memcpy bytesToSend into result
the example works without exception. Much thanks for catching that one. I
have a followup question though.
If I use my code above to serialize my message, my client (in pycapn)
fails to deserialize my message. It throws an error - No Root Node Found.
// My client fails to deserialize this packed on receipt
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
sendto(sock, bytesToSend.begin(), bytesToSend.size(), 0,
(struct sockaddr *)&destinationAddress, sizeof(struct
sockaddr_in));

messageToFlatArray() returns an Array<word>, but calling .asBytes() on it
returns ArrayPtr<byte>. Array<> owns its contents while ArrayPtr<> points
to an array owned by someone else. Since you don't assign the Array
anywhere, it is destroyed at the end of the line. So, `bytesToSend` ends up
being a dangling pointer, and the bytes it points to could be overwritten
with garbage at any time.

To fix this, you will need to assign the word array to a local variable,
and then call `.asBytes()` on a separate line.

On another note, keep in mind that messageToFlatArray() makes a copy. You
could avoid this copy by using sendmsg() instead of sendto(), which allows
you to do a gather-write with msg_iov. You would call
message.getSegmentsForOutput() to get an array of arrays of words to write.
You also need to add the segment table to the front of this; see how
writeMessage() is implemented in the capnp repo in
c++/src/capnp/serialize.c++. Note that if you always have exactly one
segment (e.g. because you made sure to configure your MessageBuilder to
always allocate enough space), then the segment table is always 8 bytes:
four bytes of zeros, followed by a four-byte little-endian integer
indicating the size of the first segment in words.

-Kenton

Post by Noah Keen
This could be an issue with my client, but with some trial and error I
found if I use the following code to serialize my message my client works
kj::byte result[capnp::messageToFlatArray(message).asBytes().size
()];
kj::ArrayPtr<kj::byte> bufferPtr = kj::arrayPtr(result, sizeof(
result));
kj::ArrayOutputStream arrayOutputStream(bufferPtr);
capnp::writeMessage(arrayOutputStream, message);
sendto(sock, arrayOutputStream.getArray().begin(),
arrayOutputStream.getArray().size(), 0,
(struct sockaddr *)&destinationAddress, sizeof(struct
sockaddr_in));
Looking at raw bytes that get written I have confirmed the two approaches
differ at the packet level. The second, working approach, has 0x10 in the
4th byte, but that's the only difference I see.
My question is, what is capnp::writeMessage(arrayOutputStream, message) doing
that my first approach is not? Why can't I just get the bytes and send?
Is my second approach the correct way? Is there a better technique? It
seems a little clunky to me - especially the first line where i use
kj::byte result[capnp::messageToFlatArray(message).asBytes().size
()];
Thanks again - I appreciate your time.
Noah

Post by n***@gmail.com
HI,
Using capnproto-c++-0.6.1 I am trying to serialize a Capnp message to
bytes to send over a multicast socket. I am getting an error when trying to
deserialize.
Here is what I am trying to do.
auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl; //
flatArray size : 17
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl; //
bytesToSend size : 136
// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);
expected array.size() >= offset; Message ends prematurely in segment table.
char result[bytesToSend.size()]; // what i would like to write to my UDP
socket
auto words = kj::heapArray<capnp::word>(bytesToSend.size()/ sizeof(capnp
::word));
std::cout << "words size : " << words.size() << std::endl; // words size
: 17 (as expected)
memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!
https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the
comments there didn't get me going.
Thank you very much for your time.
--

You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[Noah Keen]

Nice! Thank you. For you your assistance and the lib!

I'll try to remember to post my finished working solution here once I know
it works.

Post by n***@gmail.com
HI,
Using capnproto-c++-0.6.1 I am trying to serialize a Capnp message to
bytes to send over a multicast socket. I am getting an error when trying to
deserialize.
Here is what I am trying to do.
auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl; //
flatArray size : 17
auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl; //
bytesToSend size : 136
// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);
expected array.size() >= offset; Message ends prematurely in segment table.
char result[bytesToSend.size()]; // what i would like to write to my UDP
socket
word));
std::cout << "words size : " << words.size() << std::endl; // words size
: 17 (as expected)
memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!
https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the
comments there didn't get me going.
Thank you very much for your time.