[capnproto] How to create kj::ArrayPtr<capnp::word> from char *?

[z***@gmail.com]

I have raw buffer read from socket and I want to pass/add it to
kj::ArrayPtr<capnp::word>. Is there a simple way to do this (without
additional copies)?

I've tried answer to this (not the copying one) :
https://groups.google.com/forum/#!topic/capnproto/CIUxq-Y4128, with simple
code :

std::vector<unsigned char> bytes(1024);
kj::ArrayPtr<capnp::word> words(reinterpret_cast<const
capnp::byte*>(bytes.data()), bytes.size() / sizeof(capnp::word));

but it gives me

error: no matching function for call to
‘kj::ArrayPtr<capnp::word>::ArrayPtr(const byte*, std::vector<unsigned
char>::size_type)’

and I'm kind of stuck.

['Kenton Varda' via Cap'n Proto]

Hi zosiasmail,

Post by z***@gmail.com
std::vector<unsigned char> bytes(1024);
kj::ArrayPtr<capnp::word> words(reinterpret_cast<const
capnp::byte*>(bytes.data()), bytes.size() / sizeof(capnp::word));

This code will work if you change the reinterpret_cast to:

reinterpret_cast<capnp::word*>(bytes.data())

That is, you are casting to the wrong type (and wrong constness).

However, there's a deeper problem this doesn't solve, which is ensuring
that your buffer is aligned. There's no guarantee that the bytes in a
vector are aligned to a word boundary. Since Cap'n Proto doesn't have a
separate decoding step, it's necessary that the message be properly-aligned
for direct access of types up to 64 bits.

The trick is to allocate your backing buffer as words in the first place:

std::vector<capnp::word> words(128);

Now you can read into this vector like:

read(fd, words.begin(), words.size() * sizeof(capnp::word))

Better yet, don't use std::vector; use kj::Array all the way through:

auto words = kj::heapArray<capnp::word>(128);
ssize_t n = read(fd, words.begin(), words.size() * sizeof(capnp::word));
// TODO: check errors, etc.
auto messageWords = words.slice(0, n / sizeof(capnp::word));

If you are using an I/O library that insists on giving you strictly bytes
with no alignment guarantee, then you might have a problem. You may be
forced to make a copy in this case.

-Kenton

[Zosia A]

Thank you for fast reply!

Yeah, I was doing it just how you've shown, but now I'm trying to integrate
it with existing project and trying to do this with as little changes
possible. I wasn't aware, that vector doesn't guarantee alignment for any
standard type, so I will have to stick to the first version. Thank you once
again :).

Post by 'Kenton Varda' via Cap'n Proto
Hi zosiasmail,

Post by z***@gmail.com
std::vector<unsigned char> bytes(1024);
kj::ArrayPtr<capnp::word> words(reinterpret_cast<const
capnp::byte*>(bytes.data()), bytes.size() / sizeof(capnp::word));

reinterpret_cast<capnp::word*>(bytes.data())
That is, you are casting to the wrong type (and wrong constness).
However, there's a deeper problem this doesn't solve, which is ensuring
that your buffer is aligned. There's no guarantee that the bytes in a
vector are aligned to a word boundary. Since Cap'n Proto doesn't have a
separate decoding step, it's necessary that the message be properly-aligned
for direct access of types up to 64 bits.
std::vector<capnp::word> words(128);
read(fd, words.begin(), words.size() * sizeof(capnp::word))
auto words = kj::heapArray<capnp::word>(128);
ssize_t n = read(fd, words.begin(), words.size() *
sizeof(capnp::word));
// TODO: check errors, etc.
auto messageWords = words.slice(0, n / sizeof(capnp::word));
If you are using an I/O library that insists on giving you strictly bytes
with no alignment guarantee, then you might have a problem. You may be
forced to make a copy in this case.
-Kenton

['Kenton Varda' via Cap'n Proto]

FWIW, it's actually pretty likely that a vector<char>'s buffer will
actually be aligned in practice, because it is almost certainly allocated
using malloc() which always returns aligned buffers. But, there's
technically no guarantee.

Given this, perhaps you could write some code which *checks* for alignment,
and if so, does a reinterpret_cast, but if the buffer isn't aligned, then
it falls back to a copy.

bool aligned = reinterpret_cast<uintptr_t>(bytes.begin()) %
sizeof(void*) == 0;

(I use sizeof(void*) as the denominator because 32-bit systems usually
require only 32-bit alignment even for 64-bit data types.)

-Kenton

Post by Zosia A
Thank you for fast reply!
Yeah, I was doing it just how you've shown, but now I'm trying to
integrate it with existing project and trying to do this with as little
changes possible. I wasn't aware, that vector doesn't guarantee alignment
for any standard type, so I will have to stick to the first version. Thank
you once again :).

Post by 'Kenton Varda' via Cap'n Proto
Hi zosiasmail,

Post by z***@gmail.com
std::vector<unsigned char> bytes(1024);
kj::ArrayPtr<capnp::word> words(reinterpret_cast<const
capnp::byte*>(bytes.data()), bytes.size() / sizeof(capnp::word));

reinterpret_cast<capnp::word*>(bytes.data())
That is, you are casting to the wrong type (and wrong constness).
However, there's a deeper problem this doesn't solve, which is ensuring
that your buffer is aligned. There's no guarantee that the bytes in a
vector are aligned to a word boundary. Since Cap'n Proto doesn't have a
separate decoding step, it's necessary that the message be properly-aligned
for direct access of types up to 64 bits.
std::vector<capnp::word> words(128);
read(fd, words.begin(), words.size() * sizeof(capnp::word))
auto words = kj::heapArray<capnp::word>(128);
ssize_t n = read(fd, words.begin(), words.size() *
sizeof(capnp::word));
// TODO: check errors, etc.
auto messageWords = words.slice(0, n / sizeof(capnp::word));
If you are using an I/O library that insists on giving you strictly bytes
with no alignment guarantee, then you might have a problem. You may be
forced to make a copy in this case.
-Kenton