[capnproto] Working on Swift support

[Dan Appel]

Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.

Just wanted to let people know in case somebody has tips or concerns.

I'll also use this thread as a place to ask any future questions that I may
have :)

[David Renshaw]

Cool!

Do you take advantage of Swift's pattern matching in your support for Cap'n
Proto unions?

How do you manage the lifetime of a message's underlying buffer? Is it like
in capnproto-c++, where the burden is on the programmer to ensure that
references to a message's interior do not outlive the message itself? Or is
it more like capnproto-java, where that burden is taken up by automatic
memory management, at the cost of some run time performance? It's totally
unclear to me which would be the better choice for Swift.


- David

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[Dan Appel]

Post by David Renshaw
Do you take advantage of Swift's pattern matching in your support for

Cap'n Proto unions?
Yes! A union is generated as just an enum with associated values :)

Post by David Renshaw
How do you manage the lifetime of a message's underlying buffer?

Swift's native memory management model is ARC, so I'm going with that. Once
the message loses all references to it, the buffer gets deallocated.
Ideally each object the user makes has some kind of reference to the
message, so it would work right out of the box (this is the idea, but I
have not finalized the design of who references who, so it may not behave
correctly just yet).

Dan

Post by David Renshaw
Cool!
Do you take advantage of Swift's pattern matching in your support for
Cap'n Proto unions?
How do you manage the lifetime of a message's underlying buffer? Is it
like in capnproto-c++, where the burden is on the programmer to ensure that
references to a message's interior do not outlive the message itself? Or is
it more like capnproto-java, where that burden is taken up by automatic
memory management, at the cost of some run time performance? It's totally
unclear to me which would be the better choice for Swift.
- David

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

--

Dan Appel

[Dan Appel]

Okay, I have a question. In my structs I'm running into pointers that have
the value of 0 (the UInt64 pointer itself). What do I do with these? Is it
an error on my end that I'm running into these in the first place? I've
been trying to debug this for a few hours now and I can't find the source
of the issue.

Here's some logs that I got from reading CodeGeneratorRequest.ptrs[0]
(nodes @0 :List(Node)) - link
<https://gist.github.com/Danappelxx/a4536c946e56919ad6765a550a19c1e5>.

I'm leaning on this being my fault somewhere in the parsing but just wanted
to make sure I'm not missing something before I dive deeper.

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)

[David Renshaw]

Those are null pointers. A `has()` accessor would return `false` for them,
and a `get()` accessor would return the default value. See
https://capnproto.org/faq.html#how-do-i-make-a-field-optional

Post by Dan Appel
Okay, I have a question. In my structs I'm running into pointers that have
the value of 0 (the UInt64 pointer itself). What do I do with these? Is it
an error on my end that I'm running into these in the first place? I've
been trying to debug this for a few hours now and I can't find the source
of the issue.
Here's some logs that I got from reading CodeGeneratorRequest.ptrs[0]
<https://gist.github.com/Danappelxx/a4536c946e56919ad6765a550a19c1e5>.
I'm leaning on this being my fault somewhere in the parsing but just
wanted to make sure I'm not missing something before I dive deeper.

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)

--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[Dan Appel]

Oh, well now I feel silly :) Thanks for pointing that out!

Post by David Renshaw
Those are null pointers. A `has()` accessor would return `false` for them,
and a `get()` accessor would return the default value. See
https://capnproto.org/faq.html#how-do-i-make-a-field-optional

Post by Dan Appel
Okay, I have a question. In my structs I'm running into pointers that
have the value of 0 (the UInt64 pointer itself). What do I do with these?
Is it an error on my end that I'm running into these in the first place?
I've been trying to debug this for a few hours now and I can't find the
source of the issue.
Here's some logs that I got from reading CodeGeneratorRequest.ptrs[0]
<https://gist.github.com/Danappelxx/a4536c946e56919ad6765a550a19c1e5>.
I'm leaning on this being my fault somewhere in the parsing but just
wanted to make sure I'm not missing something before I dive deeper.

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or
so, and have been making decent progress. I've run into a few blind spots
in the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)

--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

--

Dan Appel

[Dan Appel]

Does this mean that every field (including lists) is nullable? That's quite
a shame. Looks like all my generated fields are going to be
implicitly-unwrapped-optional, then (reminds me of objective-c translated
into swift).

Post by Dan Appel
Oh, well now I feel silly :) Thanks for pointing that out!

Post by David Renshaw
Those are null pointers. A `has()` accessor would return `false` for
them, and a `get()` accessor would return the default value. See
https://capnproto.org/faq.html#how-do-i-make-a-field-optional

Post by Dan Appel
Okay, I have a question. In my structs I'm running into pointers that
have the value of 0 (the UInt64 pointer itself). What do I do with these?
Is it an error on my end that I'm running into these in the first place?
I've been trying to debug this for a few hours now and I can't find the
source of the issue.
Here's some logs that I got from reading CodeGeneratorRequest.ptrs[0]
<https://gist.github.com/Danappelxx/a4536c946e56919ad6765a550a19c1e5>.
I'm leaning on this being my fault somewhere in the parsing but just
wanted to make sure I'm not missing something before I dive deeper.

Post by Dan Appel
Hi! I've been working on a Swift compiler plugin for the past week or
so, and have been making decent progress. I've run into a few blind spots
in the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)

--
You received this message because you are subscribed to the Google
Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/capnproto.

--

Dan Appel

[Ross Light]

Just pointer fields. Data section fields are always "non-optional".

Post by Dan Appel
Does this mean that every field (including lists) is nullable? That's
quite a shame. Looks like all my generated fields are going to be
implicitly-unwrapped-optional, then (reminds me of objective-c translated
into swift).
Oh, well now I feel silly :) Thanks for pointing that out!
Those are null pointers. A `has()` accessor would return `false` for them,
and a `get()` accessor would return the default value. See
https://capnproto.org/faq.html#how-do-i-make-a-field-optional
Okay, I have a question. In my structs I'm running into pointers that have
the value of 0 (the UInt64 pointer itself). What do I do with these? Is it
an error on my end that I'm running into these in the first place? I've
been trying to debug this for a few hours now and I can't find the source
of the issue.
Here's some logs that I got from reading CodeGeneratorRequest.ptrs[0]
<https://gist.github.com/Danappelxx/a4536c946e56919ad6765a550a19c1e5>.
I'm leaning on this being my fault somewhere in the parsing but just
wanted to make sure I'm not missing something before I dive deeper.
Hi! I've been working on a Swift compiler plugin for the past week or so,
and have been making decent progress. I've run into a few blind spots in
the encoding spec (i.e. unions) but seemed to have gotten over the major
humps.
Just wanted to let people know in case somebody has tips or concerns.
I'll also use this thread as a place to ask any future questions that I
may have :)
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.
--
Dan Appel
--
Dan Appel
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[David Renshaw]

Post by Dan Appel
Does this mean that every field (including lists) is nullable? That's
quite a shame. Looks like all my generated fields are going to be
implicitly-unwrapped-optional, then (reminds me of objective-c translated
into swift).

All pointer fields can be null. Because all types in Cap'n Proto have a
default value, you don't need to return optionals; you can return the
default value when a `getFoo()` accessor reads a null. You should
additionally provide `hasFoo()` accessors for when the user actually wants
to distinguish the null case.

However, if you were decoding Cap'n Proto messages into stand-alone
Swift-native plain-old-data structs, then yes, you would in general need to
make pointer fields optional, because otherwise values of a recursive type
(e.g. `struct Bar { bar @0 :Bar; }`) would be infinitely large.

- David

[Dan Appel]

Post by David Renshaw
All pointer fields can be null. Because all types in Cap'n Proto have a

default value, you don't need to return optionals; you can return the
default value when a `getFoo()` accessor reads a null. You should
additionally provide `hasFoo()` accessors for when the user actually wants
to distinguish the null case.

I'm a little confused by the documentation on this one. What is the default
value of a struct? This section
<https://capnproto.org/encoding.html#default-values> seems to only be about
structs inside lists, but even so I'm struggling to understand where I
should be looking. If I have an all-zero pointer, where is the default
value stored? Where would I store the value a user sets?

Post by David Renshaw
However, if you were decoding Cap'n Proto messages into stand-alone

Swift-native plain-old-data structs, then yes, you would in general need to
make pointer fields optional, because otherwise values of a recursive type
(e.g. `struct Bar { bar @0 :Bar; }`) would be infinitely large.

Gotcha. Unrelated note: making the properties optional does not work in
swift, the following does not compile (recursive values need to be wrapped
in a reference type, optionals are still a value type. You would need a
`class Box` of some sort, which is messy)
struct Foo {
let foo: Foo?
}

Dan

Post by David Renshaw

Post by Dan Appel
Does this mean that every field (including lists) is nullable? That's
quite a shame. Looks like all my generated fields are going to be
implicitly-unwrapped-optional, then (reminds me of objective-c translated
into swift).

All pointer fields can be null. Because all types in Cap'n Proto have a
default value, you don't need to return optionals; you can return the
default value when a `getFoo()` accessor reads a null. You should
additionally provide `hasFoo()` accessors for when the user actually wants
to distinguish the null case.
However, if you were decoding Cap'n Proto messages into stand-alone
Swift-native plain-old-data structs, then yes, you would in general need to
make pointer fields optional, because otherwise values of a recursive type
- David
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[David Renshaw]

Post by David Renshaw

Post by David Renshaw
All pointer fields can be null. Because all types in Cap'n Proto have a

default value, you don't need to return optionals; you can return the
default value when a `getFoo()` accessor reads a null. You should
additionally provide `hasFoo()` accessors for when the user actually wants
to distinguish the null case.
I'm a little confused by the documentation on this one. What is the
default value of a struct?

The default value of a struct type is a struct whose fields all contain
their default values.

Post by David Renshaw
This section <https://capnproto.org/encoding.html#default-values> seems
to only be about structs inside lists, but even so I'm struggling to
understand where I should be looking. If I have an all-zero pointer, where
is the default value stored?

There's no data to store. It's as if the struct has an empty data section
and an empty pointer section.


Where would I store the value a user sets?
Ah, so we're considering mutations now, and hence Builders, which differ
significantly from Readers. If the user calls `get()` on a null pointer
field in a Builder, you do need to actually allocate space for that field,
in preparation for the user possibly setting values on it. Note that once
you do this, the field will no longer be null.


- David

[Kenton Varda]

FWIW, it's on my wishlist to add Maybe(T) as a built-in type. Then, we can
get rid of "has" methods -- Maybe(T) would generate getters which use the
language's preferred "maybe" idiom, whereas non-maybe T would generate a
getter which returns the default value if the pointer is null (with no way
to actually test if the pointer is null).

BTW, note that while the "default default" value of a struct is the
zero-sized struct, it is possible for a schema to specify some other
default for a specific struct field. Example:

struct Foo {
i @0 :Int32 = 123;
}

struct Bar {
baz @0 :Foo = (i = 456);
}

A typical implementation is to store "baz"'s default value as a static byte
buffer somewhere, and then when getBaz() is called and the pointer is null,
act like it somehow pointed to that static buffer (or in the case of
builders, initialize it to a copy of that static buffer).

It's pretty unusual for people to use this feature in practice, though.

-Kenton

Post by David Renshaw

Post by David Renshaw

Post by David Renshaw
All pointer fields can be null. Because all types in Cap'n Proto have a

default value, you don't need to return optionals; you can return the
default value when a `getFoo()` accessor reads a null. You should
additionally provide `hasFoo()` accessors for when the user actually wants
to distinguish the null case.
I'm a little confused by the documentation on this one. What is the
default value of a struct?

The default value of a struct type is a struct whose fields all contain
their default values.

Post by David Renshaw
This section <https://capnproto.org/encoding.html#default-values> seems
to only be about structs inside lists, but even so I'm struggling to
understand where I should be looking. If I have an all-zero pointer, where
is the default value stored?

There's no data to store. It's as if the struct has an empty data section
and an empty pointer section.
Where would I store the value a user sets?
Ah, so we're considering mutations now, and hence Builders, which differ
significantly from Readers. If the user calls `get()` on a null pointer
field in a Builder, you do need to actually allocate space for that field,
in preparation for the user possibly setting values on it. Note that once
you do this, the field will no longer be null.
- David
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[deansher via Cap'n Proto]

I'm considering using Cap'n Proto for a Swift project. I wouldn't mind picking up a partial implementation and completing it. Dan, or anyone, did you end up committing anything?

Dean

[Dan Appel]

I left it incomplete - I had decoding implemented but did not finish
working on traversing the schema and codegen. I could clean up what I have
and push it on GitHub if you'd like a starting point :)

Dan

On Mon, May 1, 2017 at 8:12 AM deansher via Cap'n Proto <

Post by deansher via Cap'n Proto
I'm considering using Cap'n Proto for a Swift project. I wouldn't mind
picking up a partial implementation and completing it. Dan, or anyone, did
you end up committing anything?
Dean
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

['Dean Thompson' via Cap'n Proto]

The way my project has shaped up, I won’t immediately be working on Cap’n
Proto over Swift. I’d love to know it’s out there, if you feel like
cleaning it up and pushing it to GitHub, but I have no near-term momentum
to offer as incentive.

Post by Dan Appel
I left it incomplete - I had decoding implemented but did not finish
working on traversing the schema and codegen. I could clean up what I have
and push it on GitHub if you'd like a starting point :)
Dan
On Mon, May 1, 2017 at 8:12 AM deansher via Cap'n Proto <

Post by deansher via Cap'n Proto
I'm considering using Cap'n Proto for a Swift project. I wouldn't mind
picking up a partial implementation and completing it. Dan, or anyone, did
you end up committing anything?
Dean
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[d***@gmail.com]

Dan, I’m now at a point where I’d work on this. Would you be up for throwing what you have into GitHub?

Dean

[d***@gmail.com]

Did this code ever go anywhere, or should I consider it dead for Swift?

I have a mixed product in Swift and C++ so am looking at
mutually-compatible exchange options.

[Dan Appel]

Dean, sorry I missed your email! I just threw it up, though I realize it's
been a couple months and it may be significantly less valuable now. The
runtime code should have stuff of interest, but the generator code can
largely be ignored.

Andy, there's nothing production ready available AFAIK, but if you're
willing to take on the weight of implementing it yourself there's a bunch
of people willing to help!

What I have so far: link <https://github.com/Danappelxx/capnpc-swift>

Dan

Post by d***@gmail.com
Did this code ever go anywhere, or should I consider it dead for Swift?
I have a mixed product in Swift and C++ so am looking at
mutually-compatible exchange options.
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[d***@gmail.com]

Dan

Thanks for the source link. I doubt I will find the time to do much for a
month as I'm off to a US conference (Social Media Marketing World in San
Diego) with a killer schedule before but am very interested. My first
product release is just Swift/SpriteKit in iMessage so I can defer getting
the packet format solid and just use Codable.

I have a long background in this kind of stuff, mostly in C++, with my old
OOFILE framework providing high speed mapping direct to the ISAM data store
buffers. (I still remember fun times with the SparcStation users running
into offset-based crashes.)

I've done a lot of code generation work in the past, using languages
ranging from a weird TCL variant to Ruby and XSLT.

My Realm experience may also help - I spent nearly two years on the C#
team, with our SDK mapping to the C++ core. We used Fody, an IL-generator,
to do compile-time reflection of RealmObject subclasses and generate schema
and inject mapping calls.

One strategy for the Swift parsing side I'm musing is using Codable itself
and writing the generator as a custom encoder. Mike Ash wrote an [article
about his binary
encoder](https://www.mikeash.com/pyblog/friday-qa-2017-07-28-a-binary-coder-for-swift.html)
which is on my list to play with too.

Post by Dan Appel
Andy, there's nothing production ready available AFAIK, but if you're
willing to take on the weight of implementing it yourself there's a bunch
of people willing to help!