[capnproto] RPC Level 2 questions - SturdyRefs, restorers, authentication and encryption

[Thomas Leonard]

Hi,

I'm currently trying to implement RPC level 2 (for the OCaml RPC
implementation - see
https://github.com/mirage/capnp-rpc#encryption-and-authentication for the
current status).

I have some questions...

https://capnproto.org/cxxrpc.html says:

Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is

a Level 1 implementation. Persistent capabilities, three-way introductions,
and distributed equality are not yet implemented.

But I imagine this is out of date.

The RPC spec says:

How exactly a SturdyRef is restored to a live object is specified along

with the SturdyRef definition (i.e. not by rpc.capnp).

and

However, in practice, the ability to restore SturdyRefs is itself a

capability that may require going through an authentication process to
obtain. Thus, it makes more sense to define a "restorer service" as a full
Cap'n Proto interface. If this restorer interface is offered as the vat's
bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if
every SturdyRef has its own restorer API then an RPC implementation won't
know how to authenticate to it when the user does something like:

liveRef := sturdyRef.getRcvr()

Is this restorer API specified somewhere? The Python docs mention an
*ez_restore* method and say "Refer to the Cap’n Proto docs if you don’t
know what this means", but it's not clear to me what I should be looking at.

For now, my SturdyRefs can only refer to bootstrap services.

Also, I couldn't find much about authentication.
https://capnproto.org/rpc.html#encryption says:

At this time, Cap’n Proto does not specify an encryption scheme, but as it

is a simple byte stream protocol, it can easily be layered on top of
SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just
as a container for the public key) and a validator that checks a hash of
the server key fingerprint. My SturdyRefs convert to URIs that currently
look like this:

capnp://sha-256:s16WV4JeGusAL_nTjvICiQOFqm3LqYrDj3K-***@127.0.0.1:7000

(based on the format at
http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind
)

However, https://capnproto.org/roadmap.html says:

Cap’n Proto RPC should support an encrypted transport which uses

capability-based authorization (not PKI), can accomplish zero-round-trip
three-party introductions (via a pre-shared key from the introducer) and
based on modern crypto. TLS is not designed for this, but we don’t want to
invent new crypto; we intend to build on libsodium and the Noise Protocol
Framework as much as possible.

For interoperability, I allowed "insecure@" to disable encryption. e.g. to
connect my OCaml client to the C++ calculator service I can use:

capnp://***@127.0.0.1:7000

I treat an empty host:port section as requesting a Unix-domain socket, e.g.

capnp://insecure@/tmp/calc

It's a bit of a hack, but I'm not sure what would be better.

Things I'd like to know:

- Do any other implementations support authentication or encryption?

- Has anyone else defined a URI format I can use for sturdy refs?

- Where in the URI should I put the service ID for the restorer server?

- Is there a spec defining what a StudyRef and restorer service should look
like for the default public Internet?
If not, is there anything obvious I should change in my current
implementation to match what this will look like?

Thanks,

[Thomas Leonard]

Post by Thomas Leonard
Hi,
I'm currently trying to implement RPC level 2 (for the OCaml RPC
implementation - see
https://github.com/mirage/capnp-rpc#encryption-and-authentication for the
current status).
I have some questions...
Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is

a Level 1 implementation. Persistent capabilities, three-way introductions,
and distributed equality are not yet implemented.

But I imagine this is out of date.
How exactly a SturdyRef is restored to a live object is specified along

with the SturdyRef definition (i.e. not by rpc.capnp).

and
However, in practice, the ability to restore SturdyRefs is itself a

capability that may require going through an authentication process to
obtain. Thus, it makes more sense to define a "restorer service" as a full
Cap'n Proto interface. If this restorer interface is offered as the vat's
bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if
every SturdyRef has its own restorer API then an RPC implementation won't
liveRef := sturdyRef.getRcvr()
Is this restorer API specified somewhere? The Python docs mention an
*ez_restore* method and say "Refer to the Cap’n Proto docs if you don’t
know what this means", but it's not clear to me what I should be looking at.
For now, my SturdyRefs can only refer to bootstrap services.

It seems that this uses the "deprecatedObjectId". The spec says it's
deprecated because:

1. "in practice, the ability to restore SturdyRefs is itself a capability
that may require going through an authentication process to obtain."

2. "the earliest version of the EZ RPC interfaces set a precedent of
exporting multiple main interfaces by allowing them to be exported under
string names"

3. "Often, "main" or "well-known" capabilities exported by a vat are in
fact not public: they are intended to be accessed only by clients who are
capable of forming a connection to the vat." and "the supervisor has to be
careful not to honor an external request addressing the application's
default capability".

However:

1. If a sturdy ref is a capability reference then holding it should be
sufficient to use it, without needing additional capabilities. It's not
clear why the spec thinks this isn't enough.

2. It's not clear why one implementation using string names means the
protocol needs to change.

3. This seems like a sandstorm-specific problem, and could be fixed in one
place (the supervisor). As a safer alternative, sandstorm apps could
instead use a pre-exported "export 0" for their admin API. The protocol
should already prevent external clients from getting access to unrelated
exports.

Given the lack of an alternative, I've implemented this "deprecated" scheme
in the OCaml version now. It seems to work fine, and works with the other
implementations.

Post by Thomas Leonard
Also, I couldn't find much about authentication.
At this time, Cap’n Proto does not specify an encryption scheme, but as it

is a simple byte stream protocol, it can easily be layered on top of
SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just
as a container for the public key) and a validator that checks a hash of
the server key fingerprint. My SturdyRefs convert to URIs that currently
capnp://
(based on the format at
http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind
)
Cap’n Proto RPC should support an encrypted transport which uses

capability-based authorization (not PKI), can accomplish zero-round-trip
three-party introductions (via a pre-shared key from the introducer) and
based on modern crypto. TLS is not designed for this, but we don’t want to
invent new crypto; we intend to build on libsodium and the Noise Protocol
Framework as much as possible.

I treat an empty host:port section as requesting a Unix-domain socket, e.g.
It's a bit of a hack, but I'm not sure what would be better.
- Do any other implementations support authentication or encryption?
- Has anyone else defined a URI format I can use for sturdy refs?
- Where in the URI should I put the service ID for the restorer server?

I'm now storing the service ID, base64-encoded, as the final path component
in the URI.

Post by Thomas Leonard
- Is there a spec defining what a StudyRef and restorer service should
look like for the default public Internet?
If not, is there anything obvious I should change in my current
implementation to match what this will look like?

I also have some additional questions now:

- The schema language doesn't define a SturdyRef type. Since it seems that
interfaces shouldn't contain network-specific types, this means that
SturdyRefs currently have to be defined as AnyPointer. It would be useful
if the spec could pre-define this abstract (and generic) type. Then schema
compilers could output type-safe code for building and reading them.

- It seems that sturdy refs are intended to be stored as plain data.
However, this is usually readable by the application code. Perhaps they
should instead go in the CapDescriptor table?

The spec says:

"we specify several "parameter" types. Each type is defined here as an
alias for `AnyPointer`, but a specific network will want to define a
specific set of types to use. All vats in a vat network must agree on
these parameters in order to be able to communicate. Inter-network
communication can be accomplished through "gateways" that perform
translation between the primitives used on each network; these gateways may
need to be deeply stateful, depending on the translations they perform."

Does "deeply stateful" mean that gateways need to understand the schema to
know which fields are sturdy refs?

['Kenton Varda' via Cap'n Proto]

Hi Thomas,

Sorry I missed this earlier!

So, "level 2" of the RPC protocol / SturdyRefs turn out to be something
that does not make sense to specify as part of the Cap'n Proto
implementation itself. Probably level 2 should be ripped out of the spec
entirely. This is what we learned as we build Sandstorm: SturdyRefs and how
to restore them turned out to be intrinsically tied to the Sandstorm
environment, and attempts to define an "abstract" SturdyRef format not
dependent on Sandstorm did not seem to fit what Sandstorm needed.

To understand this, consider a few different cases:
- A Sandstorm grain (app instance).
- A node in the Blackrock (clustered Sandstorm) infrastructure.
- An application on the general internet that has nothing to do with
Sandstorm.

Now try to answer the question: What does a SturdyRef express, and how does
one restore it?

The answer is totally different depending on the context:

- For a Sandstorm grain, a SturdyRef can be an opaque byte string, which
refers to an object in another grain. The client grain passes the SturdyRef
to the Sandstorm API to restore it. The Sandstorm infrastructure then looks
up the token in its database, verifies that the token belongs to the
requesting grain, finds out to what grain the token points, starts up that
grain, asks that grain for a live ref of the desired capability, and then
returns that to the requesting grain.

- For a Blackrock node, a SturdyRef typically refers to another component
of the Blackrock infrastructure: maybe an object in Blackrock storage (a
graph store of Cap'n Proto objects), or a running container on one of the
worker nodes. Or, it could also refer to something hosted in a grain, or a
totally external capability. See the definition here:
https://github.com/sandstorm-io/blackrock/blob/master/src/blackrock/cluster-rpc.capnp#L67
Notice how the namespace of SturdyRefs as seen by the infrastructure itself
is completely different from the namespace of SturdyRefs seen by apps --
although some kinds of objects can be represented by both. Also notice that
depending on the type of SturdyRef, the process for restoring is different:
for "transient" objects located on a specific machine, the restorer
connects directly to the target, but for stored object, the restorer
connects to "the storage service" which it is introduced to independently
at startup, and for external caps, it connects to "the gateways", etc.

- For the public internet, you probably want a SturdyRef to encode a
hostname and perhaps a pinned certificate list. Maybe it even encodes an
HTTP URL, to which a Cap'n Proto session can be created over WebSocket or
streaming HTTP/2. Additionally, it would encode some sort of object ID,
probably as an AnyPointer. The target host would provide a bootstrap
interface with a restore() method that takes this AnyPointer.

As you can see, in each case the format of a SturdyRef and the procedure
for restoring it is completely different, so much so that it doesn't appear
that any "standard" definition makes sense.

At some point I do want to spec out the "public internet" SturdyRef format
and protocol. But, for now, I think implementations should leave SturdyRefs
up to the application to define.

On a side note, it seems like you were confused a bit by EZ RPC's mechanism
for exporting capabilities by name. We deprecated this in favor of a
singleton bootstrap interface because you can trivially implement the same
thing by defining a bootstrap interface with a "restore(name)" method.

-Kenton

Post by Thomas Leonard
Hi,
I'm currently trying to implement RPC level 2 (for the OCaml RPC
implementation - see https://github.com/mirage/capnp-rpc#encryption-and-
authentication for the current status).
I have some questions...
Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is

a Level 1 implementation. Persistent capabilities, three-way introductions,
and distributed equality are not yet implemented.

But I imagine this is out of date.
How exactly a SturdyRef is restored to a live object is specified along

with the SturdyRef definition (i.e. not by rpc.capnp).

and
However, in practice, the ability to restore SturdyRefs is itself a

capability that may require going through an authentication process to
obtain. Thus, it makes more sense to define a "restorer service" as a full
Cap'n Proto interface. If this restorer interface is offered as the vat's
bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if
every SturdyRef has its own restorer API then an RPC implementation won't
liveRef := sturdyRef.getRcvr()
Is this restorer API specified somewhere? The Python docs mention an
*ez_restore* method and say "Refer to the Cap’n Proto docs if you don’t
know what this means", but it's not clear to me what I should be looking at.
For now, my SturdyRefs can only refer to bootstrap services.
Also, I couldn't find much about authentication.
At this time, Cap’n Proto does not specify an encryption scheme, but as it

is a simple byte stream protocol, it can easily be layered on top of
SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just
as a container for the public key) and a validator that checks a hash of
the server key fingerprint. My SturdyRefs convert to URIs that currently
127.0.0.1:7000
(based on the format at http://iiw.idcommons.net/
HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind)
Cap’n Proto RPC should support an encrypted transport which uses

capability-based authorization (not PKI), can accomplish zero-round-trip
three-party introductions (via a pre-shared key from the introducer) and
based on modern crypto. TLS is not designed for this, but we don’t want to
invent new crypto; we intend to build on libsodium and the Noise Protocol
Framework as much as possible.

I treat an empty host:port section as requesting a Unix-domain socket, e.g.
It's a bit of a hack, but I'm not sure what would be better.
- Do any other implementations support authentication or encryption?
- Has anyone else defined a URI format I can use for sturdy refs?
- Where in the URI should I put the service ID for the restorer server?
- Is there a spec defining what a StudyRef and restorer service should
look like for the default public Internet?
If not, is there anything obvious I should change in my current
implementation to match what this will look like?
Thanks,
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.

[Thomas Leonard]

Post by 'Kenton Varda' via Cap'n Proto
Hi Thomas,
Sorry I missed this earlier!
So, "level 2" of the RPC protocol / SturdyRefs turn out to be something that
does not make sense to specify as part of the Cap'n Proto implementation
itself. Probably level 2 should be ripped out of the spec entirely. This is
what we learned as we build Sandstorm: SturdyRefs and how to restore them
turned out to be intrinsically tied to the Sandstorm environment, and
attempts to define an "abstract" SturdyRef format not dependent on Sandstorm
did not seem to fit what Sandstorm needed.
- A Sandstorm grain (app instance).
- A node in the Blackrock (clustered Sandstorm) infrastructure.
- An application on the general internet that has nothing to do with
Sandstorm.
Now try to answer the question: What does a SturdyRef express, and how does
one restore it?

At the application level:

It represents the ability to get a live capability reference to a
service. You restore it by calling its "connect" method (which is
possibly the only method it has).

At the vat/network level: whatever the network defines it as.

(using "SturdyRef" to mean both of these things is perhaps confusing though)

Post by 'Kenton Varda' via Cap'n Proto
- For a Sandstorm grain, a SturdyRef can be an opaque byte string, which
refers to an object in another grain. The client grain passes the SturdyRef
to the Sandstorm API to restore it. The Sandstorm infrastructure then looks
up the token in its database, verifies that the token belongs to the
requesting grain, finds out to what grain the token points, starts up that
grain, asks that grain for a live ref of the desired capability, and then
returns that to the requesting grain.
- For a Blackrock node, a SturdyRef typically refers to another component of
the Blackrock infrastructure: maybe an object in Blackrock storage (a graph
store of Cap'n Proto objects), or a running container on one of the worker
nodes. Or, it could also refer to something hosted in a grain, or a totally
https://github.com/sandstorm-io/blackrock/blob/master/src/blackrock/cluster-rpc.capnp#L67

That's a useful link, thanks!

Post by 'Kenton Varda' via Cap'n Proto
If the sender's public key
is less than the receiver's, this number must be even, otherwise it must be odd, so that
connection IDs in opposite directions between the same vats never collide. Any existing
connections with lower connection IDs must be invalidated when a new connection starts.

That's useful to have specified. For my implementation, the rule I
used was "the connection to keep is the one initiated by the peer with
the highest vat ID." But your scheme looks better, as it copes with
one vat losing a connection and retrying while the other thinks the
old connection is still OK. Can I use this header format with e.g. the
Python client? My current code does have the advantage of being just
plain TLS with client and server certificates.

Post by 'Kenton Varda' via Cap'n Proto
Notice how the namespace of SturdyRefs as seen by the infrastructure itself
is completely different from the namespace of SturdyRefs seen by apps --
although some kinds of objects can be represented by both. Also notice that
for "transient" objects located on a specific machine, the restorer connects
directly to the target, but for stored object, the restorer connects to "the
storage service" which it is introduced to independently at startup, and for
external caps, it connects to "the gateways", etc.
- For the public internet, you probably want a SturdyRef to encode a
hostname and perhaps a pinned certificate list. Maybe it even encodes an
HTTP URL, to which a Cap'n Proto session can be created over WebSocket or
streaming HTTP/2. Additionally, it would encode some sort of object ID,
probably as an AnyPointer. The target host would provide a bootstrap
interface with a restore() method that takes this AnyPointer.

Yes, this is roughly what I have (though I encoded it as a simple
string URL, since it supports that form anyway).
It would be very useful to specify this so that different
implementations can talk to each other.

For example, I'd like to be able to cut-and-paste a URL from an OCaml
service and connect to it using the Python client.

Post by 'Kenton Varda' via Cap'n Proto
As you can see, in each case the format of a SturdyRef and the procedure for
restoring it is completely different, so much so that it doesn't appear that
any "standard" definition makes sense.

At the network level, yes. But at the application level it still makes
sense to have an abstract SturdyRef type in the schema language and in
the API I think. And it should be possible for applications to
exchange sturdy refs in messages without knowing what kind of network
their vats use.

Post by 'Kenton Varda' via Cap'n Proto
At some point I do want to spec out the "public internet" SturdyRef format
and protocol. But, for now, I think implementations should leave SturdyRefs
up to the application to define.

I don't see how this can work. For the public internet, I need to know
what to connect to, what protocol to speak (TCP, TLS, HTTP, etc) and
how to authenticate the peer. The object ID can be opaque, but the
rest must be known.

Post by 'Kenton Varda' via Cap'n Proto
On a side note, it seems like you were confused a bit by EZ RPC's mechanism
for exporting capabilities by name. We deprecated this in favor of a
singleton bootstrap interface because you can trivially implement the same
thing by defining a bootstrap interface with a "restore(name)" method.

I could. But then I'm just replacing a somewhat standard interface
with a non-standard one that the other clients don't have built-in
support for. It doesn't seem like an improvement.

Post by 'Kenton Varda' via Cap'n Proto
-Kenton

Post by Thomas Leonard
Hi,
I'm currently trying to implement RPC level 2 (for the OCaml RPC
implementation - see
https://github.com/mirage/capnp-rpc#encryption-and-authentication for the
current status).
I have some questions...

Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation
is a Level 1 implementation. Persistent capabilities, three-way
introductions, and distributed equality are not yet implemented.

But I imagine this is out of date.

How exactly a SturdyRef is restored to a live object is specified along
with the SturdyRef definition (i.e. not by rpc.capnp).

and

However, in practice, the ability to restore SturdyRefs is itself a
capability that may require going through an authentication process to
obtain. Thus, it makes more sense to define a "restorer service" as a full
Cap'n Proto interface. If this restorer interface is offered as the vat's
bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if
every SturdyRef has its own restorer API then an RPC implementation won't
liveRef := sturdyRef.getRcvr()
Is this restorer API specified somewhere? The Python docs mention an
ez_restore method and say "Refer to the Cap’n Proto docs if you don’t know
what this means", but it's not clear to me what I should be looking at.
For now, my SturdyRefs can only refer to bootstrap services.
Also, I couldn't find much about authentication.

At this time, Cap’n Proto does not specify an encryption scheme, but as
it is a simple byte stream protocol, it can easily be layered on top of
SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just
as a container for the public key) and a validator that checks a hash of the
server key fingerprint. My SturdyRefs convert to URIs that currently look
(based on the format at
http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind)

Cap’n Proto RPC should support an encrypted transport which uses
capability-based authorization (not PKI), can accomplish zero-round-trip
three-party introductions (via a pre-shared key from the introducer) and
based on modern crypto. TLS is not designed for this, but we don’t want to
invent new crypto; we intend to build on libsodium and the Noise Protocol
Framework as much as possible.

I treat an empty host:port section as requesting a Unix-domain socket, e.g.
It's a bit of a hack, but I'm not sure what would be better.
- Do any other implementations support authentication or encryption?
- Has anyone else defined a URI format I can use for sturdy refs?
- Where in the URI should I put the service ID for the restorer server?
- Is there a spec defining what a StudyRef and restorer service should
look like for the default public Internet?
If not, is there anything obvious I should change in my current
implementation to match what this will look like?
Thanks,
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/capnproto.