AnnNet/README.adoc

AnnNet Specification

Table of Contents

• 1. License
• 2. Reference Library
• 3. Disclaimer
  • 3.1. The AnnNet Protocol
  • 3.2. References
• 4. Protocol
• 5. Specification
  • 5.1. AnnNet Message Format
    • 5.1.1. Standard Prefix
      • 5.1.1.1. Version
        • 5.1.1.1.1. Major
        • 5.1.1.1.2. Minor
        • 5.1.1.1.3. Patch
        • 5.1.1.1.4. Release Flag(s)
      • 5.1.1.2. Payload Length
    • 5.1.2. Payload
  • 5.2. AnnNet Link
  • 5.3. AnnNet Broadcast
    • 5.3.1. IPv4
      • 5.3.1.1. Message Format
    • 5.3.2. IPv6
      • 5.3.2.1. Message Format
    • 5.3.3. Internet Protocol Number
• 6. Limitations
• 7. (F)AQ/FUQ (Anticipated, Frequently Asked/Anticipated/Unasked Questions)
  • 7.1. Why do you reference the OSI model instead of the TCP/IP model? It’s old/outdated/inaccurate.
  • 7.2. Why Shouldn’t I Use AnnNet for Regular Traffic?
• 8. Addendum/Errata/Administrivia
  • 8.1. Bitpacked Fields
    • 8.1.1. IP Version, IHL (IPv4)
    • 8.1.2. DSCP, ECN (IPv4)
    • 8.1.3. Flags, Fragmentation Offset (IPv4)
    • 8.1.4. IP Version, Traffic Class, Flow Label (IPv6)

1. License

Unresolved directive in - include::LICENSE.html[]

In a nutshell, this means any may:

• Use it in commercial/proprietary/internal works…​

• Expand upon/change the specification…​

  • (As long as it is released under the same Creative Commons license)

As long as you attribute the original (this document). This can be as simple as something like:

Based on AnnNet version <protocol version> as found at https://annnet.io/.

More details certainly helps, though; you may want to mention the exact date you "forked" it, etc.

Please see the full text as collapsed above or the online version of the license for full legal copy.

Note	In the event of the embedded text in this document differing from the online version, the online version is assumed to take precedence as the valid license applicable to this work.

2. Reference Library

The AnnNet Protocol is accompanied by a reference library for Golang, "AnnNet" (source):

It contains a reference "client" (listener), annie (source), and a reference "server" (announcer), anna (source).

Additional reference libraries may be available in the future.

3. Disclaimer

3.1. The AnnNet Protocol

This specification is provided as-is with no guarantees or obligations from this specification’s author(s).

The author(s) is/are not held liable for any damages caused by implementation/use of this protocol.

The AnnNet specification is subject to change, but changes are not considered definitive nor effective until/unless tied with/to a specific released version.

3.2. References

Many outside references are made throughout this document.

• The author(s) is/are not sponsored by, contribute to, nor condone any of the resources mentioned unless otherwise specified.

• The author(s) is/are not sponsored by, nor member(s),contributor(s), or sponsor of, nor condone any organizations or other entities listed or their actions unless otherwise specified.

References are suggested as-is with no guarantees or obligations from this specification’s author(s).

4. Protocol

The AnnNet (pronounced like the woman’s name "Annette") protocol is an announcement/query protocol designed for unspecified hosts on a local network. It is a shorthand for "the Announcement Network Protocol".

AnnNet is designed to operate with or without IP addressing being available, as it can operate on layer 2, layer 3, or both to unknown/unspecified destinations.

It is intended as a sort of "announcement" mechanism for sending messages to all hosts on a locally-linked (or single-organization-controlled) network (such as bootstrapping large-scale layer 3 services/clusters). It was designed with service/implementation-specific host discovery in mind, though other uses are possible.

It is recommended that it is only used sparingly for small messages (such as cluster bootstrapping for new members or initial cluster assembly). Operational messages should occur on a well-defined transport layer (e.g. TCP).
It is important to avoid the temptation of simply implementing ALL of a service’s communication via AnnNet, as this can quite easily overwhelm the physical network’s limitations as hosts participating in its communication grows. See the FAQ/FUQ entry Why Shouldn’t I Use AnnNet for Regular Traffic? for suggestions on how to manage actual service communication instead.

Note

The OSI (Open Systems Interconnection) model is going to be referenced a fair bit in this document. The linked Wikipedia article is a fantastic primer on it with a plethora of useful links, but if you’d like to read the authoritative material that’d be ISO/IEC 7498-1:1994. ISO/IEC 7498-1:1994 is one of the free-as-in-beer-for-digital-copy ISO/IEC standards available, provided you have an ISO account — which is also free to create and use. You can use the Register link at the bottom of ISO’s Webstore login to create a new purchasing account. Another fantastic reference for the OSI model (and all things networking) is the TCP/IP Guide by Charles M. Kozierok. (I am not sponsored by nor condone the material found in that re) It is available: In its entirety for free (-as-in-beer) online (if you don’t mind the large amount of ads) Note that the online version may not map perfectly to the print/PDF versions, and may have different wording. I (Brent) highly recommend supporting the author by either using the donation links at the bottom of that page or purchasing/licensing a donwloaded PDF or print copy, see below. By purchasing a PDF download directly from the author Which also allows for bulk licensing of the e-book for orginaztional use. By purchasing a print and/or PDF copy from No Starch Press As expected with all other No Starch Press book purchases, a print copy comes with a free digital/e-book copy. I (Brent) have a physical copy from No Starch Press. I find myself referencing the PDF much more often, as it has some very useful document linking within it, but moreso because the print copy is a behemoth and is very bulky for anything but a bookshelf and seated-at-a-table reading/research! The OSI model is covered in the TCP/IP Guide in Part I-2 (chapters 5 to 7 inclusive), starting on p.79. If using the online version, Part I-2 is summarized/starts here.

5. Specification

AnnNet supports two "sub-protocols":

• AnnNet Link

• AnnNet Broadcast

Both make use of the Standard Prefix, though in different places of the frame/packet depending on their implementation. See each sub-protocol below for details.

5.1. AnnNet Message Format

The general format of an AnnNet message (excluding leading/surrounding/trailing frames, headers, etc.) is:

Table 1. AnnNet Message

Standard Prefix (5 Bytes)

Payload

5.1.1. Standard Prefix

AnnNet defines a specific prefix that must be added before a message. While AnnNet does not dictate the contents of the payload in any way, all AnnNet messages must contain this prefix.

In AnnNet Link, it immediately follows the EtherType value.

In AnnNet Broadcast, it immediately follows the IPv4/IPv6 header.

In its current form, this is:

Table 2. AnnNet Prefix

Version
Major	Minor	Patch	Release Flag(s)	Payload Length

5.1.1.1. Version

Length (Octets/Bytes)	Type
4	Big-Endian Unsigned 32-Bit Integer/Container

The Version field matches the version of AnnNet being used.

Because AnnNet follows Semantic Versioning (v2.0.0) for its specification, this can either be treated as an abstract container (hereafter a "byte-slice version" in this spec) of individual bytes indicating each version sub-field or treated as a whole (hereafter a "single-value version" in this spec).

Thus for example, when determing how to parse the following fields from an AnnNet message if incompatible structuring changes are introduced, one can either:

• Perform a version comparison sequentially (e.g. via first parsing and comparing the Major, and then moving on to the Minor, and so forth) for a byte-slice version

• Or for a single-value version, a direct integer comparison — for example, if a single-value Version is greater than or equal to 16777216 to check if it is greater than or equal to version 1.0.0 of AnnNet (16777216 being equal to 0x01, 0x00, 0x00, 0x00, or more properly 0x01000000, when big-endian decoded).

Tip	Bit-shifting can be used to extract individual version fields from a single-value version. Example in Go examples/extractversion.go Unresolved directive in <stdin> - include::examples/extractversion.go[]

5.1.1.1.1. Major

Length (Octets/Bytes)	Type
1	Unsigned 8-Bit Integer

The Major is the first byte in Version (if interpreted as a byte slice and not a big-endian unsigned 32-bit integer).

5.1.1.1.2. Minor

Length (Octets/Bytes)	Type
1	Unsigned 8-Bit Integer

The Minor is the second byte in Version (if interpreted as a byte slice and not a big-endian unsigned 32-bit integer).

5.1.1.1.3. Patch

Length (Octets/Bytes)	Type
1	Unsigned 8-Bit Integer

The Patch is the third byte in Version (if interpreted as a byte slice and not a big-endian unsigned 32-bit integer).

Note	The "patch" in SemVer may be more accurate to think of it as a "revision" than a "patch", but the term "revision" means something else in Semantic Version specification so "patch" is used here instead.

5.1.1.1.4. Release Flag(s)

Length (Octets/Bytes)	Type
1	Byte/8-Bit Length Bitmask

The Release Flag(s) field is the fourth byte in Version. It is not intended as an unsigned 8-bit integer but rather a bitmasked byte.

The collection of all flags is coalesced into a single byte on the wire — an unsigned 8-bit integer — with each flag occupying a single bit.

The following flags are currently defined:

Table 3. AnnNet Flags

Integer	Flag	Name
0	ANET_NONE	None; stable release.
1	ANET_PREREL	Pre-release
2	ANET_UNSTABLE	Unstable
4	ANET_EXPERIMENT	Experimental

Additionally, several "shorthand" flags are explicitly defined by the bit-wise OR'd combination of other flags:

Table 4. AnnNet Combined Flags

Integer	Flag	Expression	Name
3	ANET_BETA	ANET_PREREL | ANET_UNSTABLE	Beta release
7	ANET_ALPHA	ANET_BETA | ANET_EXPERIMENT	Alpha release

A flag’s presence can be checked via a bit-wise AND against flag being equal to flag.

Example in Go

hasflag.go

Unresolved directive in <stdin> - include::examples/hasflag.go[]

5.1.1.2. Payload Length

Length (Octets/Bytes)	Type
4	Big-Endian Unsigned 32-Bit Integer

The Payload Length indicates the length of the Payload, which is the arbitrary/implementation-specific data that follows the Standard Prefix (and thus the Payload Length is the last field in the Prefix).

It must be the length of the entirety of the payload that follows (but not including the size of the Prefix).

Note

0 is a valid Payload Length, as is an empty Payload. However, if Payload Length is specified as 0 but there is Payload data, that data must be considered invalid by listening implementations and should not be read/should be discarded (with the exception of validation, testing, or abuse mitigation purposes).

5.1.2. Payload

The Payload is the arbitrary data that follows the Standard Prefix in the AnnNet Message Format.

It is an arbitrary length (specified/bounded by Payload Length) of payload data.

5.2. AnnNet Link

AnnNet Link operates on OSI layer 2 as an EtherType protocol.

Warning

For obvious reasons, this is not going to play well with other EtherTypes (e.g. VLAN/802.1Q tagging). Future versions of AnnNet may implement VLAN support. If VLAN support is added, it will be done so in a dedicated major release.

It may send to either a directed MAC/physical address of a host or to the broadcast MAC/PHY(sical) address ff:ff:ff:ff:ff:ff. Its source must be the MAC/PHYS address of the network interface it is sending from.

It must not traverse physical network boundaries (can’t, really, since it’s layer 2).

It currently uses the 0x88b5 EtherType by default, as AnnNet is still in experimental stages and this EtherType is reserved for prototyping per IEEE Std 802. (0x88b6 is another viable option for your implementation, as it too is reserved by IEEE Std 802 for prototyping.) See also IANA’s registration list.

If AnnNet is allocated a dedicated IEEE/IANA-registered EtherType, it will be implemented in a new major version release.

The AnnNet Message follows the EtherType value immediately:

Table 5. AnnNet Link Packet

Ethernet II Header (14 Bytes)
E:D	E:S	E:T	AnnNet Message

Where:

E:D

Destination MAC/PHY(sical) Address (6 Bytes)

E:S

Source MAC/PHY(sical) Address (6 Bytes)

E:T

EtherType (0x88b5 or 0x88b6) (2 Bytes)

5.3. AnnNet Broadcast

AnnNet Broadcast operates on layer 3, making use of IPv4 (EtherType 0x0800) and/or IPv6 (EtherType 0x86dd).

5.3.1. IPv4

If IPv4, the AnnNet Broadcast sender may send to:

• The IPv4 limited broadcast address 255.255.255.255/32 (RFC 6890 § 2.2.2, 8190 § 2.2)

• Or a direct address of a host, IPv4 link-local (also known as Automatic Private IP Addressing/APIPA, RFC 3927) or not.

  • The target host/network must not traverse organization boundary (e.g. must not be routed directly across the Internet).

If using the 255.255.255.255/32 broadcast address, the source address must be a network-reachable address of the interface the message is sending from.

If using any other address, the source address must be an address reachable by the network/host the message is being sent to.

5.3.1.1. Message Format

The AnnNet Message follows the Options value immediately:

Table 6. AnnNet Broadcast Packet, IPv4

Ethernet II Header (>= 14 Bytes)

IP Header (20-60 Bytes)

V/IHL (1 Byte)

D/E (1 Byte)

Fragmentation (1 Byte)

E:D

E:S

E:T

VI:V

VI:I

DE:D

DE:E

IP:L

IP:I

IP:F

IP:R

IP:T

IP:P

IP:C

IP:S

IP:D

IP:O

AnnNet Message

Where:

E:D

Destination MAC/PHY(sical) Address (6 Bytes)

E:S

Source MAC/PHY(sical) Address (6 Bytes)

E:T

EtherType (0x86dd) (>= 2 Bytes; may be prefixed with VLAN information etc.)

VI:V

Version (4 Bits; see Addendum)

VI:I

Internet Header Length (4 Bits; see Addendum)

DE:D

DSCP (Differentiated Services Code Point; QoS/traffic class) (6 Bits; see Addendum)

DE:E

ECN (Explicit Congestion Notification) (2 Bits; see Addendum)

IP:L

Total size of IP header + payload (not to be confused with Payload Length) (2 Bytes)

IP:I

Identification (used in fragmentation) (2 Bytes)

IP:F

IP Flags (3 bits; see Addendum)

IP:R

Fragmentation Offset (13 Bits; see Addendum)

IP:T

TTL (1 Byte)

IP:P

IP/Transport Protocol Number (1 Byte)

IP:C

Header Checksum (RFC 1071, 1141, 1624) (2 Bytes)

IP:S

Source IPv4 Address (32 Bits/4 Bytes)

IP:D

Destination IPv4 Address (32 Bits/4 Bytes)

IP:O

Options (Optional) (See the IANA Registered IP Parameters for details) (Variable Length)

5.3.2. IPv6

If IPv6, the AnnNet Broadcast sender may send to:

• The multicast addresses (ff0x) with the following scopes (scop) (RFC 3513 § 2.7, 4291 § 2.7, 7346 § 2, see also IANA IPv6 Multicast Registry)

  • except as reserved by other IANA/IETF/IESG-recognized protocols (i.e. OSPFIGP, NDP, et. al.):

    • Scope 1 (0x0001), Interface-Local

    • Scope 2 (0x0002), Link-Local

    • Scope 4 (0x0004), Admin-Local

    • Scope 5 (0x0005), Site-Local

    • Scope 8 (0x0008), Organization-Local

  • A generally safe address would be ff02::1 (All Nodes, Link-Local) but refer to the IANA registry to select the most appropriate multicast address(es).

• Or a direct IPv6 Link-Local Address (LLA) of a host (RFC 3513 § 2.5.6, 3927, 4291 § 2.5.6, 7404 (tangentially))

• Or a direct IPv6 Unique Local Address (ULA) of a host (RFC 4193)

If using a multicast address, the source address must be a network-reachable address of the interface the message is sending from.

If using any other address, the source address must be an address reachable by the network/host the message is being sent to.

5.3.2.1. Message Format

The AnnNet Message follows the Destination Address value immediately:

Table 7. AnnNet Broadcast Packet, IPv6

Ethernet II Header (>= 14 Bytes)			IP Header (40 Bytes)
			V/TC/Flow (4 Bytes)
E:D	E:S	E:T	VTF:V	VTF:T	VTF:F	IP:L	IP:N	IP:H	IP:S	IP:D	AnnNet Message

Where:

E:D

Destination MAC/PHY(sical) Address (6 Bytes)

E:S

Source MAC/PHY(sical) Address (6 Bytes)

E:T

EtherType (0x86dd) (>= 2 Bytes; may be prefixed with VLAN information etc.)

VTF:V

Version (4 Bits; see Addendum)

VTF:T

Traffic Class (8 Bits) (not 1 Byte; see Addendum)

VTF:F

Flow Label (20 bits; see Addendum)

IP:L

IP Payload Length (not to be confused with Payload Length) (2 Bytes)

IP:N

Next Header (transport protocol number) (1 Byte)

IP:H

Hop Limit (1 Byte)

IP:S

Source IPv6 Address (128 Bits/16 Bytes)

IP:D

Destination IPv6 Address (128 Bits/16 Bytes)

5.3.3. Internet Protocol Number

AnnNet currently uses the IPv4 protocol/IPv6 Next Header protocol number 253 (0xfd) by default, as AnnNet is still in experimental stages and this protocol is reserved for experimentation per IANA. (254, 0xfe, is another viable option for your implementation as it too is reserved by IANA for experimentation.)

If AnnNet is allocated a dedicated IANA-registered Protocol/Next Header, it will be implemented in a new major version release.

6. Limitations

1. AnnNet messages must fit into a single frame on the network it is being sent/received on. If message continuation or longer messages are wanted/desired, then it is up to the implementer to add a mechanism for this within their implementation’s payload.

   1. Likewise for checksumming; while most NICs will automatically handle the standard CRC32 Frame Check Sequence at the end of the entire frame (AnnNet Link is still Ethernet II), if you require checksumming within your application for payloads it must be accounted for by your application. If you are using hardware that doesn’t automatically append/strip the FCS, your application must also account for that as well.

   2. For AnnNet Broadcast over IPv4, ensure that you are including the 16-bit "Internet checksum" of the header (see RFCs 1071, 1141, and 1624).
      Most kernels can/will automatically insert this (e.g. on Linux, with IPPROTO_RAW/raw sockets with the IP_HDRINCL option, see raw(7)), but if your implementation doesn’t/can’t use such a mechanism then you must do so yourself.
      IPv6 does not require a header checksum.

7. (F)AQ/FUQ (Anticipated, Frequently Asked/Anticipated/Unasked Questions)

7.1. Why do you reference the OSI model instead of the TCP/IP model? It’s old/outdated/inaccurate.

Hold onto your seat.

Several reasons:

1. The OSI model is standardized. There is only a single, canonical, authoritative "OSI model" when it comes to network models.

   References are unambiguous. Boundaries between layers and their definitions are mostly clear. (And the ISO/IEC provides supplementals for any ambiguities that arise over the years.)

   As for "the TCP/IP model", which one are you referring to?

   There’s no less than seven different models that are referred to as "the TCP/IP model". (The linked-to article doesn’t even mention e.g. the DoD model.) And they’re all different in ways enough to make this confusing.

2. The OSI model is more nuanced and detailed.

   When one is talking about IP protocols and service protocols in the same context, things get very confusing very quickly. One can say "protocol" and it can apply to either ambiguously.

   If instead one says "L2 protocol" or "layer 2 protocol" it’s immediately clear that it operates without (IP) addressing, or "L3 protocol"/"layer 3 protocol" and it’s immediately clear that a protocol that DOES use (IP) addressing is being referenced.

7.2. Why Shouldn’t I Use AnnNet for Regular Traffic?

On a purely technical level, there’s nothing stopping you from doing so (aside from MTU windows, so you’d need to find some way of implementing message continuation, ordering, checksumming, etc. — at that point, just use UDP multicast or something else).

But AnnNet is designed to be a lightweight announcement protocol for a local (i.e. within the same subnet/link system) network.

It is designed to act like e.g.:

• An Ethernet Address Resolution Protocol …​ (ARP)

• Neighbor Discovery for IP version 6 (IPv6) (ND)

• Lease requests for:

  • Dynamic Host Configuration Protocol (DHCP)

  • Dynamic Host Configuration Protocol for IPv6 (DHCPv6)

etc., but with (very small amounts of) arbitrary data making it "plug-and-play" with >= L3 protocols/services' configuration.

It explicitly avoids defining any formation of payloads, message format, or the like — because it is designed to be entirely agnostic of whatever use case it is implemented for.

However, AnnNet strongly encourages the use of WireProto for payload packing due to its minimal overhead, its strong hierarchical structuring, and its flexibility.

Warning

Future versions of AnnNet may require use of WireProto (but WireProto itself is a very loose and flexible format, and would not add much overhead). If this requirement is put in place, it will be in its own major revision.

If one needs a much more robust solution for anything beyond e.g. announcement messages, NATS may be useful for actual messaging between known members. (AnnNet would then be used in this case to provide a mechanism for making members known/discovered with no explicit configuration.)

Note	NATS also does not define any sort of payload format in its encapsulated payloads; most implementations tend to use ProtoBuf/Protocol Buffers, MessagePack, JSON, or the like.

8. Addendum/Errata/Administrivia

8.1. Bitpacked Fields

Certain fields in IP headers are less than 1 byte or don’t align to a clean byte boundary on their own, and therefore generally require bitwise operations to read/encapsulate meaningful values within them.

8.1.1. IP Version, IHL (IPv4)

The IP version and IHL combined make up a single byte, with four-byte "nibbles" each.

Note

The IHL is calculated as the number of "32-bit words" in the IP header. In other words, IHL = (B * 8) / 32 (where B is the size of the IP header — everything from "VI:V" to "IP:O" inclusive in Message Format — in bytes). Or, alternatively, IHL = b / 32 (where b is the size of the IP header — everything from "VI:V" to "IP:O" inclusive in Message Format — in bits). Since the V/IHL is a fixed size, this does not cause a recursion/chicken-egg problem.

To create the Version/IHL value, the Version (always 4 because IPv4) is packed as a big-endian 8-bit unsigned integer, and is then bit-shifted to the left by 4 bits and bitwise-OR'd with the IHL (as a big-endian 8-bit unsigned integer bitwise-AND-masked to the lower bits via 0x0f/15).

To retrieve the Version and IHL, the value is bit-shifted to the right by 4 bits and cast as an 8-bit unsigned integer (this is the Version, 4), and the original value is also bitwise-AND'd with the lower bits via 0x0f (15) and cast as an unsigned 8-bit integer to get the IHL.

Example in Go

Unresolved directive in <stdin> - include::examples/vihl.go[]

8.1.2. DSCP, ECN (IPv4)

The DSCP and ECN fields are combined to make up a single byte with a 6-bit and 2-bit "nibble" each, respectively.

Note	The actual values for DSCP and ECN are sort of scattered about through several different RFCs. Notable DSCP mentions are: RFC 2474 § 4.2.2.1 RFC 2475 RFC 8837 § 5 Notable ECN mentions are: RFC 3168 RFC 6040 RFC 8311

Example in Go

Unresolved directive in <stdin> - include::examples/de.go[]

8.1.3. Flags, Fragmentation Offset (IPv4)

The (Fragmentation) Flags and Fragmentation Offset fields are combined to make up 2 bytes with a 3-bit and 13-bit "nibble"/"word" each, respectively.

The Fragmentation Flags and Offset are defined in RFC 791 § 3.1 and part of RFC 815.

Example in Go

Unresolved directive in <stdin> - include::examples/frag.go[]

8.1.4. IP Version, Traffic Class, Flow Label (IPv6)

IPv6 thankfully only has one bitpacking in the header. Unfortunately, it’s a triple-whammy.

These are mostly defined in:

• RFC 8200 § 3 for the header format

• RFC 8200 § 7 for the Traffic Class field

• RFC 8200 § 6 for the Flow Label field

The IP Version takes up 4 bits (just as in IPv4, except it will always be 6 this time), the Traffic Class field takes up 8 bits, and Flow Labels takes up 20 bits for a total of 32 bits (4 bytes).

Example in Go

Unresolved directive in <stdin> - include::examples/vtf.go[]