The Networking Refactor
Introduction
Reworked networking
Welcome to the reworked networking blog post for NeoForge, these changes are available in NeoForge 20.4.70-beta and above. The post will describe the changes to NeoForge to enable the configuration network protocol designed by Modmuss50, a maintainer of Fabric, with some minor modifications. I’m sharing more on those below.
SimpleChannel and EventChannel
Forge had two different ways of implementing a custom network channel. A simple registration-based approach is known as SimpleChannel, and a system that fires events for each packet received is known as EventChannel. To simplify the API and make interacting with new packet layouts easier, it was decided to rework both implementations into a single system, combining the best of both worlds.
Implementation
New network payload handling
The system is based on the CustomPacketPayload definition that Mojang uses to represent the content of custom packets. Internally, they register their custom payloads, used mainly for debugging, to a map. We need to extend this map so that a modder can send and receive a custom implementation of this CustomPacketPayload. This is what a significant part of the rework achieves. Modders can introduce new CustomPacketPayload implementations by registering them during the RegisterPayloadHandlerEvent.
The registrar
Any modder can request a registrar for any namespace they desire; however, it is recommended that a mod only requests a registrar for its namespace. Once a registrar is retrieved from the event, it can be configured with two different options: versioned(String version) to configure a version for all payloads registered after the call and optional() to mark all payloads registered after that call as not requiring a receiving side. An example of the registrar configuration can be found here:
@SubscribeEvent
public static void register(final RegisterPayloadHandlerEvent event) {
final IPayloadRegistrar registrar = event.registrar("my_mod")
.versioned("1.2.3")
.optional();
}
Note
The registrar is a semi-immutable object; calling versioned(String version) or optional() on an instance of the registrar will cause a new instance with the desired configuration to be created.
Warning
The registrar loses its validity the moment the scope of the event has been left. Registering payload handlers outside the event handling scope will result in those payloads not being known to the system and not being sent over the connection. Additionally, the attempt to register them outside the event scope will trigger an exception.
The registrar offers six different endpoints, three pairs of two, to register new payloads. A pair exists for the play phase, one pair for the special configuration sub-phase, and one pair of methods for both.
Info
It is impossible to register custom payloads that should be sent during the login phase of the connection. And the new code offers, as such, no infrastructure to achieve this.
Within each pair of registration methods for a phase, two variants (as such, six methods) are available. One that registers the same handler for both sides of the connection and one that takes a consumer, allowing for the configuration of single-sided or differentially handled payloads.
An example of the signature of the methods for the configuration phase is as follows:
<T extends CustomPacketPayload> IPayloadRegistrar configuration(ResourceLocation id, FriendlyByteBuf.Reader<T> reader, IConfigurationPayloadHandler<T> handler);
<T extends CustomPacketPayload> IPayloadRegistrar configuration(ResourceLocation id, FriendlyByteBuf.Reader<T> reader, Consumer<IDirectionAwarePayloadHandlerBuilder<T, IConfigurationPayloadHandler<T>>> handler);
For the play phase, similar methods exist. For payloads that are supposed to be sent during both the play phase and the configuration phase, there also exists a pair. However, here, the handler is a common supertype of the handling callbacks, which has a reduced superset of the information available to the two different types.
Payload discrimination
When considering this system, you might, rightfully so, ask how the system keeps the different payload types apart from each other. It writes a discriminator ID to the connection before invoking the write method for a payload. Similarly, on the client side, the discriminator is read first, then a reader is looked up so that the rest of the payload can be read.
The value of the discriminator during writing is retrieved from the CustomPacketPayload#id() method, and cannot be null.
The value against which the id, read from the connection, is compared to find a reader, is the one that is given to the registrar as the first argument.
It is as such of the upmost importance that both the registrar and the id() method of the payload instance receive the same resource location.
Tip
We recommend that you store your ID in a public static final ResourceLocation ID = new ResourceLocation("mod_id", "payload_id") field, and reference that in both places.
Warning
Given that the id is used as a discriminator, it is important that you use a unique value, especially for the path of each payload type. If you try to register the same id twice, the registrar will throw an exception. If you try to register an id with a namespace other than the one the registrar is for, the registrar will throw an exception. You are free to request registrars for other namespaces than your own.
Payload reading
Payload reading happens via the vanilla method, implementing the FriendlyByteBuf.Reader<T> functional interface. During the registration of the payload type, as can be seen above, you need to pass an implementation of this interface so that a new instance of the payload can be created by the system when a custom payload packet with it as payload arrives at the receiving end.
Tip
As we recommend that your payload implementations are records in Java, instead of classes, we also recommend that you create a custom constructor with the record, to read the records fields from the buffer. This constructor can then be passed as a method reference for the reader implementation. So if SimplePayload(String something) {} is your normal record, then adding SimplePayload(FriendlyByteBuf buf) { this(buf.readUtf()); } as a constructor to the SimplePayload record will allow you to pass it as a method reference SimplePayload::new to the registrar when it asks for an implementation of FriendlyByteBuf.Reader<SimplePayload>
Warning
There is no guarantee regarding which thread the reading or writing callback is invoked upon. It is as such important to note that the method can be called on many threads in parallel if the same packet is processed by many connections simultaneously.
Payload writing
The CustomPacketPayload interface contains a method: write(FriendlyByteBuf) method. This method is invoked when it is time to write your payload to the network connection. There is no guarantee regarding what thread invokes the writer.
Warning
As with payload reading there is no guarantee regarding which thread the writing callback is invoked upon. It is as such important to note that the method can be called on many threads in parallel if the same packet is sent on many connections simultaneously.
Warning
Payloads are only read and written if sent over a connection. This means that the host of a single-player world (even if exposed to LAN) has packets and, as such, payloads transferred in memory. This means that for those payloads, no write method is invoked, and no reader is called. Only the handler is invoked!
Payload handling
Once a payload has been written, transmitted and read, the payload handler is invoked. This handler is again looked up using the id of the payload, and then invoked with the context of the receiving end. Each handler takes two arguments the: payload instance, and the context.
Warning
Payloads are processed on the network thread, and can as such happen in parallel with other payloads of the same type being handled. If you need to ensure that the payload is processed on the main thread, serially, see the ISynchronizedWorkHandler available in the context under the workHandler() method.
The context
The context contains information, callbacks and entry points, to access the surrounding network system, the main thread, as well as handling processing other packets, or completing configuration tasks.
ReplyHandler
The reply handler can be used to quickly send a payload back to the sender. It is, for example, useful to send and answer to a query packet, or to send an acknowledgment that you received and processed the payload. You still need to register the return payload.
PacketHandler
In case you implement a packet splitting mechanism, whether that splits on full vanilla packets, or custom packet payloads, the IPacketHandler interface gives you access to the start of the processing pipeline, allowing you to process other payloads immediately.
Note
This does not transmit payloads, it purely allows for the receiving end to process additional packets or payloads constructed in memory during the processing of your payload.
The packet handler also gives you access to a disconnect(Component) method, allowing you to terminate the connection, and showing the given component as reason to the user.
WorkHandler
The work handler allows you to schedule work on the main thread of the receiving side. This might be the Minecraft class instance if the logical receiving side is the client, or the MinecraftServer instance if the receiving side is the server.
The system uses CompletableFuture instances, so you are able to schedule different follow-up tasks.
PacketFlow
The context will indicate, via the packet flow, what the receiving side currently is. If it is a serverbound flow, then the handler is currently being invoked in the context of the server. Is it a clientbound flow, then the handler is currently being invoked in the context of the client.
ConnectionProtocol
The current active connection protocol is useful if you have raw bytes of a packet wrapped in your payload, allowing your handler to decode the inner packet or payload, before passing it to the IPacketHandler for processing.
ChannelHandlerContext
The Netty channel handling context that is currently processing the payload is also given as context.
This context can be used to retrieve the raw underlying connection via ConnectionUtils, or, be used to process raw bytes of inner packets and payloads.
Player
An Optional containing a player is also provided.
If the handler is invoked on the server side, then this is the player that sent the payload.
If the handler is invoked on the client side, then this is the local player, if it is available.
Level
An Optional containing the level the player is in is provided too.
TaskCompletedHandler
This is a special contextual value only available to payloads during the configuration phase, that indicates that a specific configuration task has been completed, and that the next one can be started.
Future additions to the contexts
We are fully aware that these entry points might not be all information you, as a modder, need to process a packet. In general, it is pretty easy to extend the interfaces and records. They were specifically designed to allow for simple PRs in the future to add to them, so please do not hesitate to create a quick PR to add your needed data to the context.
Packet Sending
In tandem with the refactoring of the channel registration mechanic we added new tools and systems to allow you to more easily send a custom payload to different targets.
PacketDistributor
This wrapper class now has the ability to process custom payloads solely. Its instances and targets can be passed around since they are immutable. Several methods on extension classes will accept these, to facilitate easy transfer of payloads.
Extension objects
We extended several vanilla types allowing them to accept payloads as well, not just packets. Examples are chunk sections, listeners, entities and players.
Netty information
We store a lot of information related to the connection, for example the negotiated payload types, on the connection object itself. As such, we added several attributes to store this information in. These attributes are considered internal API, use them at your own risk.
Configuration tasks on client join
Tasks
Vanilla now provides a centralized way to perform tasks and jobs that need to be performed when a player joins. The player won’t be instantiated or added to a world until these tasks are completed.
Under normal circumstances these tasks are implementations of the ConfigurationTask interface, with a single method: start(Consumer<Packet<?>>). However, this is subpar in our situation. Modders should never really have to touch raw Packets to perform their duties, only payloads. And as such it was decided to have modders implement the ICustomConfigurationTask interface from NeoForge. This provides a wrapper around the ConfigurationTask signature and allows sending payloads instead of packets, by implementing run(Consumer<CustomPacketPayload>) instead. The given consumer will then automatically convert the payload to a packet and send it to the client that is being configured.
Note
In practice, an instance of ICustomConfigurationTask is also an instance of ConfigurationTask as one extends the other. But to provide the ability for it to be a functional interface, the start method is implemented by default. You should not override it.
OnGameConfigurationEvent
This event is fired to collect all tasks that should be run, and allows for the registration of ICustomConfigurationTask instances to the listener. It is not possible to register the vanilla ConfigurationTask instances.
Tip
This event is fired on the mod bus, to preserve dependency order. Given that configuration tasks can only be running in order of registration, you can safely assume that configuration tasks of your dependencies have been running before yours.
NeoForge packet changes:
Moved configuration sync, registry sync, and tier registry sync to configuration phase tasks.
Bundle packet processing
In 1.19.4 Mojang introduced the bundling system for packets, which is a core component that allows packets to be processed together. We anticipate that modders may want to use this system, so we adapted it to accept custom payloads. You will find a sendBundled(CustomPacketPayload... payloads) method on the ServerGamePacketListener.
Warning
Packet bundling is only supported during the play phase of the network protocol. It cannot be used during the configuration phase of the protocol.
Opening menus with custom data
In the past, NeoForge supported opening UIs from the server side with additional data, via NetworkHooks.openScreen(...). This system has been moved and is now part of the server ServerPlayer extension. You can call the method openMenu with the same parameters.
Spawning entities with custom data
The previous networking implementation allowed spawning custom entities via an overridable method in the Entity class: getAddEntityPacket. Modders that wanted to support custom additional data to be processed on the client side when the entity spawned could override this method and return a packet from the method using NetworkHooks.getEntitySpawningPacket(...).
This system has now been refactored (as Mojang spawns and processes the entity packets with a bundle). The core of this new framework is the method sendPairingData(ServerPlayer, Consumer<CustomPacketPayload>) on the Entity class. There are now two methods to configure this.
Using a custom payload
By overriding the method and invoking the consumer with a custom payload, you are guaranteeing that your payload will be processed immediately after the spawning packet. You are free to do whatever you want, however, we recommend you at least transfer the entity id, as vanilla does, to retrieve the entity instance when your packet arrives.
Using the IEntityWithComplexSpawn interface
Implementing this interface on your entity forces you to implement two methods: writeSpawnData and readSpawnData. These methods, are invoked when an entity spawn bundle is generated and when the entity has been spawned, respectively.
Removal of custom entity creation code in EntityType
It is no longer possible to use the entity spawn packet code mentioned above to create a different entity class on the client side. This was due to the refactoring of the payload mechanics, and the reliance on the vanilla spawn bundle.
NeoForge packet splitter
The NeoForge packet splitter can now be used with any packet (other than the splitting packet itself).
Mod list transfer
Currently, the new protocol does not support sending the mod list across the network, and only works on a channel and registry content basis. However, we do intend to work with the Fabric team on expanding the protocol in such a way that the server is made aware of what kind of mods the client has installed. We deem this necessary to allow server owners to block mods that would allow cheating, for example, an XRay mod, etc. This will be added in another PR after further reviewing our stance and implementation possibilities.
Further documentation
More documentation on using the networking features can be found in the Networking section of the documentation.