Results 1 to 7 of 7

Thread: CL_NetGame data types

  1. #1
    Lesser Wizard
    Join Date
    Jun 2007
    Posts
    109

    Default CL_NetGame data types

    Are there any plans to add different sized integers to the netgame system? My compiler (as most probably) defaults integers to 32-bit. Which is alright sometimes, but when data size makes a difference it effectively doubles (or more) packet sizes.

    For example if you want to send health % of your character (0-100 value), it takes 4 bytes instead of 1.

    Now, I've made changes to datatypes in the past for Netgame, but I'm unsure how the other distributions (linux, MacOS) deal with integer sizes. I don't develop for those platforms or compilers, so I'm not sure if offering up a patch with MSVC++ datatypes would be very useful.

    I don't mind doing this myself, but I'd think others would want to take advantage of the changes. Um. Any idears? Or should I just not worry about it?

  2. #2
    ClanLib Developer
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    554

    Default

    Actually the current protocol is quite different from what you imagine.

    The current protocol essentially looks like this:

    WORD packet_size
    BYTE[packet_size] packet

    Where each packet then is an UTF-8 encoded XML document.

    The format of the XML document is something like this:

    <gameEventName>
    <integer>42</integer>
    <string>foobar</string>
    </gameEventName>

    Where gameEventName is the name of the CL_NetGameEvent, and the child nodes in the XML are the parameters.

    So it is not 4 bytes for integers but 19 bytes plus the value itself.

    The protocol is in the CL_NetGameNetworkData class in network_data.cpp in ClanLib and pretty simple to improve. If your game sends a lot of data its probably a good idea too

    If you wonder why is it using such an inefficient protocol? Well its quite simple: it was fast and easy to code and DiceWar didn't send much anyway. Making a good compression algorithm also requires some work so that's another reason.

  3. #3
    Lesser Wizard
    Join Date
    Jun 2007
    Posts
    109

    Default

    Quote Originally Posted by Magnus Norddahl View Post
    Actually the current protocol is quite different from what you imagine.

    The current protocol essentially looks like this:

    WORD packet_size
    BYTE[packet_size] packet

    Where each packet then is an UTF-8 encoded XML document.

    The format of the XML document is something like this:

    <gameEventName>
    <integer>42</integer>
    <string>foobar</string>
    </gameEventName>

    Where gameEventName is the name of the CL_NetGameEvent, and the child nodes in the XML are the parameters.

    So it is not 4 bytes for integers but 19 bytes plus the value itself.

    The protocol is in the CL_NetGameNetworkData class in network_data.cpp in ClanLib and pretty simple to improve. If your game sends a lot of data its probably a good idea too

    If you wonder why is it using such an inefficient protocol? Well its quite simple: it was fast and easy to code and DiceWar didn't send much anyway. Making a good compression algorithm also requires some work so that's another reason.
    After I posted my question I began to wonder if that was the case... Hm.

    So it's essentially sending raw string data. That itself isn't all that awful (and I think RakNet does that with its own bitstreams), but compression or atomization would certainly be a requisite for anyone considering using that in a real world situation. Which I am!

    Just throwing ideas out here, but would it be possible to use zlib to do a quick compress of the data?

  4. #4
    ClanLib Developer
    Join Date
    May 2007
    Posts
    1,824

    Default

    Using zlib to compress say 32 bytes, would not gain much (and may actually increase the size).

    It could perform compression if the total xml to send is greater than say 256 bytes. But maybe the data itself should be compressed by the application, not clanNetwork.

    My personal network library (not clanlib). Sends data from an XML source, converting it to my own "binary xml" format (not the official binary xml format).
    I found that by doing it that way, did not save that much space, however the XML import/export functions were very fast.

    Another advantage of this method is that the data to send does not have to be encoded (for example, a xml content may contain a large PNG, without converting it to base64 first)

    (I am not releasing the source code for this, it's closed source)

  5. #5
    Lesser Wizard
    Join Date
    Jun 2007
    Posts
    109

    Default

    Quote Originally Posted by rombust View Post
    Using zlib to compress say 32 bytes, would not gain much (and may actually increase the size).
    Yeah I checked on this after posting, it does indeed increase the size for very small chunks of data.

    It could perform compression if the total xml to send is greater than say 256 bytes. But maybe the data itself should be compressed by the application, not clanNetwork.
    I don't see how you'd compress it via the application and not the API? Should I re-code and re-compile my own version of clan network? Possible, but not really viable. Encoding my data prior to send_event is pointless as well. Maybe I misunderstand what you meant though...

    The XML from the protocol itself is the killer.

    I found that by doing it that way, did not save that much space, however the XML import/export functions were very fast.
    Naturally. It's fairly wasteful to send any form of XML over the wire unless serialization is really something you need, in which case, it's not a bad way to go.

    I really like NetGame. It's crazy easy to use, and it works perfectly well in certain environments. I just wish it worked in mine!

  6. #6
    ClanLib Developer
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    554

    Default

    I personally don't think there's any need to change the actual API of NetGame - just the protocol. The XML protocol it uses right now was only written with the purpose of creating something fast and for that it has been a success.

    Zlib compression doesn't work very well on game packets because our packet size is very small and zlib needs a lot of data before its sliding window and huffman encoding starts to work.

    The easiest and probably most naive compression attempt would be to simply write the message with the fewest possible bits. For example, if we want to send CL_NetGameEvent("AAAAAA", -4200, "BBB", 0.0), we could use the following encoding:

    • types are sent as a byte where the low 4 bits indicate the type and the top 4 bits are used by the value
    • text strings are sent as null terminated UTF-8 strings.
    • numbers use the top 4 bits of the value type to determine sign and number of bytes used
    • floats are sent as 32 bit floats


    Such an approach would reduce the size of the above event to this in hex:

    00 41 41 41 41 41 41 00 <- type 0 = text string, value = "AAAAAA"
    01 03 <- type 1 = number, value = 3 (the parameter count)
    91 10 68 <- type 1 = number, highest 4 bits = signed+2 bytes, value = -4200
    00 42 42 42 00 <- type 0 = text string, value = "BBB"
    02 00 00 00 00 <- type 2 = float, value = 0.0

    Total length of the packet: 23 bytes.

    If the implementer just loves to juggle bits, the type byte could be reduced to 4 bits where the upper bits only are included if the type uses them. Personally I wouldn't bother though. Compressing floats is probably also not a bad idea, but I didn't bother study them.

    Even better compression would probably require adding some knowledge about past packets. The protocol could include a way for strings to be registered as atoms or it could use past packets as its sliding window. I personally really wouldn't bother with that unless I was 100% sure I had a packet size problem.

    The NetGame API itself lacks a 'flush()' function that a game should call at the end of a game tick. Right now it dispatches each message sent in a single TCP packet (thanks to the TCP no-delay flag), but if it attempted to put as many packets as it could into the same IP packet, the compression would also be greatly improved (less low level packet overhead).

  7. #7
    Lesser Wizard
    Join Date
    Jun 2007
    Posts
    109

    Default

    Quote Originally Posted by Magnus Norddahl View Post
    If the implementer just loves to juggle bits, the type byte could be reduced to 4 bits where the upper bits only are included if the type uses them. Personally I wouldn't bother though. Compressing floats is probably also not a bad idea, but I didn't bother study them.

    Even better compression would probably require adding some knowledge about past packets. The protocol could include a way for strings to be registered as atoms or it could use past packets as its sliding window. I personally really wouldn't bother with that unless I was 100% sure I had a packet size problem.
    As interesting as all the bit juggling is, it's not my fortay. I'd have to learn the whole compression thing from the ground up.

    The NetGame API itself lacks a 'flush()' function that a game should call at the end of a game tick. Right now it dispatches each message sent in a single TCP packet (thanks to the TCP no-delay flag), but if it attempted to put as many packets as it could into the same IP packet, the compression would also be greatly improved (less low level packet overhead).
    One way I was combating this was simply adding arguments for a reasonable amount of objects nearby... I'd pack as many as I could per update, but an automated feature in the API to do that would not be frowned upon

Similar Threads

  1. Inqueries for CL_NetGame
    By catch22 in forum Official ClanLib SDK Forums
    Replies: 11
    Last Post: 11-26-2010, 08:40 AM
  2. I keep losing saved data!
    By NinjaNumberNine in forum Novashell Game Creation System
    Replies: 1
    Last Post: 11-06-2009, 05:11 AM
  3. Loading files using custom resource types
    By Hypercube in forum Official ClanLib SDK Forums
    Replies: 3
    Last Post: 12-12-2008, 04:53 AM
  4. about save data
    By DavinciZhe in forum Official ClanLib SDK Forums
    Replies: 4
    Last Post: 01-24-2008, 03:18 PM
  5. brain types list
    By rabidwolf9 in forum Dink Smallwood HD
    Replies: 2
    Last Post: 01-08-2005, 08:32 PM

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •