Flakes Construction Yard

[Flake]

Great success today:



That's me inspecting from the 8th byte on a chat message packet.

If I looked at the raw data, I might not have known that was a string, but the inspector clearly shows what the data type is.

Next step will be to implement a "proxy mode" packet handler, which will just send client packets to the vanilla YSF server, and vice versa. Then I can inspect inbetween.

EDIT: Proxy mode implemented and I am now inspecting data from the vanilla server.

EDIT2: The inspector has already proven it's worth. For entity joined packet:
0-2 UInt16 ENTITYTYPE [00 Aircraft, 01 Ground]
4-8 UInt16 VERSION [Should always be 01]
108-112 UInt32 OWNERTYPE [03 Player, 02 Other]
112-116 Single HTRADIUS
116-118 UInt16 ISHELICOPTER [00 NotHelicopter, 01 IsHelicopter]
118-120 UInt16 CATEGORY INDEX [00 Normal, 01 Utility, and so on, in order on the aircraft select screen]
120-124 ??? UNUSED [UInt32???]
124-140 String USERNAME [16 chars
140-172 ??? UNUSED. [32 chars]

Looking back over my preivous assumptive code, I would have been saying: All entity joins are owned by the player, all had a radius of 10m, no aircraft were ever helicopters. All aircraft were aerobatic category.

I'm guessing entity join requests were failing because their data was 140 instead of 172. I'll try that and see what happens. I went back over the start of the packet and I would set the join version incorrectly each time an aircraft would join the server - the version was set to 0, which would surely screw everything up. I'll define the new packet structure and try again from here.

[Flake]

First multiplayer online flight.

It's just a proof of concept for now, I have each client assigned a vehicle ID.

I'd much rather assign each client a Vehicle instead, that way I can look at the clients vehicle to see where they are in the world.

So from here I'll need to add a Vehicle class much like OYS 1 had.

When that's done, I'll need to handle leaving flight gracefully, and then I can work on handling things like damage and ordinance spawning, then finally with the vehicle class and EntityJoined packets fully mapped out now, I can add vehicles to new clients that connect. I'll also add in the "query airstate" - which pretty much tells a client "these aircraft ID's are still flying, so pay attention to these flight data packets".

We're not far off a functional vanilla server.

[Flake]

Vehicles class is implemented. I actually had to do just a little bit of a work around, because most things were aircraft centric and they should not have been.



In this screenshot:

The first client joined the server.
The first client started flying.
The second client joined the server.
The second client can see the other aircraft.

I hope this will always work but this should solve a long standing bug of OYS 1 where sometimes you'd join and not see the other aircraft on the server.

EDIT:

Ordinance spawns are now being passed to all other clients.
Damages are now being passed to other clients.
'Fixed a bug where the color gold was actually green. (lol)
Stacktraces are now easier to read, should make bug reporting easy as now you can just copy paste from the program or take a screenshot, and I'll be able to solve promptly.

I still need to clean up the world loading, and then implement settings loading. Once that's done I'll release first alpha.

After that, I'll implement weapons loadouts and weather, then clean up the project and release beta.

Then it'll be periodic updates as I add each new feature. It's almost ready!

[Flake]

A lot of behind-the-scenes work has gone in over the last few days.

I cleaned up the entire project tree now things are pretty well finalised, so it's a lot easier to look at and a lot cleaner too.

Here's the project tree structure and an overview of what it is to look at in the solution explorer:




So a few changes:

1) I'm signing the assemblies now using a strong name key pair. Anyone who wants to use this source will be required to do the same.
2) Developers of this source on github can fork, then make their key (.snk) and also make a copy of DeveloperInfo.cs and edit that - in that file, they will put their username and their builds will be signed with their name, not mine.
3) Icon changes from my OfficerFlake logo to the OpenYS logo
4) Cleaned up Metadata and World loading a great deal - it's now loading from the directory specified in settings.
5) Codified the entire project structure. Much easier to see references and how the components interact together.

This is a massive change behind the scenes from the mess that was the code before I did the clean up - It doesn't look like much but now the code is lot easier to look at and work with.

EDIT: Now able to load from Settings.Dat - I stress again, able to load, not actually loading from.

I'm testing the loading of specific setting data-types right now. Eg: DateTime, Distance etc. What I've tested so far is working. I'll keep working to hopefully ensure the ongoing stability of this.

[decaff_42]

I can't imagine the amount of work you have done re-writing this and optimizing it. No doubt your work will help online flying and the ysflight community as a whole.

Keep up the amazing work!

[Flake]

More work.

PacketInspector is now hidden on launch, and can be revealed by clicking the magnifying glass icon.
Which means I am starting to implement the UI icons. I intend to link up the UI to the Settings more robustly.
Speaking of settings, I am now able to both load and save them. Before I was only loading.

I'm working on cleaning up the UI backend a little bit, so I can work on creating an OpenYS Client. Some people have been asking for a distance lag solution for some time and the program base is at a point where I can start doing that.

I'll need to have Proxy Mode track vehicles, then I will need to add vehicles and despawn vehicles when the proxy server receives this information. After that, I'll need to have the proxy send an OpenYS Server Handshake, which tells the vanilla/OYS server that the proxy connects to that it supports the custom user packets. I'll then add a formation position packet. If it all goes to plan, that should be all that is needed:

1) Vehicles for ProxyMode
2) OpenYS Handshake
3) Formation Data Packet Handling

It's actually closer than I thought! I just need to think over how I will do the formation position packet...

[Flake]

Nothing cool to show you folks but I've made great headway behind the scenes in the Connections class:

I'm now able to handle multiple data-streams per connection. In the past, I could only handle the one socket. With a proxy-server, I need to handle two: One socket for the client side of the connection (Client<->OYS) and one for the server side. (OYS<->YSF)

The data is not being processed much if at all right now, I'm just sending the data onward as a proof of concept.

Very soon, I'll be doing the following:

1. Get the username from the connecting client in proxy mode
2. Assign Vehicle ID's when aircraft are added or removed by the Server in proxy mode
3. Implement OpenYS Custom Packets Extension
4. Implement OpenYS Custom Packet Handshake
5. Implement OpenYS Formation Position Packet
6. Implement UDP Handshake
7. Implement UDP Data Sending (Faster than TCP, less bandwidth used)

If that makes sense to anyone, they should see where the project aim is for the next month or two: OpenYS Formation Client/Distance Lag fix.

After that, I would like to implement weather functions, moderation/admin controls and other desireable controls for server owners.

We're starting to transition to the point where OYS2 needs thorough stability testing and bug reporting. In particular, I want to know how the lag is on the server with one or more clients flying. I want to know the memory pattern of OYS2 and how it holds up over time; Ditto for CPU usage.

EDIT: Having issues with the formationflightdata packet - the data isn't sending correctly or converting back to the required format. I'll have to have a closer look at what I'm doing here.

EDIT2: Getting closer - the data is being read appropriately now. The problem is my logic for determining when to send formation packets, and how. It's a real head scratcher. To further complicate issues, now I've got data being sent to the wrong client/s and I need to work out what I'm doing wrong here.

EDIT3: Got it working. Mostly.



Now in theory, these two aircraft would be right on top of each other. But alas, due to the natural lag of time-delay in posting packets to the server (about 1 every 100-200ms) they are slightly offset. I'll see if I can do some math on the velocity of the aircraft and the difference in timestamps to reposition the formed aircraft more closely to where the aircraft actually is.

That might seem confusing, let me pose an analogy: Aircraft one is at position 0,0,0. It is moving at 1ms to the north. So in the next second, it should be at position 0,1,0. But we only get a flight data packet every 0.2 seconds. So by the time I get the flight data packet, at time +0.2, the aircraft is no longer at 0,0,0, but instead at 0,0.2,0. I should be placing the aicraft at 0,0.2,0 to factor the timing of delivery, however, I am not doing this right now - I am placing it at 0,0,0 at time +0.2, and by the time that makes it to the server, it's probably +0.4...

Oh yeah to put into perspective, that's the distance lag at about 500KTAS.

Now the other alternative, is to keep a register of formation flying vehicles, and everytime the server updates the position of an aircraft, it should also send an update for the formation object. I've tried that in the past and it didn't work but I also have not had this much success with OYS before in this nature. It's a very exciting time.

[Flake]

I tend to edit my posts a lot but that means people miss my updates as the thread doesn't bump. I'll stop doing that - sorry if it annoys you but I think it's the lesser of two evils.

My internship is over and I'm back to work on this project. I loved that internship, it was some of the best work I've ever done and I hope to hear back about a fulltime position soon.

I learned something today regarding how YSFlight handles timestamps: I always thought that the clients would syncronise with the server time if the server says to the client: "Start counting from this timestamp...". The client just flat out ignores that and uses it's own internal time. The result is that the client can send a packet with timestamp "5 seconds" and another could send a timestamp of, say, "300 seconds" at almost the same time.

It's actually pretty clever I think. All I need to do to convert the timestamps to the servers time is track the datetime that the client first joins the server, and then add their reported timestamp to that timestamp.

Here's an Analogy:

Client one joins the server when the server has been up for 10 seconds. JoinTime = 10 seconds.
Client two joins the server when the server has been up for 20 seconds. JoinTime = 20 seconds.

Client one sends a flight data packet when it has been on the server for 30 seconds. The packet timestamp is "30".
Client two sends a flight data packet when it has been on the server for 30 seconds. The packet timestamp is "30".

The ACTUAL timestamp of each packet, according to the timeline on the server is:

Client one: 10 seconds (join time) + 30 seconds (packet time) = 40 seconds since server start.
Client two: 20 seconds (join time) + 30 seconds (packet time) = 50 seconds since server start.

----

If we were to receive both packets at the same "real world" time or 50 seconds since server start, then client one packet timestamp would be "40" and client two timestamp would be "30". The calculations on just the server time difference would be out by 10 and 20 seconds respectively.

WHERE THIS COMES FULL CIRCLE is my calculation of the time-drift: If I'm out by like 10 seconds, and I try and calculate how far the aircraft is on the client screen vs the server screen, I'll be out by not just hundreds of meters, but potentially kilometers. The distance lag would just be straight up incorrect.

So that's what happened today: I was comparing the timestamps sent by two clients, and when one was saying "30" the other was saying "40" at the same realworld time. So the difference was 10 seconds between server and client timestamps, and that meant that the formation packet with timedrift incoroporated into the calculation was off by an astronomical distance.

Bit of a ramble but it's worth documenting this phenomenon, and what it causes and how to work around it.

So all we need to do is track client join time and adjust the reported packet time upwards by that factor. Then everyone's timestamps should be in sync. I'll go and work on that now and if it's all good, expect to hear back from me quickly.

[Flake]

Okay so I've got the delta time now working out correctly. The difference is roughly 0.004 on my rig.

If we extrapolate that difference over a high speed, say, 500 knots, that is:

500 KTAS * 0.51444444... ~= 257.222... M/S

252.222... M/S * 0.004S ~= 1.02M

Which is close to what I've come to expect local-server lagwise on the server. However, I have a new problem:

I thought the velocity component of speed would be absolute to the world, not relative to the aircraft.

That is, if heading north, vx would be +10, and if heading south, vx would be -10. Instead, vx is always +10, and only changes if the speed the aircraft is moving at changes.

This is a problem for me because that means I have to work out the inverse rotation projection of the velocity speed to get back to real world values. I need to calculate how far to move the aircraft in the real world co-ordinates for the time-drift function to work. It's something I've done before, but that was some time ago.

I'll have to work this out again.

EDIT: Nevermind, that's not the case. It is absolute. Derp.

[Flake]

Alright so I have a new problem now:

If you hold the YSFlight window and let it sit, it stops doing anything and loses sync. So if the timestamp was at 15 seconds, then you hold the YSFlight window for 10 seconds and stall the game, then when you release the window it will start counting at 15 seconds again. It does not catch itself up.

The problem with that is huge: If the client doesn't keep absolute time and can drift on it's own, then the delta-time between server and client can be well into the tens or hundreds of seconds even. This breaks my function entirely and will send aircraft careening into the ground.

Great game Soji.

So what I need to do now is completely ignore all the timestamps sent from YSFlight clients - they are nothing more then a sequential counter. They do NOT represent time that has actually gone by even though that was Soji's intention.

I'll come up with a unique way to measure the ping back to the client, and then adjust the lag from there. I want to do this with the vanilla packets. Maybe send an OP code that the client is forced to reply to? I'll work something out.

Edit: Turns out sending the "Prepare Simulation" packet over and over again doesn't break anything. The client will always respond with the appropriate acknowledge code. I can use this on a forever running loop to check the ping. On my local rig, with processing included, the ping is 11ms between all my proxy layers. It'll probably be faster still with just the OYS server.

Time to link a loop method up to update the ping every 15 seconds or so.

[Flake]

I'm a L E G E N D!

I was getting too much jerking around and lag with the Ping based solution - it was a great concept and worked okay, but there's a better way.

I can just see how long it's been since the last vehicle update for the client and extrapolate off the previous information. Before a minor change in ping would cause the aircraft to jerk about a bit. Also, YSFlight can be... inconsistent, so if the packets weren't coming in perfectly on queue, aircraft would jolt back and forth.

The solution now is to always update the position of all vehicles as the packet specifies when it comes into the server. Everytime the server gets a formation packet, it estimates the current vehicle position based on how long it has been since the last update for the formation host target.

In short - I place the new formation vehicle almost exactly where the host actually is.

And there is far less overhead - No need to update the ping every few seconds either.

I'll work on cleaning up the debug information and program just a little bit more. In particular, I need to update the vehicles database on the proxy server to also use this time extrapolation. This way, what you see on your client is what is reported to the OYS Server for relative position.

Currently, there is only ONE formation position, and that is 0,0,0 for my test of concepts. But it will be a quick data swapout to get the actual relative formation position.

I'm so glad this is working. It's better than I thought it was going to be.

[Flake]

Quick Update:

Managed to swap over the formation packets to take from the real position as reported by the YSFlight Client and not the fake 0,0,0 i was previously using.

The formation seems to gracefully adapt to when a client joins or leaves the formation. I've set the formation-packet range to be 5* htradius on the local client. So if you can't get within 5 * of your own ht radius on your client, then you will not be in the formation. This is important to set up this way as it allows for formation breaks and the like to happen - in those cases you don't want to be sending formation packets.

The way formation flying will work is it will check your aircraft tag for "#x" where x is your position number. Lower numbers can form off the higher numbers, counting down from #1. So if all the aircraft were in a tight formation with the flight lead, all their updates should be relative to the flight lead. However there are of course cases where say #3 and #4 split off, and then #3 should the "lead" for #4. #1 will never form off anyone. #2 will only form of #1. #3 could be on #1 if within range, otherwise will check for #2, and so on. The current limit is 9 aircraft but it would not be hard to extend this. I will not be doing that for now though as who really needs a formation of over 9 aircraft in YSFlight. Short of starting the virtual red arrows I don't see there ever being a need. Even then, there is only 9 aircraft in that formation anyway...

I'm trying to work out if there is a minor time-lag effect when updating data from the ysflight client and the proxy server - I suspect there was as I was updating based on the actual position only, and not the estimated position, in the proxy client. I just made that fix and will test more thoroughly. Without this little correction, the client would be out by just a few metres. It's really hard to test as I have to get the clients to form right next to each other manually, and trying to do that consistenly at over 300 knots with 3 different clients open is a real handful.

[Flake]

I was able to get the formation distance declared by OYS to be 0.5M, when I was practically right on top of the other aircraft. Give or take a little distance, this seems to be working perfectly now.

I also ran a CPU sampling test and the most samples were taken up by my debug output telling me where I was in the formation (95%) so that's great news. There is very little actual CPU overhead in the program (1%?) taking up processing resources for flights and network data.

I think I'm ready to start on the User Interface for this proxy-client. At the moment it's not very intuitive and if I'm getting people to test my shit, I want it to be legit you know?

[Flake]

User interface has been made.

I now need to code the logic behind it, and link it all up to relevent interfaces and the like.

[Flake]

So I just ran a test of the OYS server over the internet, and what works fine on my PC is not working for anyone else.

Consequently, it's very slow to start a connection on the server, then there's data send issues.

I need to work out what's wrong here but to do that, I need to have a more details log of what's happening connection-wise.

I'll have to clean up the connection-class a bit more to make it easier to maintain and look at (stuff everything into regions, refactor names and organize methods). After that, I'll add a lot more debug information.

The next release will probably have no functional improvements but will be looking to solve these problems by identifying what is causing this behavior.

[Flake]

Alright I've been thinking and there's a few things I want to do:

1) Encapsulate the Lists used by the program. There are several lists: Lists of Vehicles on the server, lists of ground objects, lists of connections and users etc. The problem with using lists directly is that you can't enumerate a list and modify it at the same time - the enumeration says "whoa hold on you changed the list!". You can enumrate using a for loop instead of a for-each loop, but that will also cause problems. This is the source of the "MoveNext" error that props up randomly. It's a sure fire way I was going this incorrectly - and that's absolutely the case. What I need to do instead is encapsulate the list (make it so I can't access it directly) and then create a class with methods to add an item to the hidden list, remove and item from that list, or provide a copy of the list. The add/remove or create copy methods will all be thread safe and lock the hidden list. If I'm adding to the list, I can do that across multiple threads safely. Allow me to illustrate:

I have a list of A, B and C in it. I want to add "D" from one thread. I also want to remove "C" on another thread. Presuming both these threads get the contents of the list originally at the same time, the final output could be "A,B" or "A,B,C,D". - which are both wrong. I need instead "A,B,D".

By using this thread safe encapsulation, I should be able to do: "A,B,C" => start a call to add "D" => "A,B,C,D" => NOW start the call to remove "C" => "A,B,D".

It's a tiny slow down in the program (barely) but everything will be in sync at least.

2) There are issues still in Data Send/Receive and I think that's to do with similar problems as the above - I can call the packet receiving method multiple times from multiple task launchers. This is not thread safe at all. If more than one task starts receiving data at one time, then the entire process will break rapidly over the internet. I cannot have that happening. I need to be sure that there is only one packet "receiver" task running in an infinite loop, and not recursive calling receiver. Ditto for the packet sending. Sending is definitely not thread safe right now, but maybe receiving is. I'll have to check this out carefully.

3) Random disconnects: I've been getting these. From what I can see in the console of YSF - there are no bad opcodes being sent from the server, which makes me say it's my program logic. I think the problem is that the server is timing out while waiting for a reply to "PrepareSimulation", and silently killing the connection. I remember my program was doing this and meant to fix it but forgot to implement. I will do a solutionwide search for "Disconnect" calls and make sure every single one has debug data attached to it.

4) I would really like to have some program logic for all my extension methods as a core plugin rather than a derivitive component. At the moment I have to add units of measurement together by calling "(<Unit>.ToMeters().RawValue + <Unit>.ToMeters().RawValue).Meters();" which is really clunky and hard to read. I would instead love that to be "<Unit> + <Unit>;" which is entirely possible if I move the extension methods outside of the component projects. That means I would have a 1.xx layer project for extension methods sitting above the interfaces. Infact, I already have this project so what I really need to do is just refactor this and make sure this is all working as intended. I should move the static lists outside of the "Extentions" model and into its own project sitting above 'Extensions' actually. So I'd be looking at a "1.01 Interfaces", "1.02 Extensions", "1.03 Lists" or something like that. I hate the name "Lists" though - not descriptive enough. There's got to be a better wording choice that's more articulate...

----

1) Move Extensions Out
2) Seperate and encapsulate Data Lists
3) Add Debug Messages for all calls to Disconnect();
4) Ensure one task for Send/Receive per connection stream

In this order.

If I can do all that, then it'll be time to test JUST THE SERVER ONLY over the internet once again, hopefully it'll be better. Actually, I want to have a look at my old netcode for the OYS 1.0 revision that Patrick says is quite stable for online server use. I think I made a bug fix version for that previously so that should be easy to pull up and inspect.

----

Lots of blah blah here, but it's my brain sneeze to work this shit out. Bare with me.

[Flake]

Just tried to download the source material on my laptop and failed to build due to missing ClickOnce manifests in 0.00_ServerMode.exe - this is an error in the project file definition .csproj that refers to ClickOnce manifest signing entries that are no longer used. You think VS would be smart enough to ignore these lines outright since the build option is turned off in the project settings but hey, when life gives you lemons...

That'll be fixed for future builds. But the instructions for porting the project clone over are 100% correct! By following the guide I was able to set my developer info correctly and build and sign with a new strong name key pair. Great to know that's easy to follow.

[Flake]

Back to work.

I've completed re-writing the connection class to be much cleaner to look at and work with by breaking into a partial class instead. Now I can dig right into where I need to be - code is much easier to read and maintain and overall just contemplate how all the pieces fit together.

Now I appear to have the data coming into the program successfully, however there is a problem with the program not recognising that data has been received already, so I'm working on that more thoroughly. I have a "packet-waiter" structure in place so far. The program logic is 1) I'm about to send xxxx to the client, but I expect to get yyyy back. So i'm going to start listening for yyyy 2) when a packet comes in that matches yyyy, that packet-waiter is signalled 3) the task that was waiting on yyyy can continue happily. Now the current problem appears to be that yyyy is not being received, or yyyy is received but it's different (yyyz?) slightly, or yyyy is being received before the program says "I'm expecting yyyy..." So that's a whole new can of worms to work through. I'm finding inefficiencies and potential problems in the code though so a work around here will hopefully sort things out.

That's what I'm working on right now.

EDIT: I'm definitely getting the exact same issues on my local rig right now as I'm getting on pats server - which is great news as I can then pinpoint and solve myself!

EDIT2: Also added debug output to files.

EDIT3: Okay the problem seems to be with the way tasks are spinning up, I'll try and work out what's going on here.

[C05MIC]

Great Job Flake looks really well developed ironing our the bugs and what not will be a very stable addition thanks for your contribution!