Light Style© by Fisana

Jump to content


Photo

Progress reports on funded work


  • Please log in to reply
239 replies to this topic

#1 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 23 October 2011 - 08:43 PM

A while ago we asked for donations via Pledgie to support a developer doing full-time work on the game for a month, and had a great response. I'm currently able to devote that time to the project (I've just finished a PhD, and haven't sorted out long-term employment yet and can wait a while) so I'm going to start doing that now. This isn't meant to detract from the open source development of the game - instead it's an added opportunity to tackle some of the larger problems that can benefit from more concentrated work than usual.

I'll use this forum topic to record my progress, so people can see what their donations are funding and hopefully get a vaguely interesting view into the development process. Detailed technical discussions should probably go in separate topics to keep things organised, but comments are generally welcome here :)

My current (but quite flexible) plan of tasks to focus on is:
  • Saved game support - particularly to allow players to reconnect to a multiplayer game after a crash or network failure, since large gameplay testing sessions are often annoyingly interrupted by dropped players. Saving single-player matches is also a frequently requested feature.
  • Better tools for measuring the game's performance. We want to optimise various aspects of performance - overall framerate, consistency of framerate (i.e. avoiding occasional pauses that make the game feel jerky), better utilisation of multi-core CPUs, etc - and I think it's worth investing in new ways of recording and analysing performance so we can improve it more effectively.
  • Better engine support for AI scripts - mainly to increase performance in various ways, and to fix some other areas that cause pain for AI scripters.
  • More efficient pathfinder - one of the current algorithms can perform particularly badly with large numbers of units in a small area (e.g. melee battles), and needs to be redesigned to provide much better performance in those cases.
This should be enough to get started with :)
  • 1
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#2 ac892006

ac892006

    Discens

  • Community Members
  • Pip
  • 30 posts

Posted 24 October 2011 - 04:02 PM

A while ago we asked for donations via Pledgie to support a developer doing full-time work on the game for a month, and had a great response. I'm currently able to devote that time to the project (I've just finished a PhD, and haven't sorted out long-term employment yet and can wait a while) so I'm going to start doing that now. This isn't meant to detract from the open source development of the game - instead it's an added opportunity to tackle some of the larger problems that can benefit from more concentrated work than usual.

I'll use this forum topic to record my progress, so people can see what their donations are funding and hopefully get a vaguely interesting view into the development process. Detailed technical discussions should probably go in separate topics to keep things organised, but comments are generally welcome here :)

My current (but quite flexible) plan of tasks to focus on is:

  • Saved game support - particularly to allow players to reconnect to a multiplayer game after a crash or network failure, since large gameplay testing sessions are often annoyingly interrupted by dropped players. Saving single-player matches is also a frequently requested feature.
  • Better tools for measuring the game's performance. We want to optimise various aspects of performance - overall framerate, consistency of framerate (i.e. avoiding occasional pauses that make the game feel jerky), better utilisation of multi-core CPUs, etc - and I think it's worth investing in new ways of recording and analysing performance so we can improve it more effectively.
  • Better engine support for AI scripts - mainly to increase performance in various ways, and to fix some other areas that cause pain for AI scripters.
  • More efficient pathfinder - one of the current algorithms can perform particularly badly with large numbers of units in a small area (e.g. melee battles), and needs to be redesigned to provide much better performance in those cases.
This should be enough to get started with :)


God speed and hope you can complete this work and hopefully more in about a month . Best of luck .



  • 0

#3 edwardlongshank

edwardlongshank

    Discens

  • Community Members
  • Pip
  • 59 posts

Posted 24 October 2011 - 04:46 PM

To be honest i don't think its worth putting much effort into reviving disrupted multiplayer games, if its any more than 2 players it becomes very difficult to get everyone ready to continue the game.
  • 0

#4 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 24 October 2011 - 07:27 PM

Day 1

Worked on improving the engine support for saved games.

As a basic introduction: The problem is that the game has a load of complex data structures storing its representation of the world in memory - data about units like their positions, hitpoints, the paths they're moving along, and hundreds of other things, along with each player's resource counts and the explored regions of the map and so on. All that data needs to be flattened into a string of bytes that can be saved into a file, and it needs to be portable between different computers. The saving process also needs to be comprehensive: if we forget to save one minor bit of data (perhaps we fail to store the wing positions of a butterfly), the game will play out differently when you load it back (perhaps a tornado will no longer form), causing unpredictable buggy behaviour. (Not that we actually have tornadoes in the game. Or butterflies.)

As an example of the data we need to save, this is a female citizen:
Spoiler


The engine already supports most of the functionality needed for this, but it's hard to verify that it doesn't have a few bugs hiding in it. I've now added some code that developers can enable to test the correctness of the saving/loading code. It'll automatically save the game after every single state update (about 5 times per second), load it again into a second copy of the game, run the gameplay update code on both copies, and verify that the result is the same in both cases. If we ever forget to save some data that will affect the behaviour of the simulation, the two copies should get out of sync and we can detect and debug the problem. Running in this mode is noticeably slower than normal but still quite playable.

I've also added a quicksave/quickload feature (press shift+F5 to save, shift+F8 to load) which provides another way to test this code. One known problem is that animation state is not saved - if a unit is walking when you save, it'll slide across the ground while standing perfectly upright after you reload. Also there's no support for saving AI yet, and it'll explode catastrophically if you try it in multiplayer, but the rest of it should work.

Now I'm working on extending the networking code to support transferring large files between players, with the aim of letting players reconnect to a multiplayer game if they crash or temporarily lose their network connection instead of having to abandon the match. One of the already-connected players will automatically save their game and transfer it to the new player, who will load it and then can continue with the match. This file-transfer code should be reusable for some future features too, like automatically downloading maps if not all players have it installed. It's half working now, so I'll try to finish it off tomorrow.
  • 0
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#5 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 24 October 2011 - 07:32 PM

To be honest i don't think its worth putting much effort into reviving disrupted multiplayer games, if its any more than 2 players it becomes very difficult to get everyone ready to continue the game.

This is mainly for when a single player has lost their connection to the server - the other players carry on without them, and that player can rejoin just by connecting to the server again, so it doesn't need any complex coordination between players. That's been a fairly common situation when I've been playtesting (with WFG developers and other people on IRC) and it's a major pain to lose half an hour of progress just before you were about to launch an attack on your enemy, so it seems worthwhile spending maybe a day making it possible to recover from that situation :)
  • 0
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#6 Mythos_Ruler

Mythos_Ruler

    Senator

  • WFG Retired
  • 14,961 posts

Posted 24 October 2011 - 07:37 PM

This is mainly for when a single player has lost their connection to the server - the other players carry on without them, and that player can rejoin just by connecting to the server again, so it doesn't need any complex coordination between players. That's been a fairly common situation when I've been playtesting (with WFG developers and other people on IRC) and it's a major pain to lose half an hour of progress just before you were about to launch an attack on your enemy, so it seems worthwhile spending maybe a day making it possible to recover from that situation :)

That would be an excellent feature for multiplayer. Not sure I've ever seen an RTS with such an outstanding feature.
  • 0

#7 historic_bruno

historic_bruno

    Primus Pilus

  • WFG Programming Team
  • 2,385 posts

Posted 24 October 2011 - 10:16 PM

That would be an excellent feature for multiplayer. Not sure I've ever seen an RTS with such an outstanding feature.

Agreed. Will the disconnected client automatically attempt a reconnect, and will the game have to pause while the simulation state is sent to them? It would be amazing if this could happen transparently.
  • 0
Ben Brian [ aka historic_bruno ]

Wildfire Games Programmer
Contact me: ben [at] wildfiregames [dot] com

#8 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 25 October 2011 - 12:01 AM

My current plan is to do as little GUI work as possible, so I won't change how disconnections are handled - the player will get sent back to the menu screen, and the other players will carry on as normal. (We might want to change it later so e.g. the host can voluntarily pause the game if they're waiting for the player to return, and so the player gets some informative automatic reconnection screen). Then the first player can exit and restart their game if they want (it doesn't store any state on the client), and join the server's IP as normal, and if they have the same name as a previously-disconnected player then they'll be given control of that player slot. (If we had some persistent user account identifier, we should use that instead of name). Then (at least this is the theory and I think it probably should be easy enough to implement) the host will save the state at the current turn N; transmit it; the client will load the map and deserialise the state; the host will transmit the commands from other players between turn N and the new current turn M (it carries on running while transmitting); the player fast-forwards their simulation up to turn M; maybe iterate the last two steps until the player catches up completely; then the server starts accepting commands from the new player and they've been spliced back into the game.

If other games don't do this, I don't know why, because it seems obvious when you have saved games and deterministic replay :)
  • 0
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#9 k776

k776

    Centurio

  • WFG Retired
  • 724 posts

Posted 25 October 2011 - 12:34 AM

My two cents worth: Don't use just usernames to determine who to match up. Short of implementing a login system, for now, use the same ID used in profile reporting as the re-entry key. Would that work? Any big issues doing that?

I think if a player leaves they go to the main menu. If they disconnect, the game should pause for them, the screen darkens (like most games pause screens), and the text "Disconnected. Attempting reconnect...". After 10 seconds, if it doesn't reconnect, then throw them to the main menu with "You were disconnected from the game and could not be reconnected. Please contact the game host.".

As for the transmit file plan, sounds good. Will need some type of GUI (a progress bar for the % of the file size, with status updates: "Downloading map...", "Downloading turns...", "Finalizing..." (like the map loading screen, but doesn't need to have the pictures/tips).
  • 0

Kieran P [ aka k776 ]


#10 historic_bruno

historic_bruno

    Primus Pilus

  • WFG Programming Team
  • 2,385 posts

Posted 25 October 2011 - 01:06 AM

If other games don't do this, I don't know why, because it seems obvious when you have saved games and deterministic replay :)

I guess the only "danger" is if the file transfer is really, really slow, it might not be feasible for the reconnecting player to receive the complete state and fast-forward before the game ends. In general though I think that is the solution we should strive for. AOK would pause the game when a player disconnected, and it had really buggy disconnect handling anyway (due to reliance on UDP?), so we should aim higher :)
  • 0
Ben Brian [ aka historic_bruno ]

Wildfire Games Programmer
Contact me: ben [at] wildfiregames [dot] com

#11 historic_bruno

historic_bruno

    Primus Pilus

  • WFG Programming Team
  • 2,385 posts

Posted 25 October 2011 - 01:11 AM

After 10 seconds, if it doesn't reconnect, then throw them to the main menu with "You were disconnected from the game and could not be reconnected. Please contact the game host."

Hmm, that message is bad, because it implies they can't try connecting again, but I think what Philip was saying is that they can try to reconnect.
  • 0
Ben Brian [ aka historic_bruno ]

Wildfire Games Programmer
Contact me: ben [at] wildfiregames [dot] com

#12 k776

k776

    Centurio

  • WFG Retired
  • 724 posts

Posted 25 October 2011 - 02:32 AM

I guess the only "danger" is if the file transfer is really, really slow, it might not be feasible for the reconnecting player to receive the complete state and fast-forward before the game ends.

What about implementing auto save and then when a host gets a request from a player:

if (game->savedState().size() > game->turnsSince(request->lastTurn()).toFile().size() ) {
  sendTurnsFrom(request->lastTurn())
} else {
  sendFullMap()
}

So on a huge map with 100's of units, if the player was only disconnected for 2-3 seconds, send the turns rather than the full map. If the turns are greater than the map size though, send the whole map (to avoid as many issues as possible).

Would something like that be feasible?


Hmm, that message is bad, because it implies they can't try connecting again, but I think what Philip was saying is that they can try to reconnect.


True, perhaps something like this: "You were disconnected from the game. Please try reconnect. If problems persist, contact the game host."
  • 0

Kieran P [ aka k776 ]


#13 feneur

feneur

    Cartographer of imaginary worlds

  • 0 A.D. Project Leader
  • 7,755 posts

Posted 25 October 2011 - 02:46 AM

My two cents worth: Don't use just usernames to determine who to match up. Short of implementing a login system, for now, use the same ID used in profile reporting as the re-entry key. Would that work? Any big issues doing that?

Wouldn't that interfere with the purpose of those IDs? (Namely to make the profiles reasonably anonymous while still allowing stats from the same user to be registered as just that.)
  • 0

Erik Johansson [ aka feneur ]

Wildfire Games
Contact me: feneur@wildfiregames.com



Support Wildfire Games!


#14 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 25 October 2011 - 09:31 AM

My two cents worth: Don't use just usernames to determine who to match up. Short of implementing a login system, for now, use the same ID used in profile reporting as the re-entry key. Would that work? Any big issues doing that?

What Erik said - those IDs are designed to have some limited security properties (like anonymity (or at least pseudonymity)), and reusing them in other contexts would risk weakening those properties. It'd be better to have a separate persistent ID that's designed for multiplayer usage (so e.g. you can't impersonate another player's ID, and so it could be integrated with the lobby server and rankings and global bans and whatever) so we can avoid unexpected side-effects. If we want to stop people cheating in multiplayer (e.g. connecting as other players) there's a lot more we'd need to do anyway (there's basically no security in the current implementation), so I think that should be approached as a separate task.

I guess the only "danger" is if the file transfer is really, really slow, it might not be feasible for the reconnecting player to receive the complete state and fast-forward before the game ends.

Yeah, it might not be ideal if you have a dialup network connection and an extremely slow CPU. But it's probably still better to make one player wait for 5 minutes when reconnecting than to make every other player pause for 4 minutes while the first is downloading the state. (And the (compressed) state should be pretty small (maybe 100KB but I haven't measured accurately yet) so it shouldn't be too bad anyway.)

What about implementing auto save and then when a host gets a request from a player:
[...]
So on a huge map with 100's of units, if the player was only disconnected for 2-3 seconds, send the turns rather than the full map. If the turns are greater than the map size though, send the whole map (to avoid as many issues as possible).

That should work but adds a bit more complexity (e.g. making sure the player doesn't load the autosave from a different match) - it'll probably be worthwhile if disconnections are common and the simpler full resychronisation approach is too slow in practice.
  • 0
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#15 k776

k776

    Centurio

  • WFG Retired
  • 724 posts

Posted 25 October 2011 - 06:46 PM

That should work but adds a bit more complexity (e.g. making sure the player doesn't load the autosave from a different match) - it'll probably be worthwhile if disconnections are common and the simpler full resychronisation approach is too slow in practice.

If the game were to try and auto connect for up to 10 seconds after a disconnection, I think this approach (sending turns instead of the map) makes the most sense.

My most common use case is my internet connection cutting out for just 2 seconds (thankfully not often, but it does). It comes back online fairly quickly, and the game should be able to recover from that.


So forgetting file size then (to avoid desync issues), how's this:

* Player disconnects
* Game tries to reconnect right afterwards (with screen saying so: "Reconnecting...", stops trying to reconnect either after say 30 seconds or the user hits a cancel button)
Either:
* Game successfully reconnects, using the same map it was just playing, downloads turns since last turn and applys them and the user is back in
Or:
* Game fails to reconnect. User gets "You were disconnected from the game. Please try reconnect. If problems persist, contact the game host." message. They try rejoin the game, and the entire map downloads, and then the turns, all are applied and user is back in.


That sounds like it'd work, avoid any sync issues, avoid needless user interacting for small connection hickups..
  • 0

Kieran P [ aka k776 ]


#16 Ykkrosh

Ykkrosh

    Primus Pilus

  • WFG Programming Team
  • 4,921 posts

Posted 25 October 2011 - 10:15 PM

Day 2

http://www.youtube.com/watch?v=pbjvU69mm9A

Worked more on reconnection support. It's a bit more painful to implement than I hoped (the saving/loading is fine but the coordination between server and clients is already complex and this extension makes it worse). But the above video shows a fundamentally working version, which is nice :). Should just need to clean up the code now, then test it more carefully and fix any remaining problems.

With this example (Oasis II at the start of a match (i.e. with very few units, except for trees and animals)) the entire saved game state is only about 80KB, so it should download pretty quickly to a reconnecting player.
  • 0
Philip Taylor [aka Ykkrosh]

Wildfire Games Programmer
Contact me: philip@wildfiregames.com

#17 tinoesroho

tinoesroho

    Discens

  • Community Members
  • Pip
  • 81 posts

Posted 25 October 2011 - 10:53 PM

Nice! Lack of saving games and connection issues have kept away several friends of mine, so I just wanted to give a big thumbs up. Keep it comin'! (y)
  • 0
0 AD - it amazes me ™

#18 Wijitmaker

Wijitmaker

    0 A.D. Old Timer

  • WFG Retired
  • 9,536 posts

Posted 26 October 2011 - 01:15 AM

Wow, that is an awesome feature Philip. Oh how I would have loved to have such a feature 15 years ago back in the modem days. Who knows... I may have enjoyed playing on the internet in multiplayer games. Keep up the good work.
  • 0

Jason Bishop [ aka Wijitmaker ]

0 A.D. Founder

Contact me: jason@wildfiregames.com


#19 Mythos_Ruler

Mythos_Ruler

    Senator

  • WFG Retired
  • 14,961 posts

Posted 26 October 2011 - 01:20 AM

When the Godfather is pleased, we are all pleased. Posted Image
  • 0

#20 tribalbeat

tribalbeat

    Discens

  • Community Members
  • Pip
  • 88 posts

Posted 26 October 2011 - 03:22 AM

Great work! One must wonder why more modern RTS's don't have this feature.
  • 0