White Shining Rock Logo
White Shining Rock Logo

Forward Compatiblity

July 13, 2014 | Dukus | 63 Comments

Since Banished was shipped, when I go through the process of adding to or changing the game, I'm usually thinking about what might happen in the future. Will I add DLC? What form will it take? What are the implications of the mod kit? Will future changes break mods? Can mods break future DLC? Am I going to ship OSX and Linux builds? How do my current code and data changes effect those future desires?

So I find myself at a point where the mod kit is (sort of) close to complete. I need to make a few more code changes to really have good support for user changes and additions, but the overall ability of users to provide binary compiled assets to other users poses a problem.

The problem is that if people start making mods, and I then release an OSX or Linux version, those mods aren't going to work on those platforms without the mod being recompiled.

Resources in Banished work a little bit like writing a program. When writing the code, usually you write some C or C++ code, compile it, and give end users a binary executable.

Resources for Banished are similar. If you make a model, a texture, an audio file, or general resource, you have an FBX, PNG, WAV, or text file as the user created resource. Those resources are compiled by a tool into a binary format that is fast to load and ready to use - the difference in load time between the game using the compiled version vs raw version is several seconds compared to several minutes. So shipping the raw resources really isn't worth that wait time.

So what's the problem? There's currently a lot of binary graphics and audio data that is Windows only. OSX and Linux will use OpenGL, and there are no GLSL shaders stored in the game binary data. There's only compiled DirectX byte code. And audio? It's compressed and stored in XMA - XAudio's format - similar to WMA.

If I were to just ship Linux and OSX at future date without the mod kit existing, I could patch the game data to be more cross platform, or even ship separate data - and no one would know a difference. But since there will most likely be mods out there - and in binary form, they won't work on those platforms since key data doesn't exist. The audio is in the wrong format, and there are no GLSL shaders.

Solutions? I know some companies are parsing compiled HLSL byte code and outputting GLSL. While this is a possibility, I'd like to use the actual features of GLSL and not be locked into what I can parse out of compiled byte code. Plus it makes sense for future projects to support it properly.

The audio side could be hard to deal with as well. Decoding XMA and reencoding to another audio format at load time isn't something I want to get into.

I could just ship the mod kit, and new platforms just won't work, and it will be up to mod authors to recompile their mod - and it may not just be a recompile. What if in implementing OpenGL, its better to modify some data for better compatibility? I'd probably do that. I like good code with good performance, instead of hacks to make things work. If the resource format changes, then mod authors will have to update any new materials they built. This really isn't ideal.

I could have mods include the source data, and have newer releases include the entire toolset to recompile resources as needed, but again, that locks me into not drastically changing any file formats. Plus the raw data is a lot bigger. The Banished resource folder before compilation is around 700 megabytes - the shipped data is just over 100. Mods would have similar data bloat.

I could also put out a beta of the mod kit, let people get used to it while I take care of these cross platform issues, and then deprecate all binary mods. The mods would have to be recompiled and redistributed. This isn't ideal for end users - especially if an author of a great mod stops maintaining it.

The best solution is really to make sure GLSL shaders are part of the data, and use an audio format that is usable on all three platforms.

So this weekend I've been looking at the feasibility of doing that.

I've been experimenting with OpenGL 3.2 and trying to get a sense of how long an implementation will take. It's partially done already. If nothing else this will make sure I can build GLSL shaders using the material system and shaders already in use. Ideally the windows version will have a new OpenGL renderer that will be selectable just like DX9 and DX11.

On one hand this is great - From my graphics programmers background perspective, I'll finally have a great test bed to really see which API runs the fastest (since this has been a debate from the beginning of time), since I can pick a scene and switch between rendering APIs at runtime. On the downside this increases development time before the mod kit appears. Although the DirectX implementations are only 44K(DX9) and 65K(DX11) of platform specific code so I assume GL will be similar. That little code shouldn't take too long, right?

On the audio side FMOD seems capable of playing XMA samples, so switching over to that might not be so bad either, and if I do a little research to make sure it'll work, it may not have to be done right away. Once I do port though, the API is available for all the platforms I want to support. Other cross platform audio libraries are out there - really i just have to decide on what format the audio will ship in.

On the upside of the extra development time to make sure mods work in the future, having OpenGL and FMOD (or other cross platform audio library) implemented will certainly bring me closer to having ports done as well. There's not too many other platform specific parts of the engine. There's file I/O which should be super easy, the math library, which is currently written in SSE - I just need a C++ port of it - also easy(ish), and input handling - reading from mouse, keyboard, and game controllers.

Now if I can stop falling asleep reading the OpenGL spec while trying to get the right triangles displayed on screen...

Leave a Reply

Your email address will not be published.

63 comments on “Forward Compatiblity”

  1. Wow, that sounds complicated.
    I would have released the Windows mod & let modders / fans remod for non Windows builds later.
    However, your way probably sounds intellectually more satisfying to you as the programmer.
    As always, good luck.

  2. Seems like things are coming along nicely! All I gotta say is take your time, and it'll turn out right. I'm in no rush for the mod kid to be rushed. I'm keeping quite busy, haha. Keep up the great work! ^_^

  3. I can sorta understand what you're getting at with the mods and cross-platforms. I think that if you want, you should see if you can cross platform, but if you can't do that, add a mod kit. But I like the idea of DLC, like add new buildings, professions and character models. I think DLC is a good idea. Whether you make it free or paid for, it's your choice.

    Keep up the great work!

  4. Most mods out there require reprogramming/updating when the developer makes an update - Minecraft is a perfect example of this. Have faith in your modders! They can keep up ... just my two cents.

  5. It's appear that i was afraid for totally nothing on last post, soooo nice to hear you talking about OpenGL...
    Also, with allowing user to use new OpenGL as DX9 & DX11, you should also choose OpenGL as default on Windows, OpenGL is damn good on windows, OpenGL simply need less creepy games for user to find it out.

    Good continuations & hope i'll be able to buy it soon as linux user,
    -ร‰ric

  6. You've been doing such a great job. Most developers wouldn't even think of, much less care, about the people who will be modding the game. I appreciate it.
    I'd say do whatever makes it easier for YOU to do. If it's going to create 3x more work for you to put it out quick, go ahead and take the extra time. We can have fun spending the time by planning all the fun mods we'll make. Lol! To me it sounds like it makes more sense to make it cross platform to start with... But then I didn't understand half of that. Lol!
    On the other hand... There are a lot of skilled modders out there. And let's face it, even if someone stops updating a mod, there will be someone else out there capable of updating it for us...
    So, do whatever you think will be better in the long run. It will be worth it. ๐Ÿ˜€

  7. Oh, and I forgot to add, I've been hoping since the beginning that we'd have a chance to see more of your genious with more DLC. I was especially hoping for more animals and more processing options... (ie: flour) ๐Ÿ˜‰

  8. DirectX vs OpenGL speed still won't be an easy comparison, since video drivers are an important part of how each platform performs. That's the main reason the debate still rages.

    I'd agree with Omarflyjoemacky, modders are a resilient bunch - most devs don't show your level of consideration for modders, so they've gotten used to having to redo mods for every update. That said, it's great that you're thinking of those sorts of things, and modders will appreciate the lengths you're going to to make things easier.

  9. Well, you're already a step in the right direction compared to games like Minecraft, that decided to use Java instead.

    Keep up the good work, I'm sure you'll work something out.

  10. > I could also put out a beta of the mod kit, let people get used to it while I take care of these cross platform issues, and then deprecate all binary mods. The mods would have to be recompiled and redistributed.

    Please do this! ๐Ÿ˜€

  11. If OpenGL proves to be faster by any margin than DX I wouldn't mind an altogether dropping support of DX since there will be no benefit since with features both are very evenly matched. All DX has is going the Microsoft Marketing Engine. This is hurt since Microsoft has lost much hold among gamers and enthusiasts combined thanks to the tile-ization of Metro-UI.

    I wouldn't mind waiting but the beta sounds good too.

  12. Personally, I feel like as a user of this product I am getting antsy for the ability to create mods. You have a whole section dedicated to modding in this website's forums and this is the second recent announcement about the development of a mod platform. I appreciate you keeping the community up to date, but aren't you partially motivated to communicate with the community because you feel like a modkit is overdue?

    Just come out with a Beta version. Let people play with that, and you can completely wipe or make otherwise irrelevant anything people make in the meantime until you are satisfied with the real, final product.

    Don't be so concerned about what users are going to spend their time on in the short-run or middle-run. But also don't keep everybody waiting for the long-run. Personally, I want to experiment and I don't care if I have to remake whatever I like for a final version. I've checked this website daily waiting for these last two updates and I'm anxious for some flexibility. I'm not playing as much now because, based on these recent community updates, I'm expecting that I can soon start a new game with 1 or 2 major tweaks. And I'm excited for that! But that is just my two-cents.

  13. Speaking as a software dev, your proposed plan sounds like the best one. Take the time to do things right.

    Love the game! Thanks for all your hard work.

  14. this all sounds very complicated to me, all i want to say please make sure for us mod users make it easy to install them, last thing i want is having to watch youtube videos on how to install banished mods...but whatever, keep up the good work and let us know how things are going along ...like you have so far , A+ to you my friend.

  15. Nice effort but is there any progress on the game crashes when saving bug? I can't play any more until this is fixed, totally game breaking.

  16. I like this game, but i hope that you wll take your time to bring in new ideas into this game.
    Let it be a windows game, mods will be very nice .... but is there a chance to make the game more interessting.
    E.g. like Anno: Settler, citizen, merchants and all need special goods. Something like this!

  17. Good to hear the game is coming to or hopefully coming to Mac and Linux. I know its hard to change the code, but it should run smoother to have a native build instead of just porting. Continue the good work, and take a break once you've finished this mod kit and the Mac and Linux support. You deserve it.

  18. Awesome, I still keep progress of your development, although I honestly stopped playing because it got a bit boring.
    But if I were you, try to get it cross-platform all the way, that way you keep the end user and mod developers happy.
    Yes, it might take a little bit more time, but it will definitely be worth it.
    If it works well on Linux, perhaps I will start using Linux more in stead of windows, because the biggest reason I am not using Linux is because there are just a few (lame) games.

  19. I was expecting something like this and completely understand your choice.
    Good luck Luke!

  20. I admire your courage to address the cross-platform issue, as a mac user, I have often been deprived of features by lazy programers, even it only involves manual installs of mods. The quality of this game is due to your dedication to making the best game possible. Thank you for being different!

  21. "Quality First" This seems to be a recurring theme with you Luke. We appreciate such excellence and wish more people/companies did likewise. We know that whatever you create will be worth the wait.

  22. i think if a mac and linux build are available
    the game will sell more than if mod's are added

    people say linux users should use wine
    but im not going to buy a game where i dont know if it will run or not

  23. Wow, this is what I call ยซupdating fans with current developmentยซ. Kudos!

    However, to be honest, the whole mod kit thing is not for me. Do you remember back in the days, in the 90s early 00s, games were like they were. No one could modify or play around games files. I just don't understand why people absolutely want to modify things, play with files and whatnot. The game is great as it is, few enhancements and suggestions implementations would be much more appreciated from the general public who play the game. Taking all your precious time for stuff that only a few will enjoy, is kinda pointless. But hey, kudos to those who enjoy that part of gaming... I surely don't. I would prefer DLCs, my wallet is ready to open for more content. Keep the DLCs coming, updates for minor issues, and I'm a customer forever. If there's too much time spent on those mod kits, I might just lose interest in the long haul. And I'm speaking on behalf of majority of players... people interested in mod kits might make great additions to the game, but at the end, it won't be fair for the casual gamer who likes the game as it is and if achievements are attainable with mods, that would suck even more. Perhaps more achievements? DLCs? Can't wait to see what will happen.

  24. Just support windows versions, it will be easier on you Luke. Everyone else can just buy a windows PC, for gaming.

  25. Make the modders recompile their mods, since it:
    A sounds like the best option for you.
    B trust in the community to put time and effort into the mods, as well as recompiling them.
    Or if every mod is shipped uncompiled and has to be compiled once before starting them, mods wont starve from a modder losing interest.
    Greets

  26. To Abalister, as a long time modder of games and as a programmer I can say that getting into the guts of a game is fun for me in seeing how it all is tied together. Getting to change things is a perk, adding things is a sweet victory!

    I say release the mod kit, then depreciate the mods when cross platform is done. Most of the folks who can mod are capable of updating them.

    The game is excellent, but late game really needs some boosts. When a city is well established it comes down to space as resources can be easily traded for. So mods that expand late game enjoyment is what I'm after.

    That's my 2 cents.

  27. I'd say if putting in Forward Compatibility takes a few months then release the Mod kit for Windows and let the modders handle changes moving forward. They're pretty capable on the whole ๐Ÿ™‚

    If however it's a short delay then having everything in place is pretty useful and would probably be appreciated by many.

    The less things that will stuff things up moving forward the better.

    IMO

  28. Keep up the great work! You've got thousands of people here support'n ya. Do what ya need to do to make your game great!

  29. A lot of game designers begin with a single platform in mind, because that way, something can be marketed quickly. Programming across platforms takes a lot of time and effort, and yet, for a game to reach its true potential, that's exactly what has to happen. For many reasons, I'm not a huge fan of Steam, and would not have purchased it had it not been offered elsewhere. I do not like the idea of the DLC, because I think it encourages community fragmentation, and is likely to be a distraction.

    I think that the only viable option you have currently is determine which platforms you intend to support, and work to insure compatibility. OSX and Linux are essential, particularly considering the potential for mod designers to manipulate the game into a multiplayer capability (Minecraft). Yes, that means a large delay in expansion and completely derails the mod system you were going to implement, but it has to be done.

    This is an interesting game for many reasons, and I think those interested in seeing how this plays out will wait patiently.

  30. Most AAA PC games parse all the necessary art assets when they are needed. Is compiling art assets ahead of time a common thing to do on PC now? Seems like overkill.

  31. Banished is an incredible game with a lot of potential to be even better with the addition of extra gameplay modules. Insuring a smooth gaming experience is key and should be held above all else, so there isn't any rush!

    Great work and I love your honest posts.

  32. The things you're doing are going to make sure your game will last for a long time to come, at a quality that few AAA games manage to go for. (Looking at you, EA.)
    Please keep up the good work. Those of us who enjoy your game don't mind waiting a little longer for features which will enhance our playing experiences. ^-^

  33. your quality as person reflects in your work, and you sir is far ahead from any other dev/gamer I ever seeing. Thanks

  34. Your player audience is going to expand ten-fold after this mod kit comes out. Banished is an amazing game, and your continued dedication is a tribute to it. You're giving us what we asked for, and few would do that for free.

    I would honestly pay another $15 easy to see an expansion, but I am also excited to see what other games you will make. Don't lose hope! Thank you!

  35. Banished is a great game. If I was in your position I wouldn't bother with mac and linux. You might loose a bit of clients by not offering the game on those platform, but you will gain some by providing mods.

    Good luck ๐Ÿ™‚

  36. Releasing a beta mod kit would be a good idea in my opinion. While you are working on the code changes the community could be testing out the mod side of the game. The majority of the mods would be recompiled by the authors and any good mods that aren't I'm sure someone else would redo.

  37. I bought this game (twice!) hoping you would create a Linux Port. I'm on Linux 99% of my time. I develop here, I work here, I even play under Linux most of the time. (And I have a huge library of games, many of those which only run under Windows)

    If you don't support Linux in the future, well, you never promised to. But speaking for me, that would've been the very last game I supported that hinted multi-platform support but was not released with it.

    Personally I'm sick with DirectX and all the quirks of Windows.

  38. Just wanted to say thank you for Banished, and thank you for working toward a linux port!

  39. I don't see you getting much return on your investment for Mac/Linux. If you want to do it based on your own values I can respect that but my opinion is you should keep things simple for yourself and just keep adding value for your Windows audience.

    Great game, appreciate what you've done.

  40. I bought the game when it came out and loved it, I check up on you for updates every few months. I admire your dedication and honesty.
    I decided that all future games from Shining Rock Software get priority from me and will always be considered when looking for new games to buy. If there is one company out there, that I would trust with Quality over Quantity, it's this one.. it's you .. Luke

  41. I would advice to wait until the cross-plateform moding is done before releasing it if it's not pushing back the release of a few months, because it would be nice to be able to use mods created on other plateformes.

  42. I don't think modders will have a problem updating their mods later, especially if you label the current build "beta" with a big notice pointing out that it will be deprecated when the final release comes out.

    That said, I agree you should go without whatever allows you to build the neatest, smartest code (no hacks) on your side, even if that means we all wait a bit longer. I suspect the Banished community has above average patience anyway ๐Ÿ˜›

  43. Luke,

    You're a beast amigo. The amount of work you put into this project is simply epic. It's so rare in these days of micro transactions and Day 1 DLC to find a game where the heart of a programmer is so evident. Thanks for sticking with the community and continuing to make an already great game even better.

    Cheers and many thanks m8!

  44. I wont go into any lengthy discussion since i'm not a graphics programmer. However i am a linux user since ~15 years back. You should read about what steam's testing on Windows/Linux DirectX/OpenGL resulted in. Theres been many articles, here's one: http://www.extremetech.com/gaming/133824-valve-opengl-is-faster-than-directx-even-on-windows

    It seems that DirectX simply isn't adding any value anymore, if you're already used to it and have a background with it then it's still a time saver. But if your re-writing, or willing to learn writing in OpenGL can make you more or less cross-compatible with Windows, Mac and Linux from scratch.

  45. if you do think of doing any more updates could you please have an option of upgrading the farms and other things as well like say for i.e. the year is 1/5/22 things could look more factorised like the farms could produce 10x more with like machines so it is not always as hard to keep the population up.

  46. Please, please develop the game as long as possible. Banished is the Reason why i come back to build-up-strategy (dunno if this word exists in english language, i hope you know what i mean). i completly forgot how this kind of strategy generates so much fun.

  47. I'm dying for an OS X release of the game. Whether it be independent, on the Apple app store or on Steam, count me in!

    The addition of modding sounds great. Would love to have a bakery - an old style windmill to grind wheat into flour and have cows make milk too.

  48. As someone else mentioned, I love your honesty as well.

    You literally take us through your thought process, something that so few developers do.

    I'm eager to see what mods arise, but don't rush. Keep up the excellent work!

  49. Do what you think is best Luke. I and many thousands of others admire your work very much, and we trust you to do what is best for Banished and possibly future works!

  50. The fact that you're still handling all of this personally is amazing. I can't wait to see what happens next!

  51. Pardon if this is a fairly uninformed question -- I wonder if your mod kit will allow users to build mods of the processes used for each simulated villager's decision making. For example, say I wanted to mod the game to randomly assign villagers different profession preferences, would that be possible? Maybe you'll tell me to just go play SIMS ๐Ÿ™‚

  52. You really should process with the linux developement, but also keep things compatible. There are a lot of ppl waiting for this game released on linux.

  53. I'm a software developer myself and I _absolutely_ understand your need to have a clean and good solution for these things. This makes things so much easier to extend and maintain in the future. And also it's more fun to program if you are proud of your solution to common problems!

    Please do not haste through this process, take your time, it will save lots of work in the future. Judging from your clear thought process in your blog-posts I know that you are aware of that.

  54. Or make compilation a part of mod loading process. Ship mods as raw files, compile them at runtime, when loading first time. This could solve all compatibility issues vs how much bigger will the files get?

  55. Luke confirmed in an currently running AI related livestream that mods will indeed not allow own coding but we only can re-configure existing behavior. So no "real" mods as compared to games like Minecraft or KSP with that mod kit ๐Ÿ™

  56. Damn, I gotta say, these 20 bucks are some of the best I ever spent.
    Reading your posts about cross-platform support really shows that you actually care about the product you're releasing and, all things aside, FMOD and OpenGL is a great solution. It's far from perfect, but for future projects it will really help you.
    I know severaly programmers who love arguing about DX and GLSL, but at the end of the day, Linux and Mac support means going something OpenGL. ๐Ÿ™‚

    Great news ! ๐Ÿ™‚

  57. Any news? It's been nearly a month since we've heard anything and frankly I am really waiting on the modkit release.

    I know a lot of people have said the same, but I want to add my support to the idea; MOST games with mods put the onus on the mod creators to keep the mods compatible with new releases, it's not generally the responsibility of the game creator.

    Really looking forward to modding!

More Posts

Code Rot

April 17, 2022
1 2 3 47
Back to devlog
Back to devlog
ยฉ Copyright 2021 Shining Rock Software
Website Design & Branding by Carrboro Creative
menu-circlecross-circle linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram