White Shining Rock Logo
White Shining Rock Logo

Thoughts on Linux

April 9, 2016 | Dukus | 53 Comments

I wanted to share a few things about my experience getting Banished working on Linux. I haven't seriously used Linux to develop software since around 1997-1999, where I just used vi and a Makefile.

Getting Started

Getting linux going should be easy right? Download ISO, burn disc, install in new machine. But then the installer fails to make partitions on my brand new SSD! That was ok, nothing is ever easy - so I decided to setup the partitions myself using gparted. Does it work? No. After two days I figured out that for some reason the install wouldn't work with the drive plugged into easy-to-access SATA5. Plugged the drive into SATA0 underneath the video card, and we're running!

I like to have multiple machines that can compile the code, which ensures all required files are checked into source control, and that multiple GPUs render correctly. I had an extra laptop with a 600 series Nvidia GPU, so I got linux going on that, but had another few days of trying to get the Intel/Nvidia Optimus setup working under linux. I'm still failing on that, it seems overly complicated, but I can compile on it - just not run. Bah.

Development Environment

There are a zillion IDEs available under linux. I read about a ton of them and their associated debugging options. There are too many to actually try them all in a reasonable amount on time. It almost made me want to just go back to my roots and use vi and a make file. But I don't really want a Makefile, and I don't want to use cmake. Especially since the Mac and Windows IDEs just have a project and do their own dependency checking.

So I ended up going with SlickEdit. Slick edit has no need of a make file, just add files to the project. It's relatively cheap for a single user license, and has key bindings that match Visual Studio. If nothing else, it let me jump right in and not fight too much learning a new IDE.

Graphics Drivers

My linux machine is an 8 core AMD with and AMD R9 graphics card. Installing the graphics driver was no problem, and getting OpenGL working wasn't too bad once I read through the GLX documentation. I did go through a few days of random crashes that would freeze the entire X11 desktop (always on a GL context related call), my only recourse being Alt-SysReq-B. Boo.

Turns out I forgot to implement a certain platform specific locking mechanism that kept my background loading thread for executing graphics calls at the same time as the main rendering thread, causing some data corruption of lists of resources. Not so bad, but the entire system halting made for very slow debugging.

X11

So when I told fellow programmers I was going to deal with straight X windows and GLX, they told me I was crazy and would regret it. But really, all I need is to create a window, and render OpenGL into it, and get some keyboard/mouse input. So that's what I did. However when searching for answers to questions that weren't made quite clear in the documentation, often people that had the same questions I did received answers like 'Use SDL', or 'Use GTK or Qt'. That's amazingly frustrating and just adds noise to the internet.

The only real problem I had with X11 is that the includes #define a lot of symbols that conflicted with identifiers in my code. I guess that goes with the territory of using a C only library that was written long long ago. I ended up having to forward some X11 symbols for use in my headers, and only include the actual header where needed.

When said and done the X11 code I had to write was far far smaller than any 3rd party library that handled the same things, has no extra dependancies, and compiles fast.

Audio

There seem to be a lot of options for outputting audio on Linux, but all signs seemed to pointed toward ALSA for my purposes. Since I have my own mixer that outputs PCM data, all that was required was to play a single stream of audio, which is pretty easy.

The only major frustration here is that the method I like to use (and use similar methods on Windows and OSX) is to get a callback every N milliseconds and then fill in an audio buffer. However under Ubuntu the callback mechanism of ALSA isn't implemented (apparently unsafe?). So I ended up writing my own audio thread that polls for the audio buffer to become available, fills in the audio data and then the thread goes back to waiting.

Porting everything else

Most of the remaining code was easy. It shares with OSX, so threading, memory management, critical sections, file/io, etc was already written and worked without an issue, except for a few date and timer functions.

Over all I liked the linux experience of porting much better than OSX. I didn't have to use another language, and I get to control the overall game loop and when events get processed. Then again, I already had a lot of compatible code ready from doing the OSX port, so there could be some bias there.

Going forward

Linux as a development environment is pretty good. But.

One of the more frustrating things about doing these ports is that currently the data is only compiled on Windows, then has to be packaged into a single file, and then copied over to the target machine. This is fine when things are said and done code-wise, but during development a lot of debugging requires changing data.

For example, lets say I just want to debug vertex skinning of animated characters because it's not working. This requires a change to the vertex shaders that will be used. So I turn to the Windows machine, change the shader, compile all resources, build a pack file, copy the pack files to a shared location, and then start debugging on Linux or OSX. I go back and forth doing this 20 times before fixing the issue. It's slow.

I really need to get my full toolchain working on both OSX and Linux - so that regardless of what machine I have with me I can fully build my game and work normally and faster. Once I do, the game will recognize that a resource has changed, and compiles it just as it's loaded.

But this requires me changing a bunch of Windows only compilation functions. This includes loading images, building compressed textures, building fonts, loading and converting audio data, and setting up FBX to work on other platforms. There's probably more. It's not insurmountable, but it's a good chunk on code. So it'll probably wait until the next project.

Leave a Reply

Your email address will not be published.

53 comments on “Thoughts on Linux”

  1. A fascinating read! thanks for all your effort, I salute you!
    Waiting anxiously for the Mac OSX version...

  2. There's a more than one reason as to why people tell you to use SDL. You get OSS, PulseAudio, Wayland and Mir support for free with it, and you don't need to deal with Obj-C on OSX.

    Still, I get that you don't really like using other people's code.

  3. I´m user of all OS above, and it was a nice reading "report" of a programer that don´t use fancy words that even for another programer is hard to get just to sound smartass.

    You made it interesting. Thanks.

  4. FYI, Xlib/XCB are going into obsolescence.

    Just use CMake. Build servers understand it fine. No need to multiply weird build procedures.

  5. Have you thought about using a continuous integration service to combine an asset build on Windows and several platform specific code builds? I think Appveyor support Windows builds. A CI service takes a lot of the repetitive work out of the build and test cycle!

  6. I have to say, this was a nice read. I do have concerns about using X11 and OpenGL directly. Have you tested it with more than one monitor, SDL generally takes that pain away (there's a reason people use it, and why it's bigger than your code).

  7. Thanks for sharing. Kudos on getting naked X11 to work.

    Pulseaudio can do the callback buffer filling method.

  8. Interesting read.

    Are you going to be making 1.0.5 not beta any time soon? You said "about a week" like 4 months ago.

  9. Please please please use SDL2, nothing worse then some dev anywhere complaining about issues and making it harder for users based on some ideology. Using SDL2 will make your life easier and everyone else's life easier. Developers start complaining about stuff broken on other platforms but refuse to use working 3rd party libraries. When you start getting a mass of input/monitor support issues maybe you should go back to the drawing board an use SDL2 instead. ETS2 and American truck simulator developers refuse to use SDL and its makes it a nightmare and their input is broken on linux because of it.

  10. Although I'm not a programmer I could understand everything. Very interesting to read Luke, good luck with future development!

  11. You can use code::blocks. It's a cross platform IDE which work well with vc gcc clang.

  12. About your laptop, the only way I got Optimus working properly on Linux is with Ubuntu (or derived) and Nvidia Prime.

    First, get rid of all previous attempts:
    sudo apt-get purge bumblebee* primus libvdpau-va-gl1

    Then install nvidia prime and drivers (assuming 352 is the last one available on standard repositories):
    sudo apt-get install nvidia-352 nvidia-prime mesa-utils

    Nvidia Prime does not auto-switch between both GPUs. In order to switch between Intel and Nvidia, you have to launch Nvidia Settings. In Prime Profiles select the GPU you want, enter your password, then log out and log in back to your session. And voilà.
    You also need to know Vsync does not work on Nvidia GPU.

    To get access to the most recent Nvidia drivers (currently 364.15), you can add this PPA to your source list:
    https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa

  13. I'm a full time Linux user and a developer.

    For the sake of all that is holy, *please* use SDL! There is a very good reason it exists!

    A lot of what we have on Linux is pretty horrible, and SDL saves *both* the developer *and* the users from that pain.

    X11? Horrible. And outdated. And will most likely be replaced relatively soon with Wayland/Mir. Are you going to update your game when that time comes? SDL will give you that for free.

    Multi monitor support. With raw X11 without explicit support (read: effort on your part) your game will most likely be broken on a multi monitor setup. Do you want to deal with this? SDL will give you that for free.

    Multiple sound subsystems. In the old times we used OSS. Then we switched to ALSA. Then we switched to PulseAudio. Some users (like me!) use alternative sound systems like JACK. Every one of those has its pros and cons, and there are reasons why some people prefer one over another. Are you going to support every and each one of those? SDL will give you that for free.

    Do you see where I'm going? People that say to "use SDL" do *not* "just add noise to the internet" - they do it for *very* good reasons. By doing everything manually (interfacing with X11 and ALSA directly) you're just wasting your time *and* you're going to give the people that play your game on Linux a horrible experience.

    Please don't do it. Please save yourself (and us!) the pain and use SDL.

  14. @ J

    You forgot to say :
    "SDL officially supports Windows, Mac OS X, Linux, iOS, and Android."

    One lib to rule them all! 😛

  15. Nice read. going directly at X11 seems like really bad idea though. Its the plague of older games and even some SDL1 games that dont know how to use multimonitor correctly, cannot alt+tab out of them and so on. SDL2 fixes all that, makes user experience a lot nicer.

    so please consider using SDL.

  16. PLLLLEASSE USE SDL..... It's going to make things FAR less painful for you for future proofing and making sure it works for most people.

  17. Really glad you are working on a Linux version, wanted to get your game for a while but am Linux only. and avoid wine.

    good luck with the porting efforts.

    I think everyone's responses to "USE SDL" are hilariously insensitive. when has arguing with someone ever convinced them of doing something.

    So instead, I offer encouragement to do it your own way and find your own path after all, that's how Linux works. And maybe in the future you will change your mind, but that will be your choice.

    Is there a way I can be notified of a Linux release down the track so I can purchase?

  18. Like @RedKetchup and @JohnnyCash I was holding my breath for the completion of patch 1.0.5. Still waiting - but had to start breathing again.....

  19. To the people advising third-party libraries like SDL : when will stop being updated/supported ?

    He basically made his project immune to dependency support termination, ensuring his software perenity far into the future.

  20. @Awilen

    SDL was under active development from 1998 to 2012. SDL2 picks up from there with some minor porting and is still being developed to this day. So roughly speaking, 18 years so far and still going.

    Even if they were to terminate the project, another developer could easily fork it and continue maintaining it since it's completely open-source. "Dependency support termination" is more an issue with proprietary libraries than open-source ones.

    We're mostly concerned about him directly using X11 because the Linux ecosystem is moving towards a new display server called Wayland. On top of that SDL adds a bunch of important abstractions for handling different controllers and audio servers too, unlike QT and GTK it's overhead is rather minimal and there's little reason not to use it.

  21. Please use SDL2!
    X11 is not the way to go. You will have a hard time and will regret using X :-/

  22. @enetheru: Well, he could have open sourced the engine and sell only the assets. Fans would probably do the porting themselves. It's not like people don't want to help.

    As for the dependencies; please use SDL2. I understand that you lose your own event loop, but X11 is going away in ~2 years at most.

  23. talk about x11 becoming deprecated is insane, and is coming from freedesktop echo chamber.

    the basic strategy that freedesktop has been following for years is to replace working and tested libraries and applications with their own abstraction layer. the abstraction layer in turn makes it easier to replace traditional unix base with custom freedesktop created code of dubious quality. they take cavalier attitude to dependency hell and flood any constructive discussion with "just use gtk3" or "just use sdl2" non-advise. you can always spot one of the infected apps, because they start a dozen of service, depend on a slew of freedesktop libraries, have insane memory profiles, all tell tale signs of bloat.

    while sdl2 will provide you with handy abstractions, there's really no reason to use it, if you don't need it. if and when you genuinely run into problems that you can't solve, then it would be perhaps a time to switch.

  24. Install Mint 17.3, then open the Driver Manager to install Nvidia drivers. That'll give you working Optimus.

  25. You're obviously a true hacker 😉
    I admire your direct approach, but I would rather have considered "Linux" as a very generic technology and focus on getting the best from a specific distribution: either Ubuntu or SteamOS. When you write "There are a zillion IDEs available under linux. ", you might as well be talking about Android as some realtime OS for micro controllers in a nuclear station. Pick a dist with some interest in being a gaming platform. 😉
    Then I would have picked a graphics-library which freed me from most issues with X-window managers.

  26. For the love of all that is holy, NEVER use a laptop that has ANY kind of GPU switching, particularly not Optimus, under any operating system, for serious work.

  27. The amount of people who are trying to convince other people on the internet to "use foo" or "not use bar", is blowing my mind.

    Kudos for the port, and the writeup of your experience with it.

    Fwiw, I still write on vim and build with makefiles. I love the flexibility and power of this approach.

  28. Its not people trying to convince you to do something, its people giving you solid good advice that you should take, because its tried and true and has been vetted, xorg and xlib won't be here forever, but sdl will have a longer life, until sdl3 comes out...it the next 15 years.

  29. I don't usually butt in and make unsolicited recommendations, but as a potential customer (thinking of other potential users) I need to say that I may not be able to buy Banished if you continue to use X directly... I was waiting to buy it when I heard it was being ported.

    Using X11 directly will cause problems. I can't stress this enough. I can guarantee that you will receive a huge number of bug reports because of unpredictable X behavior, especially across different configurations.

    X11 also causes various graphical issues such as tearing, screen blanking, and performance penalties. These are intrinsic to X and cannot be avoided, with different configurations having different issues. Many desktops on Linux are now closing these issue reports as "fixed" because of the Wayland migration; the current mindset among desktop developers is that X is a lost cause only fixed by not using X. As time goes on, X11 on Linux for many users will become increasingly broken as systems shift to Wayland and mir.

    I can guarantee you that SDL2 is the right choice for Linux. Treat SDL2 like the 'native' Linux windowing system. It's backed by several game developers including Valve, who has videos on porting games to Linux using SDL2. If you are interested, it will also give you a leg-up in porting to iOS and Android - though I haven't played your game yet to know if this is practical for touch anyway.

  30. On behalf of all the Linux community you can release 1.0.6 now for windows and mac and we will wait for 1.0.6.* for the SDL2 port, we would rather wait for a decent working version of a game then have to flood your mail with bug reports because it will only work on like 50% of the linux boxes out there.

  31. Thank you for taking the time to make a Linux version of your game. We really do appreciate all of your efforts. The 'nix operating systems are the future. I use SteamOS.

  32. Don't use X directly. Don't just grab input devices and try to recreate the OS.

    I'm not putting that software on my box. I'm just not.

  33. I don't really understand all of the negative comments here, as they seem to stem from some misconceptions (or from drinking too much freedesktop koolaid).

    X11 will _not_ be removed in future versions of Linux distributions. This would be stupid since there are A LOT of X11 programs in existance, which will not simply be "automagically updated" to use Wayland. Even the Wayland FAQ itself states that X11 support will probably always exist. So stop claiming that Banished "won't run" in the future, that is just wrong.

    ALSA is the only official sound system supported by the kernel. (OSS has been deprecated since kernel 2.6)
    Using the ALSA library does not hinder anyone from using another sound server. Pulseaudio, for instance, can act as an ALSA sound sink. It has to, otherwise older applications simply would not work when using Pulseaudio. JACK can also work as an ALSA sound sink. Hence, ALSA is the lowest common denominator in the Linux world, and it does not hinder anyone from using something else as the backend. The only functionality which would be lost from users using Pulseaudio is per-application sound volume. Not a big deal.

    As long as the X11 port does not grab keyboard input, and the application respects EWMH/ICCC hints, you should be fine. For instance, the "correct" way to set your program to fullscreen is by sending the window manager a _NET_WM_STATE_FULLSCREEN atom, and then it is up to the window manager to change your application to fullscreen, in the current monitor. So claims that "your application won't support fullscreen on multiple monitors" is a bit misleading. In modern X11 applications, this simply is not the job of the application itself, but a job of the window manager. And as mentioned above, X11 support will simply not cease to exist.

    The claims of "tearing" on X11 might be true on some systems, I don't know. But even if it is, it's not like Banished is a fast-paced FPS or something. There is not a lot changing from frame to frame, and I'm skeptical if tearing would be very noticable even on systems where it is a problem.

    Stop the fear mongering, everyone.

    Instead, I applaud the effort to make a long-awaited Linux port! 🙂

  34. Oh, and I have been using systems with buggy video drivers before, and on one system, the only programs which would (predictably) NOT run were the ones using SDL.
    I suppose the SDL library did some very specific thing at startup which my graphic drivers/hardware did not like.

    So, using SDL is not a guarantee for anything.

  35. I hope that linux version will be opensource and someone will rebuild the game and add some kind of multiplayer

  36. @Thomas there are tons of free and opensource engines out there, use one of them to make your own game like this with multiplayer.

  37. not sure how you can justify coding on linux then... porting this is going to take a huge amount of time etc, when something else could be done in windows?

    How many people are going to play this on Linux?

  38. @Hayden: Probably because it's an engine port as well which means you could bringt future games directly to windows, mac & linux. What's your problem?

    Probably his income is still fine from the banish sales.

  39. Facinating read, both the article and the comments. Something tells me SDL may be the way to go :O
    I saw Banished on youtube and thought, that looks good!, I don't use windows these days, and finally got around to building a steam box, so the sooner the better, keep up the good work

  40. Hey, I don't mean to be short on this topic.

    There is a game on the app store for phones called "War Ages" is using footage from both your game + "Ark-Survival Evolved" in his/her ad's. I'm not sure if you are aware of this, but I am sure something can be done about it.

  41. GLFW is way smaller than SDL and more focused on OpenGL apps (while supporting controllers abstraction).

    It's widely used in the linux gaming ecosystem.

  42. You done a great job so far and had a lot of work and created a beatiful result.

    But you don't need to make it harder for you than neccessary!

    - To check your code repository, why not use a virtual machine(s)?

    - I think every linuxuser faces installation problems on new machines. As your system requirements are relatively low, at least for simple tests you could also use a virtual machine setup with linux to run your code for testing

    - Just some usefull, free IDE's, som for Linux, some plattform independent:
    - Geany
    - GNAT Programming Studio
    - KDevelop
    - NetBeans IDE
    - Code::Blocks

    - As many said, use SDL. It solves all your problems with X11 and audio.

  43. I appreciate your effort, but honestly, I can't not hide my disappointment regarding the current state of the game. You say you're having serious trouble porting the game to Linux and while the development version of Windows is stopped, it's not touched for a long time and it is incomplete, when do you think upgrade and finish algorithms citizens which they are slow and sometimes erratic?. When will you extend the game with new buildings and functionalities?. The game is beautiful and it has much pottential, but there are many things that need to improve in the game. From my point of view, you are wasting your time and effort in a ported that is to much time consuming and I feel you are abandoning the game development to focus on the Pharaonic adventure of porting o Linux. Seriously, why not forget about it and focus on the game?. Thank you.

  44. In addition there seems to be some kind of problem with steam mods workshop because some mods causing crashes, they can be installed even being of an earlier version. I had to remove all my mods because I didn't know which was the conflictive. Also I think that the worker distribution system is too tedious and should be redesigned. I see you have a beta branch which is still not finished and have not yet published as stable. I'm more interested in all these things fixed to a ported from a system that I don't even use to play. I'm sorry, but this is my opinion with all my respect to your efforts. Thank you.

  45. In your next post, could you please let us know when you project the Beta will be completed so new/updated mods can be uploaded to the Steam Workshop again? Thank you.

  46. @Salamanderrake: On behalf of the linux community I'd like to ask you to shut up.

    I'd be glad to see the release soon! Thank you for porting! <3

  47. What an interesting insight, although I am no developer I could understand most of it.

    Thanks for porting the game to Linux, can't wait to play this natively.

More Posts

Code Rot

April 17, 2022
1 2 3 47
Back to devlog
Back to devlog
© Copyright 2021 Shining Rock Software
Website Design & Branding by Carrboro Creative
menu-circlecross-circle linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram