189 replies to this topic

[Yves]

I decided to open a thread about this topic because I think we need to collect the available information somewhere.
There are different people who already worked on improvements of Mac support, but we're not yet where we want to be and there are still some missing bits before we can put everything together.
This involves different aspects and we will only be successful if we cover all of them or at least the important ones.

First let's give an overview for those who didn't look at our build-system in detail:

The basics:

update-workspaces.sh / update-workspaces.bat:
The sh-script on Linux/*nix/OSX runs the compilation of some external dependencies and also premake.
On Windows the batchfile only executes premake because most of the dependencies are precompiled.

Premake:
Premake creates our workspaces (also called solutions or projects) for the different platforms.
Currently we support Makefiles, VisualStudio (2005, 2008, 2010), Xcode3, Codeblocks.

Workspaces:
The workspaces are used for building the code on different platform and also for development with different programs (IDE's).

Deployment:
For (alpha-)releases we make a special deployment for various reasons. The main reason is that we want to make it easier and more convenient for the user.
Everything is precompiled and distributed in appropriate containers for the different operating systems and platforms (e.g. .deb files for debian based linux systems and .exe files for windows etc.).
I don't know who exactly does that for which system.



OK now our current status:

XCode support:
Currently we have support for XCode3 but It's not particularly useful as far as I can tell and XCode 4 is already out quite a while.
The most current version of premake (4.4beta) supports XCode4, but we don't yet know how well.

Premake:
We are using premake4.3 with quite a lot of customizations. Upgrading is a lot easier than from premake 3.x to 4.3, but I requires some efforts to reimplement the customizations that are still needed and to test everything.

Here's a list I made in a few minutes... but It's probably not complete.

• General: Customizations for using nasm assembler. It's not needed anymore, but the customizations are still in our code...
  
• General: PCHHeader as „path“ instead of „string“ in api.lua
  
• Windows: trimmprefix for removing the „source“-prefix
  
• Makefiles: Fix for parallel builds
  
• Makefiles: Test generation
  
• Linux+Makefiles, Codeblocks: --start-group and –end-group to work around circular referencing
  
• Codeblocks: Fix Codeblocks not relinking after static libraries change (changeset 10814)
  
• Workaround some linking problems for ActorEditor, ArchiveViewer and FileConverter (changeset 10749)
  
• ... maybe forgot some

Build-process on the Mac:
The build-process used during development generally works. There's one little problem because both the Makefiles and XCode generate a .app bundle that does not work.
You have to copy the binary inside the app-bundle to /binaries/system before it can be executed properly.

Deployment process on the Mac:
We currently don't have one. We just link to the development build-instructions and sometimes there are people on the forums who create a .app bundle.

Tickets:
There are currently at least two open tickets about this topic:
#947
1073

The patches in both tickets don't offer completed solutions yet.



Next steps / open questions

XCode support:
I think we must still stick to premake and should not use manually created XCode projects.
Changing one workspace manually is error-prone and time-consuming.

I've created a quick-and-dirty integration of the premake4.4beta and sent it to historic_bruno because he has XCode4 and knows what to check. He will test if it's worth the efforts of upgrading.

Build-process on the Mac:
Changing the kind of our WindowedApp-Projects to "ConsoleApp" in premake4.lua causes XCode and the Makefiles to generated simple binaries instead of the broken .app bundles on OSX. That's a quick fix and we just need to test if we can change this settings on all environments or if we should do it just for the Mac.

Deployment process on the Mac:
I've discussed this with historic_bruno and we both think that we should seperate the deployment from the build-system.
It's something that should come after everything is build and the build-process should stay as it is for development.
I've red some documentation about .app bundles and started creating a shell-script to but one together.

We aren't sure if we should put everything into a single .app bundle or if we should use more advanced packaging tools.
I think we have to do the .app bundle anyway, because those packages seem to be just some kind of container. If there are no important downsides of "simple" .app bundles, we should avoid the additional complexity added by the packager.
One important question that could make a decision obsolete is where we want to put the different resources like the data-directory.
We know about two potentially suitable locations:
1. The Resources directory of the .app bundle
2. Library/Application Support/0AD/

I tend to the first because of two reasons:
1. According to the Bundle Programming Guide a bundle should contain all resources required to run propperly.
2. Having anything outside the bundle forces us to use the packager.


Code for resource-paths:
We definitely have to make some changes to source/ps/GameSetup/Paths.cpp about the location where it tries to find the data-directory and other directories.
Echelon9 proposed something about that in his patch. I think we should do something like that for the default behaviour on the Mac (either for Library/Application Support/0AD/ or the bundle's Resources-folder) and in addition add some possibilities to override the default behaviour by command-line parameters or environment variables.

Prebuilt libraries:
I don't know which libraries need to be prebuilt yet and it will require more work and more testing.
One important thing is that the libraries should be compatible for different versions of OSX (which?) and 32 and 64 bit.
The Bundle Programming Guide states

Quote

A bundle can support multiple chip architectures (PowerPC, Intel) and different address space requirements (32-bit/64-bit). It can also support the inclusion of specialized executables (for example, libraries optimized for a particular set of vector instructions).

I'm not sure if that's something that can be done with the bundles or if it just means using universal binaries.



What do you think?
Probably that's a topic we could also discuss on our team-meeting.

[Geek377]

Here is what I think: this would make getting the game a heck lot easier and me a whole lot happier! Also, if it makes your job easier, drop PPC support. Seeing the market trends, it is still used but very few people, but not a significant amount.

[historic_bruno]

Yves, on 06 January 2012 - 03:19 AM, said:

Everything is precompiled and distributed in appropriate containers for the different operating systems and platforms (e.g. .deb files for debian based linux systems and .exe files for windows etc.).
I don't know who exactly does that for which system.

I believe that's Philip who does the Windows release packaging, uploads the source and data archives for Linux, and probably does some other behind the scenes stuff I'm sure there are a number of maintainers for the various Linux packages.

Quote

Currently we have support for XCode3 but It's not particularly useful as far as I can tell

It works well enough, the biggest issue is having so many Xcode projects and the pyrogenesis/Collada/AtlasUI/ActorViewer projects are not linked together. It would be nice if there was only one project that linked the parts together - as on Linux and Windows.

Quote

Deployment process on the Mac:
I've discussed this with historic_bruno and we both think that we should seperate the deployment from the build-system.
It's something that should come after everything is build and the build-process should stay as it is for development.
I've red some documentation about .app bundles and started creating a shell-script to but one together.

Also it's common to pack the app bundle into a dmg (disk image) for distribution.

Quote

We aren't sure if we should put everything into a single .app bundle or if we should use more advanced packaging tools.

It's not necessarily either/or, take Xcode as an example, you download it from the App Store, getting an "Install Xcode" app bundle which is run and installs Xcode permanently on the system. I don't believe all the files installed by Xcode are in a single bundle

Quote

Prebuilt libraries:
I don't know which libraries need to be prebuilt yet and it will require more work and more testing.

Most of the dependencies in our build instructions, or use the list of suggested ports. I'd like to drop Mac Ports as a requirement for building the game on OS X, because it would be nice to not be dependent on their maintaining port files or having to worry about what versions they provide or being limited by the build configurations they use.

Quote

One important thing is that the libraries should be compatible for different versions of OSX (which?) and 32 and 64 bit.
The Bundle Programming Guide states
I'm not sure if that's something that can be done with the bundles or if it just means using universal binaries.

10.5 is the cutoff I've had in my head for a while, most people here on the forums who are following Mac development seem to have either 10.6 or 10.7. I'd be fine with just distributing a 64-bit build of the game and all libraries, at least for a start.

Geek377, on 06 January 2012 - 03:38 AM, said:

Also, if it makes your job easier, drop PPC support.

PPC support is not even on the table, unless a community enthusiast wants to develop and test that When we say "universal" we are only meaning 32/64-bit Intel.

[Echelon9]

Bringing these topics together is going to help.
In my short time in the community, it seems the Mac build infrastructure is leaping ahead with some pace. Shouldn't be too long until the remaining issues are resolved and it starts spining off the benefits it offers towards improving performance and feature equivalence on OS X.

[wraitii]

This seems to be basically what's needed, from a fast-read... As for deployment, on Mac, the apps usually require only to download them and play... Things more complicated, like xCode, which needs dependencies and many many things usually install in a "package" that puts everything where needed.

Linking to resource paths must be done on both the executable and the libraries. Given the variety of libraries used by O&d, I'm quite sure many are already in OSX, but I'm not sure exactly which, so we'll have to troubleshoot this.

And yeah, dropping anything earlier than 10.5 is likely not a problem, mac OS release are usually well followed ( 10.6 was in particular since it was 20$ ).

[wraitii]

Okay, so to further develop your "open questions".

The biggest downside of an app bundle can be weight. However, this is pretty much irrelevant for 0 A.D. since the libraries are basically lightweight. it's however much more convenient for the rest since it allows a user to simply put the app bundle where he wants and run it, nothing else needed, like most apps on OSX.

As for where to put what, Data and things like that likely ought to be put in "AppPath/Contents/Resources". Libraries should be put in "AppPath/Contents/Frameworks".


What you should actually put in "~/Library/Application Support/0AD" is more likely the hidden "config" folder that currently lies in "~". It should also not be hidden if put there.

[historic_bruno]

wraitii, on 06 January 2012 - 05:04 PM, said:

As for where to put what, Data and things like that likely ought to be put in "AppPath/Contents/Resources". Libraries should be put in "AppPath/Contents/Frameworks".

What you should actually put in "~/Library/Application Support/0AD" is more likely the hidden "config" folder that currently lies in "~". It should also not be hidden if put there.

What about cache, logs, screenshots, and saved games? I almost feel like those too should go in the user's home directory, but then it depends on how we think of an app bundle, is it a.) 100% self-contained, writable and valid for one user only, or b.) more like our Linux packages that are essentially read-only and files created by the game end up in the user's home directory? This question may be best answered by observing how other OS X games work.

[Echelon9]

Mac's are like b.)

Edited by Echelon9, 07 January 2012 - 01:14 AM.

[wraitii]

To tell you what AOE III does:
everything "static" is in the bundle. Data and dependencies.
Everything else is in "~/Documents/Age of Empires III". This includes AI files, Savegames, scenarios, homecities.

I think it's basically what you want for 0 A.D.

[Yves]

I've tried to make a summary of what's been said about the paths and what I've found out.
Most of it is how it is at the moment. I think we should put the readonly stuff into the app bundle as shown in the column "Path MAC".
I'm wondering if the paths for Userconfig and Logs are right.
Why don't we put Userconfig into ~./config/0ad directly and change Logs to ~/.local/share/0ad/logs?

Btw. Do you have a better idea for posting tables?
The bb-codes seem to be a bit broken and doing the search-and-replacing on the csv is a bit cumbersome.
I don't like attaching office documents.

Name	Readonly	Description	Path MAC	Determined by/Set where?
Game data root	Y	Data-Directory for static game data	[Bundle]/Contents/Resources/data	m_rdata in Paths.cpp
User data root	N	Root-Directory for data the user creates with the game (savegames, screenshots etc...)	~/.local/share/0ad	m_Data in Paths.cpp
savegame directory	N	Directory where savegames are saved to and loaded from	~/.local/share/0ad/saves	As [User data root]/saves in GameSetup.cpp InitVFS
screenshot directory	N	Directory where screnshots are saved by the game	~/.local/share/0ad/screenshots	AS [User data root]/screenshots in GameSetup.cpp InitVFS
Default config	Y	Defaullt config-file which is readonly	[Bundle]/Contents/Resources/data/config	As [Game data root]/config in GameSetup.cpp IniVFS
Userconfig root	N	Currently unclear... why is [Logs] here and not in [User data root]?	~/.config/0ad	As xdgConfig in Paths.cpp
Userconfig	N	Contains user specific configuration settings	~/.config/0ad/config	m_config = [Userconfig root]/config in Paths.cpp
Logs	N	Contains logs created by the game	~/.config/0ad/logs	m_logs = [Userconfig root]/logs in Paths.cpp
Cache	N	Cache directory	~/.cache/0ad	m_cache
Mods	Y	Directory where „modable“ data is places,,, not quite clear yet how this should work	???	???

[wraitii]

To be perfectly in the "Mac OS X" spirit, you should likely put anything that's "~/.config/Oad" in "~/Documents/Oad/config" and likewise, cache and "local" would be in this "0ad" folder... Hiding stuffs in the root ~ folder is very un-OSX-y. Alternately, you could use "~/Library/Application Support/Oad", if you want to make it more "hidden".

[Ykkrosh]

Yves, on 06 January 2012 - 03:19 AM, said:

Deployment:
For (alpha-)releases we make a special deployment for various reasons. The main reason is that we want to make it easier and more convenient for the user.
Everything is precompiled and distributed in appropriate containers for the different operating systems and platforms (e.g. .deb files for debian based linux systems and .exe files for windows etc.).
I don't know who exactly does that for which system.

I run the steps that are mostly documented here, except not the PPA packages (ricotz does that in some unknown way) or Gentoo (random people are updating the ebuilds now). Other distros are handled by random people in unknown ways.

Quote

I've discussed this with historic_bruno and we both think that we should seperate the deployment from the build-system.
It's something that should come after everything is build and the build-process should stay as it is for development.
I've red some documentation about .app bundles and started creating a shell-script to but one together.

That makes sense to me - packaging is a separate problem from building, and developers want to build and run the game from SVN without packaging at all, so the problems should be solved separately.

(Also, I think building and packaging should preferably be separate from the problem of editing code in an IDE. Premake-generated Makefiles and custom scripts should be sufficient for producing packages; Xcode support should only be needed if developers like using it to develop code integratedly, it shouldn't be a necessary part of the packaging solution, since it adds more layers of complexity.)

Quote

We definitely have to make some changes to source/ps/GameSetup/Paths.cpp about the location where it tries to find the data-directory and other directories.
Echelon9 proposed something about that in his patch. I think we should do something like that for the default behaviour on the Mac (either for Library/Application Support/0AD/ or the bundle's Resources-folder) and in addition add some possibilities to override the default behaviour by command-line parameters or environment variables.

The default when building and running from SVN should be the same as on other platforms if possible, i.e. use the data files directly from SVN. On Linux we set some preprocessor definitions when specially building the game for packaging (so it looks in /usr/share/0ad/ or wherever) - we could do the same on OS X, or I assume it could easily detect at runtime whether it's running from SVN or an app bundle.

Quote

One important thing is that the libraries should be compatible for different versions of OSX (which?) and 32 and 64 bit.

I think 10.5 is old enough and different enough and rare enough that it's not worth supporting, but 10.6 probably is worthwhile. I think 32-bit Macs probably aren't worth supporting either (since I guess there's not many that came after PPC, and before Core 2, and have a decent enough GPU to run the game, and are still in active use), so 64-bit only would be sufficient (and much easier than trying to get universal binaries to work).

[Yves]

Ykkrosh, on 07 January 2012 - 05:29 PM, said:

The default when building and running from SVN should be the same as on other platforms if possible, i.e. use the data files directly from SVN. On Linux we set some preprocessor definitions when specially building the game for packaging (so it looks in /usr/share/0ad/ or wherever) - we could do the same on OS X, or I assume it could easily detect at runtime whether it's running from SVN or an app bundle.

Currently the design focuses mainly on the development build and allows some limited room(preprocessor directive) for adjusting the deployment.
For the future it's probably better to shift this focus a bit and concentrate on a more flexible design which does some runtime checking for finding the directories and maybe also allows customizations after buildtime... so I would tend to the second approach.
I'm working on a patch how this could work exactly, but I'll need some more time until it is ready.

[janwas]

Quote

PPC support is not even on the table, unless a community enthusiast wants to develop and test that When we say "universal" we are only meaning 32/64-bit Intel.

Yes, just to expand on this a bit - there would probably be some additional pain due to byte alignment and endianess (PPC is stricter/different on both counts). That's why we have always assumed only x86/x64 would be supported.

[historic_bruno]

Yves, on 07 January 2012 - 03:08 PM, said:

Why don't we put Userconfig into ~./config/0ad directly and change Logs to ~/.local/share/0ad/logs?

I don't think this makes sense on OS X. Technically we should use the Preferences API instead of our own config management, but that sounds like a pain to special case for OS X, so I would prefer a standard visible directory. Part of the problem with ~/.foo is no one can find them, which mostly defeats the purpose I think cache should go in "~/Library/Application Support/0ad/cache" because that sounds nice and obscure, perhaps all other data could go in "~/Documents/0ad" - roughly equivalent to My Documents on Windows?

The mods path is a tricky question and not fully resolved even on Windows or Linux. Clearly it doesn't make sense for the user to have to copy new files into the app bundle, just as it hardly makes sense for people to copy mod files into "%localappdata%\0 A.D. alpha\binaries\data\mods\" on Windows. We should address this when we generally improve mod support.

[Yves]

The information from today's meeting:
We are going to separate the prebuilt libraries from the game's code and data and keep an own directory for each platform.
To get the right precompiled libraries for their system, people will have to do a second svn checkout of the corresponding libraries.
The revision of each library will be stored in a file and update-workspaces.sh/update-workspaces.bat will check if the libraries are present and if the versions in the file match the libraries versions. This file will most likely be kept up-to-date manually.

This way we avoid unnecessary downloads of precompiled libraries for other systems and reduce download-durations for developers/users and bandwidth requirements on the server. We avoid problems related to outdated libraries with the revision-checking described above.

[historic_bruno]

As a reference, see the build process for Blender and their SVN structure, especially the lib/ directory. I've compiled Blender on Windows and it was a very smooth process, their Mac instructions look good, too

[Yves]

Bad news about XCode4 support:
Premake doesn't seem to really support XCode4. They have one file with a few dozen lines of code that puts all the projects into one workspace, but for everything else, they use the same code as for XCode3.
We either have to wait until they support it better or do it ourselves.

I think it's better to concentrate on the deployable bundle now and continue working on better XCode support once this is done.

[Sonarpulse]

could you guys just build it once with Macports and/or this? From my experience, most mac users won't show look into a game until they have a precompiled bundle. But once that happens they should appear in droves.

[Yves]

About four people (Echelon9, historic_bruno, Juicyfruit, Yves, ...?) are currently working on tasks related to the build and deployment environment, which is good in the first place. We should try to get as much benefit from this work as possible.

Unfortunately it seems a bit uncoordinated at the moment and I see the danger that in the end someone will have a usable solution without using the other peoples work.
That's bad for three reasons. First, it's a waste of resources and time and second, it's demotivating and third, It's likely that this solution wouldn't be an optimal one because good inputs won't be considered.

I've had a short chat with Echelon9 about this topic and suggested creating a big picture documentation first. Everyone could contribute to that documentation and once we agree on it, we can split it into several tasks and assign those tasks.
That way everyone can contribute and we will most likely have higher quality results earlier.

I've started creating such a documentation in the wiki: BuildAndDeploymentEnvironment
Please post your Inputs here or add them to the Wiki.
I'll continue working on that article because there are still some topics missing.