by Ferry “Wyz” Timmers, February 2012
Version 1.0.0
Abstract
This document will feature a plan to push AGS development forward. First it discusses the current state of AGS and its development. There are a number of things that needs to be improved like the website and issues with the forums. Also it states what the community can contribute and what potential developers should expect.
The plan it self will span eleven phases, each describing a clear goal. It will describe a method of organizing the future AGS developers, how to manage the development resources and development itself. It will span the complete life cycle of the 3.x branch.
In the last part of the document there will be views on several topic that have been debated in the AGS community like: the license, ScummVM integration and encryption.
Table of contents
Moving the old branch to ScummVM
Well I don’t know if it will be up to me but yet I feel it might be useful anyway to let everyone know how I would proceed AGS development. We are at the point now that CJ clearly indicated he can (or wish) no longer be actively involved in AGS development. Because it would be a waste of a very fine tool if development stopped completely and letting it fall into obscurity slowly, we (the community) need to get involved. There have been numerous discussions and also a few branches of the existing source; it is time to bundle our efforts and create some sort of pipeline so we can actually push things forward. In this document I will give my take on it.
First let me summarise where we are at right now. A large part of the AGS source has been made public under the Artistic License 2.0[1]. This includes the current engine branch (3.x) and the editor. What remains unreleased (Feb 2012) are the 2.x engine branch and the Native subsystem from the editor. Native is what compiles scripts and wraps games into a single file. It has not been released because it needs the non-free version of VisualStudio Express to compile[2]. It might be released in the future.
In my opinion the current state of the AGS code base is not subjectable to decentralised development with lots of different developers working on it because the code is monolithic and has loads of history. It needs to be refactored, split up in modules and be documented.
There are as I speak a few branches formed that are mostly ports (like PSP and android[3]) or branches that address of popular demands (custom resolutions[4]). There already has been a somewhat joined effort to implement a few feature requests for the editor.
During discussions on the boards I realized AGS is more then the program itself; also elements like the forums and website are not maintained without the presence of CJ. Currently some information on the website is obsolete, links are broken and some facts stated might simply be wrong. The forums might also have a few issues unaddressed (the PM system is buggy, the calender has not been working for a long time and it suffers some downtime every now and then).
All in all it is not in a bad shape, but it needs work!
Well this is how I would like things to go: First a small group of AGS veterans will join an effort to get issues that affect the community solved. I think top priority is for someone to gain full access to the website and forums. The website needs to be updated where necessary, the forums will need new boards (I’ll get to that) and I root for a better integration of the wiki.
There have been a new website in development for a while. I don’t know in what shape it is and who has access to it. I do think that the old site should be updated first instead of cracking out a new website first, since I gather that is going to take a while. All links need to be checked, corrected when sites are moved, removed when broken and possibly new ones need to be added. The same goes for all information: inaccurate and infactial information needs to be corrected, obsolete information needs to be removed and new information needs to be added.
The AGS wiki is riddled with useful information and I’ve used it many times, yet I get the idea it is not used very widely: what a waste! I think it should be integrated into the existing website in a way you can’t avoid it. The information on it needs to be updated but this should happen automatically since that is whole point of a wiki. All it needs is loads of visitors that feel welcome enough to also add to it. Maybe we should integrate it also with the forums, so you can use a single account?
Update:CJ has let us know that the new website is 95% done, it might be worth the wait after all.
When we’re starting to work on the AGS code base you can escape it: we NEED a new board for it. If you’d ask me we need two: One for developers working on the actual code, one for users with questions about the code base, and possible discussions about certain features. Not feature requests themselves because I have a different idea for that, I’ll get to that in a minute.
Some of the moderators has been long gone or possibly their existence all to gather has been disputed. Well actually I don’t mind, the forums are moderated very well if you ask me: tolerant when needed, strict when needed and always fair. But we will need moderators that can do more extensive things like adding new ones, or new boards. Also the new boards will need moderators I guess. (I don’t feel like it)
Also we need to increase the security for registering users since spam-bots are getting in: Capchas I guess.
Well yes, I feel the need to state a few things. The community is what AGS made successful. I mean, CJ did a hell of a job on the program but if no-one used it it would have been all in vain. Therefore whoever is going to work on AGS related topics should know the community is king. Whenever someone suggest something they should not just wave it away but listen to it and take it seriously. I can’t stress this enough. This is the reason why I trust the AGS vets, they’ve seen the community develop and that’s why they’ll even treat blatant newcomers with respect. Do we need to? Yes we need to, because we’ve all been blatant newcomers at some point in our lives, and if we were denied access everywhere back then we would probably never have been where we are now.
Does that mean we should tolerate every profanity people throw at us? No of course not, the community already deals with it and that is just fine. Actually most people who did so were trolls anyway, and we did not let them troll us so much, yay AGS!
What to expect from the community? Not so much but for this project to work we will need their full support. Cheer for the developers so they’ll stay motivated, but also nag them when they are off track as to keep them focused. I mean, generally this is done by the lead programmer (as I will get to) but sometimes there might be a gap between what the programmers imagine is good for the users and what actually is good for the users. They will never know until someone tells them.
On the other hand being supportive means not getting worked up about things that are not all that important. What I know from previous experiences is people reading about some plug-in or application and getting all enthusiastic about it and demanding it to be added to AGS without really thinking it through or even without being capable of seeing the big picture. Please don’t start needless discussions about it, simply state to the lead: “hey have a look at this”, something the lead should take really serious and respond to in a respectful way. Being supportive in that case would be trusting the developers when they decide it would not be a good idea to include it.
Again it is all about the big picture, and the developers need to poll what the community wants regularly and the community needs to make feature requests, bug listings and give feedback to the developers regularly. That’s how it works if you ask me.
We need a group of interested C++ programmers that also have lots of experience using AGS. I know such group already exists, and some of them are already working on things. The biggest problem with programmers is that they often don’t have a lot of free time, something we should take in account. It does not have to be a showstopper, we’ll simply have to be creative and split up work in an ad hoc fashion. There will be need for a set of standards so the developers can produce uniform code.
With the project I mean the further development of the AGS engine and editor. At this point there should already be a group of people that make the wishes of the community clear and have a sense of direction to present to the future developers. Then someone should come forward to coordinate the actual development (forth called the lead (programmer)). In this section I will describe what I would do if I ever took on this position and how I would see the involvement of everyone else. The project will be made up of different phases, each having a clear target so that we have a clear goal to work towards.
New boards will be created on the forums. After that has done a stickied thread will be made with a description of the project. Any goals and information will be made public to anyone as this is an open source project. There will also be an open invitation for any programmer (or possible other trades we might need) interested in joining the project. It will be very clear from the beginning what we would expect from them, and what not. The lead will have conversations with the interested to see what there fortes are and see what they would like to do, and also poll to check motivation and free time. But it is more like a survey then an interview really.
There will be a dedicated website made just for everything related to development. This website will feature the following key components:
All users of the software can submit a feature request on the site. One of the developers or the lead will have to process them[7]. After that they will be put on a list. This list will be visible in its entire on the site and the features will be ordered by popularity. Everyone can up or down rate feature requests. The features in the top section of the list will be the most likely to be implemented by the developers, although they will still have the final call. It is more like the list is there as to make the wishes of the community omnipresent. Yet it is still up to the developers to select what they think is important. Some feature might not be so popular, in that case users simply have to convince other users (NOT start nagging at the developers) the feature is useful so they’ll up-vote it. On the other hand, developers are still responsible for selection so they might promote certain features their selves.
Sometimes bugs might also be put on the list, for instance bugs that are likely to take a lot of time to get fixed. It might be useful to give priority to certain new feature rather then have the bug fixed, but then again it might not be. The community knows!
Now it is time to from a group. Due to the lack of time of a lot of developers it might be a good idea to do this in a slightly different fashion then usual. There might be developers that can commit to certain tasks as soon as their agenda opens but are unable to commit to the project for an extended period. However they would still be a good addition to the project. Picture two groups of developers: an inner and an outer group.
The inner group contains developers that can be actively involved in the project, they are part of determining what needs to be done and are involved in decision making. They might be appointed certain areas of the engine as it might be their expertise or so that they can really understand that part well and document it.
The outer group contains the developers that come by and want to help out. Everyone is free to join this group (when they have some experience programming) and take on certain tasks. Developers from the inner group will guide them as to get them started and see if their work meets the requirements. People are also free to leave from this group, no questions asked. I take the swarm project as an example[8]. There we did not push people yet without the pressure people felt free to join and work and actually delivered.
Developers from the inner group are also free to join the inner group when they feel they are up to it. When the inner group also trusts this developer they will be added. The other way around, developers might choose to leave the inner group when they feel they can’t commit anymore.
In phase three the two groups will take their initial shape.
There is a lot of stuff that we need to decide on first and further in development even though less frequent there will still always be stuff we need to decide on. For the big decisions we can make a thread in the developer forum but it might still be a bit inconvenient. Therefore a better idea is a regular developers meeting where the inner group developers are ought to be present and the outer group developers are free to join. It will be a strictly organised meeting as so we get as much done as possible, people that just want to rant about things or enjoy to talk off topic can better stay away. There will be an agenda and minutes for each meeting that are publicly available. Initially the meeting might be once a week but when the necessity drops the frequency might be something like once a month.
Assuming that the developer site is up, the forums have boards and the developers have met it is time to start working on actual content. However before the first line of code is written there needs to be a document containing coding conventions and general project guidelines. This document will be formed with the help of the developers and the lead programmer. There needs to be a consensus but for the love of AGS don’t be a dick about it. You can never ever completely work in your own comfort zone when you are in a joint effort.
The coding style convention will state the use of case in variable names and functions, a general naming conventions, how code should be commented, which white spacing methods are used, the bracket format and more of this. It is quite arbitrary so just go with the flow, even though it will look silly at some points to you.
General guide lines are there to make sure efficiency and readability is uphold and also to discourage practises that are prone to cause bugs. We want to avoid hacks as to make the source code as friendly as possible, although for efficiency it might be sometimes necessary. In those cases it should be well documented.
After the convention and guideline document is completed it will be published on the forums and site.
The existing source code will be copied to the new repository. It is time to refactor the existing code. Don’t underestimate the amount of work this takes. We’ll spend weeks working on it without ever changing what the code does. The rest of the community will wonder when all the cool new features will be included, well, refactoring needs to been done first.
In this phase the existing code will be split up, documented and made consistent with the convention and guideline document made in the previous state. The result will be a number of well documented modules each with a header file. The fact that for each area of the engine there exists a separate module means we can test them separately. This will cut the number of parameters that have an effect to the outcome and thus making it easier to test. It will even cut the number of bugs that will go unnoticed.
In the refactoring we will also need to make the existing code abide to the coding style standard we have set. If this turns out to be a lot of work, note that there exist programs that can change existing code to various coding styles: I’ve used them to despaghettify some code in the past.
The naming conventions are not only for clarity in the code itself, but also to aide the generation of documentation. Along with specially annotated comments programs like Doxygen can generate documentation directly from the source code. This will help new developers greatly in their understanding of the code. Of course this in-place documentation needs to be added first. In order to do so we first must understand what the code does. The lead with the developers should have a look at the source and if they find areas they do not understand they should ask CJ.
During refactoring we might stumble on code that needs to be replace all together because it might not be released, or it is archaic, or it depends on libraries then can best be replaced immediately other then further in the process. This will be taken care of in an ad hoc fashion by the lead together with the developers.
After phase six we will have a clean, documented and easy to understand code base that is conform the conventions we set. This will allow us to change the actual functionality.
We assume the engine’s code base has been refactored at this point. Refactoring the editor’s might still be in progress.
During the refactoring process the community is likely to have submitted feature requests and bug reports already. It is now time to implement/fix them. One of the features or bugs is selected and assigned to a free developer or group of developers. This is done till every developer has something to do.
The role of the lead programmer in this phase (and future phases presumably) is to coordinate the tasks. This means the lead will help developers selecting tasks and is there to make sure each developer understands the task they will be doing. When tasks are done the lead will do check if the code is up to the standards and change it when it is not. Again I took inspiration from the swarm project here: Baron would do this with the submitted art work so it would all fit together nicely.
Also the lead will test if the code works by doing various representative tests as to complete coverage. When the code is ought to be okay the lead will merge it with the main (development) branch. The lead will see if there are conflicts and either correct them (when small) or ask help from developers to fix them (when complicated).
The development branch will be available in the form of snap shots and is built regularly (once a week or so) so the community can test it. This version however is never stable and should not be used in production.
After a number of milestone features have been implemented the newest (tested) version will be released to the public. Any bugs found by the users testing the snapshot versions should first have been fixed. It will be packed with installers and without, in the various versions we are used to on the download page[9] of AGS. It might however be useful to release engine and editor separately, yet we need to be sure when we do this the engine and editor always match and do not get conflicts. Previous version will always be kept on the download page so people can downgrade whenever a version is not stable enough.
Each release will also be announced on the boards complete with a change log.
We will repeat phase seven and eight a few times and then it is time to evaluate the project so far. At this point the editor should also be completely refactored and worked on.
We will evaluate the efficiency of the lead and the work done both by the inner and outer developer groups. The quality and quantity of the produced work is accounted. At this point we also make sure the community is still involved. We ask the community what they think of the current progress. Also keeping in mind the community often wants things to happen faster then in reality is reasonable. Still they must feel welcome to give input and feedback to the project.
In phase nine we will evaluate the current process and document this.
After the evaluation we might find some useful changes to the project. Maybe we can optimize the process a bit. Or we might have to change the document formed in phase five because we ended up doing things differently. Now is the time to refine our methods and streamline the project. There is not really much else we can say about it right now; the future will tell us.
Phase seven up to including the previous phase will be repeated a few times. We assume things are getting smoothly now and AGS development has become a routine. This the the moment to start thinking about AGS 4.0. This version is allowed to break compatibility so we can take out any legacy part that has held us back. What AGS 4.0 is supposed to look like is beyond the scope of this document though. The work done for the AGS 3.x branch will continue as usual but some (dedicated) developers might now start working on AGS 4.0 instead. This will continue until the first stable version of AGS 4.0 is released. At this point the AGS 3.x branch will still be serviced and bugs when found will be removed but some new feature might not be implemented anymore. We will do whatever it takes to make the transition from 3.x to 4.0 as smooth as possible. There needs to be a tool that can convert AGS 3.x projects to AGS 4.0 to its best extent. Now slowly AGS 4.0 will gain the main focus. This will be the end of this plan. AGS 3.x might still be serviced for a long time, what happens to it ultimately, who knows. I’m sure it will all work out, maybe not entirely as we wished but good in the end.
There have been tons of discussions on the boards about various topics and what direction AGS should take. I like to address a few of them and give my view on them. I know some are much debated and I have a very clear opinion on them yet in this document I’ll think from the perspective of the user since fan boy-isms and arrogance will not get us anywhere.
AGS has been released under the Artistic License 2.0. This makes a distinction between two kinds of versions: standard versions and the modified versions. A version may be called standard when the original author (CJ in this case) approves of this version. Some things are implied: versions that derived of the original release but have bug fixes or are ported to other platforms are still standard versions.
All other versions are modified versions and they can only be distributed when a few conditions are met. Most notably, it must explicitly state it’s a modified version and include a list of changes. Further more it must not hinder the original version by either making it look unrelated or releasing the source code under the same license to either the original author (CJ) or the public. The modified version may also be released under a different licence (like GPL) as long as that license retains the free nature of the original license.
It might not be very clear to the user’s point of view what all this means, and even developers can have a hard time understanding it. It comes down to this: the code when worked on by developers from the AGS community will be noted as ‘the original’ (as long as CJ approves of our work). When developers apart from the community effort release altered code it needs to be either renamed to something that does not relate to the AGS community or make it available at the same extend to the community (or CJ). I guess the reason why this is in place is to guarantee that the most recent engine version is kept close to the AGS community. People might argue whether this license is too liberal or should also be copyleft[10], but what I like to point out is that this is the license it has been released under.
Lets look at the average AGS user: they don’t really care about the source code and what license it has; they simply want to use the program and do so without paying for it. The current license will disallow people to create a commercial version of AGS (the software) without the consent of CJ. Some want to use AGS for commercial games. They also might want to edit the engine for their own needs (add plug-ins right into the executable). With the current license they can and also do it without the necessity of releasing their source code[11]. Let me note that need to release source code for commercial game developers can be a threshold as to use AGS as much as a trifle it may seem. This license gives commercial developers the room to use AGS as freely as they want.
Some have proposed it would be a good idea to merge AGS with ScummVM as it could benefit from its portability and stability as in the future. There have been sounds the ScummVM developers might not take on a project that is still in motion[12]: when AGS is still worked on actively and new features are added. It might slow down AGS development like the addition of new features. On the other hand users have been looking for ways to create games for ScummVM and this might give them a way. There are advantages and disadvantages to this; both ways not necessarily mutual exclusive. I can picture ways in which users from both communities can benefit form AGS being open source. I picture two scenarios:
There are a lot of old AGS games that will most likely not run on the latest systems. ScummVM’s goal has always been enabling people to play old games on whatever system they have. I see a connection there! The AGS community can stop worrying about being backwards compatible to even the oldest games, and people that want to play these games can do it on an increased number of systems. If I’m not misinformed the old AGS games will work with the newest engine (give take a few minor quirks). If that‘s true we can hand our version after we are done refactoring to the ScummVM development team and have them support all AGS games up to that point. They can work out the quirks that the really old games have by patching the engine and give support when games break. As of that point the AGS team does not have to worry about breaking the really old games anymore and can improve some long standing issues.
Another way of integrating AGS into ScummVM is by treating ScummVM as a platform and include it into the platform specific builds. That we AGS will and up as a program library the ScummVM developers only need to include. Each subsequent build will result in a version that can also be run in ScummVM and on whatever platform that supports. The question is whether all this is really useful because we will port AGS to different platforms anyways. It might be useful when Allegro does not support certain platforms and have it replaced by OSystem (ScummVM’s subsystem). This might be a useful way to expand AGS’ portability to systems not easily supported.
Both options will have the added bonus of allowing ScummVM users to create games for it using AGS.
First of all, without doubt, the engine should be ported to as many platforms as possible (including HTML5 if you ask me). The following bit is about porting the editor.
Apparently there are people out there that like AGS so much they are willing to have a separate partition with windows just so they can run the editor. I guess that says enough, the motivation definitely is there. Yet, the editor is new and pretty much in good shape. It uses Microsoft’s .NET framework and even though that is pretty much linked with Windows there is a library that can port it: Mono[13]. I know some developers have tried this in the past but because the Native subsystem has not yet been released then were unable to port it completely[14]. We may need to wait for that last bit to be released or else replace it. It might not be our top priority right now but I think it’s in the interest of everyone if the editor is available to at least every popular desktop system out there. This is something we should take in account when we’re done refactoring the engine.
One of the biggest issues CJ had when releasing the source code is that it would enable people to decompile games and have access to the game resources like sprites, backgrounds and the music. I’m not really afraid of that myself but there might be users out there that don’t really feel comfortable if people could break open their games. It would be a bit insensitive to tell them to get over it and just make everything open. Don’t get me wrong, as a developer that is exactly what I would like to do, but when I look from the user’s point of view I can see how this will create issues. I will not go in depth about how AGS handles resources right now but I will say that with the source code released everyone can create a decoder. Actually, and this is really important to note for everyone that is not into computers, everything the game has access to can also be accessed by someone else. They don’t even need to have the source code, when a motivated cracker starts working on it, it can be broken in a matter of days. The reason why no-one ever did this is because they did not see the value of it. Frankly this is the best solution; if you try to encrypt things really well all you are accomplishing is attracting code crackers.
Still, we don’t want to make it too easy. All we need is a simple encryption algorithm (there are tons of them out there) to obscure the package files a bit so they can’t be read by programmers.
Still anyone with the source code could write a decoder so this might not be enough for some users. I think this can be solved with a separate plugin that can encrypt it in a secret way, with no source code available of it. We don’t have to supply this plugin ourselves, but we should enable plugins to have control over package loading.
I’m suppressing a long and fierce rant about DRM here, but let me say this to all future developers: any time spend working on DRM instead of making the actual game better is utterly wasted! Everything can be cracked, and instead of fearing your potential customers lets give them a reason to love you. Then they will not even think about messing with it; and lets say they do rip off sprites and music from your game, is that not the ultimate compliment?
AGS uses quite a few libs and some of them are archaic. The main lib it uses is Allegro 4.2.2 and although it has stand the time it might be needed to upgrade it to a modern lib at some point in development. I guess the most likely two libs are Allegro5 and SDL. Which one we should pick is beyond the scope of this document, but there is something I like to state regarding picking libraries in general.
Any lib used for the engine or editor should be compatible with the license AGS currently has been released in. Bare in mind that a license and being licenced are two different things. The license can be found in the source code or redistribution of the library and specifies what can and can’t be done with it. Most importantly the license of the lib should allow it to be redistributed as part of AGS freely.
Because our goal is to port AGS to as many platforms as possible there will be high demands for portability for the libraries we use. Sometimes we might replace one on specific platforms but this practise is cumbersome generally so the best way is to choose libs that are already available on lots of platforms.
The libs must also stand the time, so preferably the lib is still actively developed or has indication that support will be there for a while.
The libraries are used to add functionality to to the code base. It’s important that any lib completely covers a certain function but also not much more then that. We should steer way from bulky libraries with loads of dependencies.
I hope this document has been informative and cleared up any uncertainties about the current state of AGS development. The plan is meant as a source for inspiration and is not meant to dictate how I think things should go. The order of things mentioned might differ or some things might end up differently, it does not matter, whatever gets it done; however at moments of uncertainty you will have this document to help. I hope AGS will still have many successful years to go for both the tool and the community. It hope it will take full advantage of the new impulse of this generation of adventure games and be a versatile and modern tool for creating the adventure games of tomorrow. I thank CJ for everything he did; making this a fun to use tool that sprung thousands of adventure games; realizing an amazing community to be part of. I already want to thank the future developers that will continue to make AGS a great tool for present users and those to come.
Now all that is left for me is to say: Good luck!
[7] This is mainly so we don’t get duplicate feature requests and make the description clearer when needed.
[10] Copyleft is the counterpart of Copyright and states that the work and all derivatives should remain free to use, modify and share.
[11] Given they don’t hinder the original AGS version with their version.
[12] I can’t find the original topic; also it might not have been involved ScummVM developers who said it.