Dawn of Light

First of all, kudos to all of you hardworking members of the core development team. This project is simply astounding in size and complexity and best of all; it really works!

After taking a few days to sink my teeth into the project source code and try to understand a bit of it, it appears to me that the major bottleneck in all of it is the database consistency and synchronization.

I've written a couple of game back-ends myself (significantly smaller in comparison to DAoC) with more platform-independent-friendly/data-consistent development packages.

I was wondering if there might be any interest in the community (developers included) for a rewrite of the back-end with a language/platform that comes with a stock ORM (instead of custom written, which is very well done in DOL, I might add) and support for schema and data migrations that help mitigate the headache that comes from major database schema changes or huge data additions, which are inevitable with an ever-changing dedicated game back-end.

Additionally, some of these languages and platforms can support a wide array of web-facing technologies and applications which would allow one to convert the platform-specific executables to web facing environments (i.e. the config application and server launcher).

I've got a few ideas myself; first thing that comes to mind is python using Twisted Matrix as the broker for client connections, pure python for scripts and game logic, then Django for the ORM/Data Migration support and the added bonus of the quick ramp-up for a web front-end that would easily support server/data management.

I know there's a ridiculous amount of support for Ruby (on Rails) out there and would be surprised if an Async SocketIO package wasn't out there to rival it.

Thoughts, comments? Has anyone thought about/tried this in the past? If so, why did it stop? What roadblocks prevented things from progressing?

Alternatively if this has never been attempted, why?

Hi Xanderal,

The first attempt was to move data layer to NHibernate. This was planned for DOL 2.0 then aborted, not really due to ORM, but NHibernate is all in all a complex choice.

My goal was then to set a LINQ-based provider (thourgh LinqToSql, DBLinQ) and thus abstract our mini ORM and make it support MSSQL, SQLite, PostgreSQL effortless.

Recently i had to work with MicroORM, and i was interested in pushing PetaPoco or Dapper. Entity Framework is voluntary not evaluted deeply, because it lacks open-source providers.

To answer clearly, the DAL is actually working rather correctly, and we don't really have in the roadmap shortcoming plan to support anything else than MySQL.

Else, moving a part of our code to Python is not something we can afford, but many thanks for your help

Changing the DB access layer is a huge task and not really necessary imho since it does a good job. I don't see much advantage to change it at this state.

Major schema update data migrations must be handcoded and thats ok as you have to move and transform data. I don't see that you can avoid and automate that. You would at least need a set of sql commands to perform the data transform/movement. Support for adding new columns and deprecating unused columns is already builtin and works out of the box.

@Graveen

Thanks for the detailed information. This is exactly the kind of stuff I was trying to figure out.

@Blue

There is quite a lot of support out there for automated schema migrations with some of the more recent ORMs. One I know of specifically is South for Django, but if a language transfer is out of the question than it really doesn't do much good. You're correct, data migrations must be somewhat hard coded, but using the ORM to perform the migration is much easier and a lot cleaner than writing SQL by hand.

Also, I hope that I didn't offend -- the majority of the code I've poured over is fairly well written and, for the most part, easy to work with. Again; the Development Team has done a fantastic job with this product. I was just interested to find out if anybody would have been interested in making it a bit more flexible for deployment (I'm a long time linux user and a huge fan of DAoC, my first instinct was to try and figure out how to combine the two ).

So it's clear a back-end rewrite isn't necessary or interesting, but what about the management console and admin tools? Any interest in replicating that from a web front-end for those of us in the community that want to host our own shard? My ideal production environment would consist of the server running by itself on a windows box and the database running independently on a separate server. As I run all of my server instances through a hypervisor, I'd really like to be able to manage it from a web front-end instead of on the windows machine via RDP or VNC.

Also, I'd like to make clear that I'm not suggesting the development team plan efforts to do these things themselves. It was more of an offering to write these third party tools if there was interest from the Development Team and/or community in having them.

No offenses ! Indeed, some parts are correctly written, some are not, but we try to go forward, and have a better emulator

This is interesting. Notice we have a mini management tool we use for Storm (official DOL server) through IRC; also, players can use some commands directly our IRC channel (understand, it is not only admin commands).

All propositions are welcome. What could be really interesting is a dashboard monitoring server health and reporting it to a convenient interface (all is log based: errors, warnings for SQL requests, heavy load, possible detected cheats, ...).

Woosh

I, personally, am not interested in the typical management tools because most are, shall we say, hackable. No web interface to the DB server, no web interface to the game server. Just my opinion, of course.

I have beefed up our IRC bot to be able to do most things from IRC on both Storm and D2. It's not perfect, but it allows me to solve some issues without having to remote in.

Obviously security is a primary concern for these kinds of things . I've written a couple of web-based authentication systems myself (and used plenty more out-of-the-box), in the past, and I've found that when you use the right tools, you can be fairly unassailable against the common script kiddie (let's face it; I doubt someone who's skill level is above the script kiddie rank would really be trying to hack into these systems anyway).

Additionally, judging by how many active live users there are in the first place, most of them (I would imagine) aren't in the hosting/development business. They just want to play the game. Simple as that. I wouldn't, in general, be afraid of them interfering. Those who know a thing or two about the DOL setup probably aren't looking to sabotage anyone -- why would they? They can always run a shard of their own if they don't want to play by the rules.

As far as IRC-level admin tools are concerned, any initial design/architecture process for a web-interface like this would take these things into consideration. I wouldn't want to reinvent the wheel, as it were -- I would simply want to make the tools that may or may not already be available less threatening for newcomers that might want to run/manage their own private shards.

I appreciate the availability of things such as Storm and D2 (really anxious to get my hands on the Storm DB -- very complete!), but there's something about running my own shard that's really attractive. I want to experience that MMO server-admin role; invite my friends and others to come play on my own private DAoC server whenever I want. But I'd also like to be able to manage it easily, without a bunch of hoops, hard-to-find instructions, or resorting to hunting through code to figure out how to do what I want to do. Lastly; I'd like to share that availability with other people who might want to do the same.

I can only speak from Uthgard view: We manage the server from IRC and an internal GM interface which is web based. Live interaction with the server is made by using IRC commands by authenticated GM's. We can poll all kind of statistics from the server like memory usage, timer usage, CPU usage and so on and issue sophisticated commands to the running server. Reboots are controlled by this command interface too. Its rarely necessary to login into VNC.

The GM web interface is more for things like accessing logs or access to ban tables but I'm very strict and careful when allowing access to the database from the public. Its clear that you can't edit in the database itself from a webfrontend as the running server doesn't sync with these data changes automatical. I would more prefer an ssh like interface to the game server itself. Some tools could connect to this and have advanced interfaces for highspeed editing (not the IRC delay which we have to get used to). Imagine a world editor which connects live to the running game server and is able to move mobs, change spawn rates, drop rates and so on. Would be awesome.

Hello! I think we may be on to something here. Unless I'm wrong (which I very well may be), DoL doesn't have any kind of API facing anyone. Maybe if an API to tap into the server were exposed from the server's end (again, security being a huge deal, obviously), there could be a fairly thin CLI client written on-top of it that could be accessed remotely via login.

Am I treading down the right path here?

Honestly this is not something i believe we need, but the path is totally open and any contribution is welcome.

All you are proposing can be a 3rd party tool that can help numerous shards. The thing is really see interesting is a monitoring dashboard based on log parsing. You can integrate an IRC client for bidirectionnal control (with GUI wrapping).

Hello! I think we may be on to something here. Unless I'm wrong (which I very well may be), DoL doesn't have any kind of API facing anyone. Maybe if an API to tap into the server were exposed from the server's end (again, security being a huge deal, obviously), there could be a fairly thin CLI client written on-top of it that could be accessed remotely via login.

Am I treading down the right path here?

That was what I mentioned. To attach tools like a world editor to a running server would require an API which we currently don't have. Thats why I said some type of SSL secured connection with a simple CLI would be best. The commands issued in the CLI itself can be normal ingame commands or specially designed commands to be used by a tool with parsable xml command output. I had that idea for quite some time but did not get around to do this.

I was actually planning on writing an RCON client for my server when I get to it eventually, but that is a ways off ;d

Was gonna use SSL + regular authentication like how DAOC authenticates so gms/admins can login as normal.