Drow language

[Editedmind]

Interesting topic. I'm a bit under the weather, so I might not be very coherent, my apologies for this...

My thoughts on this subject are simple, I can't see this being done as described, not without breaking the back of any popular server you're running it on. The DMFI languages were actually disabled on several servers I played on with NWN, because the admin deemed it far too resource intensive for what it was worth.

Such a language translation system would be even heavier on system resources if done in a similar manner, though instead substituting word for word. I would also caution against using a custom 2da, or even dealing directly with any 2da in a script, or any external server side application.

On another note, I had a thought of perhaps being able to create a client side GUI plugin that could be used to copy/paste translations for ease of use, rather than referring to a translation website. The downside to this is that as such, it wouldn't be used to translate real-time for those characters that "know" the language. There could be a work around though, like having the plugin send both the translated and untranslated string to the server via GUI scripting, and then have the player speak the translated text, while the untranslated original text is sent to those with the language... If any of that happens to make much sense.

Other sensible features, for however it's eventually done, would be to have case that catches unregistered words, and appropriately garbles them so that you avoid a mishmash of the player and the character's respective languages.
The other feature would then, of course, be to pick up any words that start and end with a pair of characters. For example ##blah blah blah## might pass through the system without being translated. This would be very useful for the cases in which somebody uses a name.

[Velkyn]

If 2DA caching has been implemented fully then it isn't so much a problem for making sure the speed of looking up refs for language isn't compromised, and same for system resources. If not, then build the listing in a script and cache that.

In regards to the DMFI language tools being too resource intensive, wouldn't that then be a fair assumption of most all the DMFI scripted tools? None of them use 2DA's, some of those scripts are quite extensive and make use of multiple callbacks/outsourcing to other scripts. Unless the module designer went and cached the most widely used scripts, yeah, they're gonna start to eat up resourcees everytime they're opened and then closed.

I have played on several servers where the DMFI language tools are enabled and didn't notice any real performance hits. Of course, I'm not gonna worry about a little lag here and there, there's more of a wait when I'm standing around doing nothing while someone types a reply - now THAT'S a total pain in the arse, lemme tell you!

As far as making an external system that you type into, hit "translate" and then copy/paste back into the chat window? That's even worse than waiting a few miliseconds for the server to check the input and spit out the translated equivalent.

Now, I haven't done any speed tests on this "true-form" translator, but I don't see it being too heavy. 'Course, I'm no programmer so it could wind up being so resource heavy that it kills the server at startup. Only by testing it will it be known, and from there a work-around or tweak could be found to boost performance.

It's still an idea I have in the far reaches of my mind (lotta acreage for rent in here, so keeping track of ideas not nailed down means they take longer to find and work on later) and I'll give it a shot even just for my own curiosity with my results posted here for others to decide upon.

At any rate, some form of language system, whether true-to-life or mish-mashed letter swapping, can only add a little more flavour to the RP aspect of a server. If you aren't interested in RP, don't use it and let everyone go as they are.

Oh! Your idea of having a "backup" system to catch worrds that aren't known by the language database is an excellent one! I was wondering how that would be handled, so kudos to you! You've given me another component to think on and work out. That alone should be enough, combined with a somewhat complete "dictionary" of words, to slate anyone's appetite for a language system.

Heh... now I really wanna get this working! But that'll come *AFTER* I get the Caster Mana System and [freaking!] walkmeshes finalized an working.

Until then!

-Velkyn

[Editedmind]

I think the last I heard of 2da caching it had some quirks, and I believe it only cached a couple (Disregard, I just checked, and it seem to have been changed a number of patches ago). Though calling on 2da's still seems to be generally discouraged, if you do happen to test it, I would also be interested in knowing the speeds for both cached and uncached script-based and 2da-based language files.

The main difference with the DMFI scrambling method, as opposed to translation, is that you're dealing with an entire vocabulary, as opposed to an alphabetic cypher that only has 27 cases. It's the sheer quantity, that alone will require quite a robust algorithm to either loop through definitions, or sift through grouped associations.

It's not that I mind a bit of lag, but these things tend to add up, at the very least, it's always best to release work that runs as quickly and efficiently as is possible.

I'm currently busy with some work myself, so I don't think I will have much time for this. I am however interested in seeing how it goes, and perhaps if I find myself with a lot of free time one weekend and a pot of coffee, I might give it a shot myself.

[Editedmind]

I've recently started some work on this. So far I've got a basic set of functions to reconstruct, translate, and reconstruct the translated strings. I've also decided to go with the 2da method for now, simply to keep the script clean while I fine tune and add some more functionality. I have noticed that the 2da method will require some extra work as opposed to keeping it all within the actual script files... Eventually I'll have both methods complete for comparisons in speed and functionality.

I'll be basing this off the language resources at Eilistraee.com, largely because the Grey Company's drow language pages seem to be down. With that said, there are several words that don't appear to be quite authentic, but I'll get back to that once I've actually compiled the entire database.

Something else which is disturbing me is my inability to cache scripts in the module. I'm not sure if I missed something, or if my toolset is bugged, but when I try to cache scripts no entry field appears after I press the 'add' button. Perhaps somebody here could shed some light on that?

[Velkyn]

I will check later tonight about the script caching setting and get back to you on it.

About the speed of a 2DA file reference system, I did have another thought on it which could make things a hundredfold faster. Well, I may be exaggerating a bit, but perhaps not.

A while back in NWN1 someone came up with an ingenious way of increasing the speed of the built-in Bioware database system by a factor of roughly 500%. Now, using that principal and putting it to use for the language reference system, I think it can cut out nearly all the lag from translating aside from server resource issues.

Here goes...

Put somewhere into an unaccessible area (similar to the DMFI anti-duping crafting area of NWN1) an NPC. Call this NPC "Universal Translator." Now, with some extra scripting, once the module has finished loading, the language.2da file will be accessed and every entry in it will be passed off as variables and saved onto the "Universal Translator" NPC. The scripts that handle the reading, crossmatching and reconstructing of speech would then be directed to check for matching variables on the NPC.

Now this whole system would be an undocumented and currently untested amount of a speed increase as opposed to using a 2da file simply because the system can access a variable on an object Greater-Planar-Deity faster than opening, referencing and closing a 2DA file multiple times for each word.

But we don't do away with the 2DA file, oh no! 2DA files have a special ability: updatable on the fly! Unless the 2DA file is a *system* file, it can be altered and updated without shutting down and restarting the server program. This means that for a stable server PW, the language.2da file can be updated with new words externally and from within the game server, a DM or just the Admin can run a maintenance script that reloads all the info from the language.2da back onto the NPC so that it is kept constantly current.

For this to work there shouldn't really be any more commands or anything too extreme for accessing the 2DA file initially and then afterwards accessing the NPC variables. It will be a little bit of a switching sequence but is completely doable and in the long run should cut down on system resources being tied up and bogging everything else down.

As you're already heading into the area of testing portions of this system, I'd be willing to work with you for either ideas (I'm big on those in the area of scripting) or even testing the system out. Between the two of us (and anyone else hankering to toss out some help) we should be able to devise a slick li'l process of adding even more flair to the overall game.

Sincerely,

-Velkyn

[dunniteowl]

Just a thought:

Would such a Universal Translator NPC be required for each language that has to be translated?

I'm just wondering as I have no idea how that might work. It does occur to me to ask, though, would there be an NPC required for each paired translation?

(ie: ) Common/Drow, Orcish/Drow, Dwarven/Common, Orcish/Common, Orcish/Dwarven and so on...?

Am I making any sense here?

dunniteowl

[Velkyn]

dunniteowl,

I understand where you're coming from, and no, you wouldn't need a separate NPC or pair of NPCs for each language. Because the system can access and work with the variables so much faster than an external file (no opening/closing of files) everything can be handled on the NPC. The scripted process for handling each request would only need to include what language you are trying to communicate with thereby indicating which set of variables to cross reference. Everything setup on the NPC would mimic a Database with some obvious modification. For instance, for the Common word "hidden" the variable on the NPC could contain all the words in the other languages that mean "hidden" but have a prefix attached so as to be able to identify which language it is, ie: "d_velkyn" for Drow. In this respect, when the translator gets the word "hidden" and the speaker is talking in Drow, the translator checks for the variable "hidden" on the NPC and looks inside to see what word begins with "d_" and puts the word found into the translated sentence.

Now you could go through and setup different Translator NPCs for each language, it could make the process work even faster as there would only be the lookup and reconstruction phases of the process to go through. That is a good idea dunniteowl, one definitely worth looking at and putting in, especially with a system that could be as intense as this one.

What are your thoughts Editedmind?

Sincerely,

-Velkyn

[Editedmind]

Alright, well I've got roughly half the compiled list of words into the 2da, so it will probably be early next week before I can give real answers on the speed of the system.

I would have to agree that a local variable system is looking pretty enticing at the moment, more so because we can call values by name rather than number. On the other hand we're also talking about roughly three to four thousand strings for each column. So with a robust system that also features multilingual support we could be talking about roughly 50,000 variables... To be honest, I'm not entirely sure what kind of effect that might have on performance. It's really a shame that we can't save a struct as a variable for something like this... Actually, I've assembled and disassembled multiple variable as one before, so it shouldn't be too much of a problem.

The other thing I was thinking of was perhaps keeping the 2D array cached and then using a system of local variables to match the reference/input language with the variable name, and subsequently provide the correct row via the returned variable.

So far it seems that each method has it's own set of merits and pitfalls, so we're just going to have to tinker away till we find which works best, or create an amalgam. Either way it looks like we're on the right track as far as the system's speed and efficiency goes.

It's still pretty early on in development to really involve other people, as I'm still making little tweaks here and there, like testing buggy prototype functions to get an idea of what will work best. Heck, I'm still using a listener (Illithid in a cave named Translator) to handle the input and give me test readouts. If I had to think of anything that would lighten the eventual workload, in order of importance...

1. GUI scenes/screens.
2. GUI related scripts for input/output. Though this would be pretty trivial.
3. A quick scrambling function for unknown words. Currently I'm just letting it omit them from the translated string.
4. Adding columns for other output (fantasy) languages. Though, this can only really be done once I've finished the initial columns or it will require too much editing work later.
5. Adding columns for other input (real world) languages, for multilingual support. Again, same problem as above.
6. Authenticity & spell checking. This is going to be a huge pain, but maybe we will get beta-testers for that with any luck.

Realistically speaking, it's probably going to take a month before things come far enough for beta testing. If someone else (sacrificial lamb) could offer to help with 4 & 5, that would take a huge load of work off my shoulders.

Oh, I would also appreciate if someone could double check the script caching to see if it works for them, or if they have any information about it. Otherwise I may have to try and reinstall, hmm...

Anyway, I'll leave it at that, as I have some work that needs to get done by tomorrow.

[Velkyn]

Just curioius how this project has been going. Much like a working mana system, a more-or-less true language system would be even more icing on the cake. I don't know if the recent 1.13.409 patch has put in the "listening" channel yet so as to remove the dependency for invisible listening creatures, but even if not, having a way of translating to/from a *real* (cough-hack-hack-choke!) language would be stellar.

Sincerely,

-Velkyn, taking my turn at poking the bear in the cage ^_^

[birtomshri]

Originally Posted by Xandurpein

I think your idea of something that turns real language into gibberish has some merit, some people may want help to create lines that looks like a language, just like some people like the random name generator. I have no idea how hard it would be to incorporate this into the Toolset language tool, but to achieve the same effect, all you really need in a dialogue is two lines one with plain text and one with your gibberish, however you generated it, and a simple set of conditions for which line is displayed.

I do not think a drow translator is the same as such a tool however. For those who enjoy the added immersion of a "real" drow translator, it is not the same as random gobblegook.

I agree with you.
Good post. I appreciate it
This is an interesting discussion. thank you for sharing