Yeah, I'd also like to know the reason for not merging that into the package.

The github repo for the Internationalization add-on seems to be pretty dead in general. My other recent pull request (a simple bugfix) has been sitting there for a month without any comments.

I'm not sure why I closed that. Looking through it, I can only guess that at the time I didn't think it was needed. I may have been testing with only one translation active in which case the code that exists would translate the site interface to the non-English language, but with multiple translation locales it might not set the appropriate Translate object to active.

Ok, I think I tried to explain it previously somewhere but I guess my explanation went to the middle of a desert.

But the explanation is briefly that you have e.g. core blocks that are actually used on the front end of the site. E.g. the form block has a button with a "Submit" label and the guestbook block has e.g. "You must be logged in to leave a reply." text in the view layer.

I understand that most of the core translation file contains is dashboard translations and other admin interface translations (like block editing) that are not needed in the front-end of the site but there are actually places (like the ones mentioned above) where the core translations are also needed for the site context. Of course, these can be manually added to a dummy file containing these needed strings only within the t() function and then including those when parsing through the source for site translations but it would just be easier to include the core translations right away as they already contain these strings and they are usually translated (assuming the translation is somewhat complete).

So, until there is clearer separation for the translation files in the core (in case there is at some day), I think internationalization just needs to load in the core translation. I don't think it's that big of a performance hit and like Franz has said at recently in some TR episode: "If you're running a multilingual site, buy a better server".

By the way, while we're at it and someone from the core team is listening: there is also an issue with the form block error text translations when the form is submitted. This is because they are loaded in the form action (action_submit_form) where the translations have not been loaded yet. This is because the Internationalization add-on loads the translations during the "on_start" event fired by the view class, not during the add-on's own on_start which would be run before the block actions are run.


Antti

Antti,

The internationalization package should load all the core translation files:
https://github.com/concrete5/addon_internationalization/blob/master/...

I'll have to look into the form block and the on_start issue in a day or two when I get time.

Best,
Ryan

I'm not sure if you're referring to the correct LOC but no I don't believe it loads those.

Lines 89 and 95 load the SITE INTERFACE translations, not core translations:
https://github.com/concrete5/addon_internationalization/blob/master/...
https://github.com/concrete5/addon_internationalization/blob/master/...

And line 108 loads the PACKAGE translations, not core translations:
https://github.com/concrete5/addon_internationalization/blob/master/...

If you check the Localization::changeLocale code from the core (in the original pull request referred in this thread), you can see that it would actually load the CORE translations.
https://github.com/concrete5/concrete5/blob/master/web/concrete/core...

And just to be clear, e.g. for locale fi_FI:
SITE INTERFACE: /languages/site/fi_FI.mo
PACKAGE: /packages/pkg_hdl/languages/fi_FI/messages.mo

and CORE (not loaded by Internationalization): /languages/fi_FI/messages.mo


Antti

Ahh, ok, I see where the disconnect is. The approach that I had taken was to start my site translation process with the core translation PO files which isn't really a sustainable approach for core upgrades and such.

Ok yeah, I don't find any documentation on this but my understanding has so far been:

Core translations should include: everything inside /concrete folder
Site translations should include: everything EXCEPT for the /concrete and /packages folder
Package translations should include: everything inside the package (which is probably the most obvious part here)

Or that's what we explain in the Translations Manager docs at least:
http://www.concrete5.org/marketplace/addons/translations-manager/do...

I don't know if there is also an official source for that but I believe that's how it should be...

And I don't say we wouldn't have our "own cow in the ditch" (like we say "vested interest" in my language) as we've got some support requests on these issues for Translations manager. But then again, it would be better for all if they just worked without having to do any workarounds.

Yeah, that's the way it should be done. I'll add that when I get time over the next few days, of if a pull request shows up I'll happily accept it.

Thanks Antti, ryan, I am glad to read this discussion.
And sure, messages in the core blocks (like form, guestbook, survey, etc.) also should be changed to section's language message.
I am looking forward to update of it.

Hi ryan,

Any news about the form block errors loading issue?

Quite frankly I'm not sure myself which would be the BEST way to solve that but one way would be to do the translations loading already in the on_start method of the package instead of waiting for the view to launch its own on_start event like it's currently doing.

That would not be as optimal but at least it might work.

Best,
Antti

Other issue with regarding to this is translations in the tools requests. E.g. Jordan's ajax form add-on uses tools to get the responses for the forms:
https://github.com/jordanlev/c5_ajax_form/blob/master/blocks/form/to...

During that tools-request, the translations are not loaded at all.

I confirm the ajax request issue.

There is a solution :
http://www.concrete5.org/marketplace/addons/internationalization/fo...

Inside your tool file you can put this :

Hi ryan,

Still no news regarding to this?

One thing I'd also like to point out that in terms of loading the translations, it might be a very handy way to add an event to the core in any place where you'd need translations in. Then Internationalization could simply attach into this event and do its magic.

At least some places I can think of where we need translations:
- normal requests
- tools (core, package)
- block tools
- block actions

One issue is also how to detect the current language section in those places. I know there is the "getSessionDefaultLocale()" method in the "default_language" helper but it doesn't get stored to the current language section unless the user visits the home page where that is called during the detection or if that function is called manually somewhere else during the requests. I.e. if the browser detection wants my default language to be Finnish, the "session default locale" will always store "Finnish" as the locale even though I switched over to the English section of the site.


Antti

Are we sure this isn't bleeding into a core related issue?

Best Wishes,
Mike

Yeah, it might be core related also but the point here is that it should be somehow possible to translate those requests as well. Mostly that was a question towards you guys: "should that be on core level or not?"

I know loading the translations is also currently possible on add-on level also in the listed cases through slightly hacky ways but what I was trying to ask is that is there any chance of thinking this already in the core level which would help loading those translations (without "hacky ways").

Sorry if I'm misleading this conversation but I'm just trying to bring these issues into attention and bring up conversation here. Every time this has been discussed in the forums, it has had very little or no feedback from a member of the core team.

And after all, this is all Internationalization-related because those cases should also be translatable. E.g. one common case is the already mentioned case of the "ajax form" block extension which uses a block tool to respond to the form request.

Hello,
We have been tackling a lot of translation issues with the help of our community. @mlocati, @Remo and others have been extremely helpful. To be honest, it gets a bit confusing for me sometimes as I don't have as much experience as some of our international community developers do with translations.

So maybe open up what is probably the most important issue for you in the bug tracker and we'll take it one at a time and see where we get.

Best Wishes,
Mike

Also, if you were not aware the current git version is ahead of the current version by a fair amount right now, so you might want to check that out as well.

https://github.com/concrete5/addon_internationalization...

Best Wishes,
Mike

Mike,

Thank you very much for your input!

Yes, I'm greatly aware of the i18n improvements in the latest release and that the github version of the Internationalization is more up-to-date than the one available at the marketplace.

You asked me to point out the most important issue related to this bug. I can understand that it might not be that easy to understand not having much of experience of i18n projects. In Europe, it's just very common that the site is in multiple languages so these issues strike us over here in almost every project.

So, my biggest issue related to this bug is what I tried to emphasize: not all content is translated. And this applies also to the very issue in the message start: the core translations. I greatly appreciate that ryan agreed to include my initial fix for this (loading the core translation files) in the internationalization add-on but my point has here been that it's not still enough.

To demonstrate, I think you're able to understand this issue better if I actually show it to you. I've created this very basic demo site with 2 different languages (Finnish and English). I will link to the Finnish section of the site here to demonstrate the issue.

The site is running the very bleeding edge version of internationalization downloaded from github 10mins ago.

A) 1st issue ("block action translations" that I previously brought up):
1. Visit this page (Contact Us in Finnish):
http://demo.mainiotech.fi/c5i18n/fi/yhteys/...
2. Send the form without filling it
3. Check the error messages you see there (I can assure they are included in the Finnish translation file)

B) 2nd issue ("block tool translations" that I previously brought up):
1. Visit this page (Contact Us (AJAX) in Finnish):
http://demo.mainiotech.fi/c5i18n/fi/yhteys-ajax/...
2. Send the form without filling it
3. Check the error messages you see there (I can assure they are included in the Finnish translation file)


In both of these situations the texts are not translated. They are both different kinds of requests, the other one is a "block action" request and the other one is a "block tool" request.

What I've also tried to brought up here is that if the "block tool" issue is fixed, it should also be considered fixing the translations in other tool requests as well (core, package, and site specific tools requests). It's basically the same issue with those that you see with the block tools request.

Hopefully this clears it up for you?


Best,
Antti

I made a pull request that fixes the "1st issue" - problem of block action translations.
here:https://github.com/concrete5/addon_internationalization/pull/43...

However, this can not fix "2nd issue".

I've been created another pull request that fixes "1st issue". I hope to it will be merged!
https://github.com/concrete5/addon_internationalization/pull/47...

Still we need to fix to translate tools message.

All right, this has been pushed to the internationalization repository. Thanks!

Thanks guys, great to see them getting fixed!

I hope that there will be a new Internationalization package soon!
:-)

A new version was released a week or two ago for anyone using concrete5 version 5.6.1 or above:

http://www.concrete5.org/marketplace/addons/internationalization/ch...

I can give a little more background to this:
This user first came to us regarding to a support thread related to the translations manager. He/she reported exactly the same issues what I've listed in this thread, so I suggested to come here and give a shoutout to these issues (so that you'd realize it's just not me being totally wacky and crazy). This was the "issue A" listed above.

And this definitely was not the first time we got a support thread about issues related to the Internationalization add-on.

Once again, I'd like to point out that the issues listed in my latest post are still not fixed in the latest version of Internationalization.

Please see the message above that lists these issues: #535703

Hi Manio,
I would very much encourage you to get in contact with @mlocati, @Remo et al. We have a lot of multilingual activity going on in the core on Github. A lot of these issues are partly core issues so sometimes it's easier to describe there. Also, if you are able to make a small pull request for the fix for this, or send in a patch it might help explain it better. Thanks for all your guidance about internationalization issues. It is very much appreciated.

Best Wishes,
Mike

Thanks Mike!

The problem is that I'm not 100% sure whether it would be actually better to come up with a fix to that in the core than try and hack a workaround in the Internationalization add-on itself.

And that's basically what I've tried to ask you guys. It's really hard to come up with a solution to this if I don't know your point of view.

I can sure let mlocati and Remo (and maybe PatrickHeck) know about these but to me the discussion seems a bit scattered. I think this thread here covers the most of issues that I'm aware of in the Internationalization which is why I've tried to bring this up (so that I wouldn't have to copy-paste these somewhere else).

I'm not actually sure what I'd expect to hear from those guys but if you feel that they might be able to come into some kind of conclusion of where to fix this and how, I can let them know about this thread.

I can sure give my input to the discussion but quite frankly (as said), I'm not 100% sure what would be the best approach into solving these issues (without hacking it into the core).

One suggestion that I've already brought up here is that there would be some kind of generalized event fired in the core that internationalization could attach into and load the proper translations always when that event is fired, whether that would be a tool request, a normal request, or something else.

Best,
Antti

Hello,
The reason I bring up other community members is that at this time, multilingual issues are largely community driven. The reason I brought them up(PatrickHeck as well) is that they have been submitting a lot of code dealing with the core and internationalization issues. To be honest, they would likely understand what needed to be changed more than I would, but I'm going to look into these myself as well.

Best Wishes,
Mike

Thanks!

I also dropped a message to those guys (pointing to this thread) so hopefully we'll hear from them as well.

I've read through all the messages and I can't tell you much more than Antti already did. We currently have these issues but we're using a heavily forked core forever, I assume that we've already fixed that in one way - maybe the snippet I've posted below does the trick..

Whether this should be fixed in the core or not: Personally, I'd integrate this whole add-on into the core, that would make it easier to fix the date-format issue we still have. i18n is such a basic requirement that I wouldn't want to tell anyone to install an add-on. It's like if you'd buy a car without tires - sure you can get them yourself but it's kind of strange if they would sell them without tires..

I guess our US focused core team doesn't agree in which case I'd probably patch the add-on. As far as I can see, this should be pretty easy, but it's possible that I've missed something we'd have to fix in the core.

Hello everybody

I think that the main problem here is caused by the fact that the localization system is setup too early.

We have that the current language is currently determined at the system level, not at the page level.

I suggested two patches (one for the core and one for the multilingual package) that should fix this problem:
core patch:https://github.com/concrete5/concrete5/pull/1375...
multilingual patch:https://github.com/concrete5/addon_internationalization/pull/39...

Please remark that I've not fully tested these patches, I'm simply writing down some ideas I'm having to fix some problems about multi-language sites.

@mlocati. Indeed. I was thinking that pull request dealt with this issue. It was why I brought you up to Antti. Your solution looks pretty straightforward. Thanks.

Antti sent me a message about this discussion. I'll carefully read through these messages but as a start: We're using a patched version for ages and certainly don't have these problems.

I might be missing something, but this is a method I certainly patched:



The only i18n issues we still have are date format related, mlocati has already created two working solutions to this problem:https://github.com/concrete5/concrete5/pull/1375...

I also asked whether the internationalization package should be integrated into the core. That would make a few things we've discussed related to PR 1375 a bit easier. We certainly have functionality in the core which is a lot less important. To me, a software should be multilingual out of the box, but that might just be a European way of thinking.

I also think it would be the best solution to integrate the i18n addon into the core. For me it's really needed with every project - also because it provides funtionality like the site-specific translations.
If this is really no option then I'd go for Micheles first approach using a new "warmup" method.

BTW: Big thanks to Michele for hacking so many great i18n iprovements recently!

EDIT: The only thing, that we'd still need is a solution for single pages like Remo showed it (using $_SESSION["last_page"])

Thanks Patrick ;)

Just a side note: I called the new package method "warmup" to highlight the fact that when it's called the system is not fully initialized: we're just in the middle of the dispatcher, so we need to take care a lot of what's available and what's still to be setup...

Our native speakers might have an even better name, but whatever the name of the method is, I agree!

@Remo, thanks for this information and code. Its helpful in getting an understanding of the issue at hand. I'm going to take a deeper look into this soon.

@mkly: I had a closer look at this issue and the problem is rather simple.

The language switch in the multilingual add-on happens in the on_start event. Please note: It doesn't happen in the on_start method but rather in the event. The event is fired in this line:https://github.com/concrete5/concrete5/blob/master/web/concrete/core...

However, when a block action is processed, the dispatcher runs the code from here:https://github.com/concrete5/concrete5/blob/master/web/concrete/disp...

A "normal" page request goes through this:https://github.com/concrete5/concrete5/blob/master/web/concrete/disp...

That simply means that the order of the calls is wrong. We either have to rewrite the i8n add-on to handle everything in the on_start method or fire the on_start before the actions are processed.

However, there's an even more elegant way that was mentioned before: Why not integrate this whole add-on into the core? It's such an essential function that definitely belongs into the core of any CMS these days...

It's super ugly but this code should explain the issue as well:

https://github.com/concrete5/concrete5/pull/1441...

First of all, great to see all this conversation that this has brought up! I'm very confident this will get resolved at some point.

@Remo, your message above kind of perfectly concludes what I've tried to point out as the reason for this bug. You just put it in a more understandable form above, that is exactly the problem why the translations are not loaded early enough.

There was also a suggestion here by @mlocati that a "warmUp" method would be introduced into the package-level but I don't see that as good option because it would mean that all the packages would be foreached twice during concrete5 startup. I'm not sure how big of a performance hit that would be with a lot of add-ons installed but if it's not, no problems with it.

I'm also with you in the idea that it would be very good to ship the "concrete5 car with wheels" as you put it. But the alternative solution works as well, at least for now.

Why I also think that Internationalization should be thought of at the core level is that it's currently very hard to e.g. know what language should be served in situations when there are tools requests from the site. How can you know for sure (on code level) whether you should load the system interface translation or site-specific translations in those cases? And tools requests can happen in both places.

I have not fully tested these issues on 5.6.2+ version of concrete5 (so skip this if these are already fixed) but at least in the past there has been e.g. different texts loaded during different requests. E.g. let's assume this kind of situation:
- User's system interface language (dashboard) is in English
- Site has a language section for Finnish that the user is currently editing

In this kind of situation at least some items used to load Finnish translations (because of the section of the site) although the user's interface language was set to English. E.g. when opening the editing dialog, the wrong translation was loaded. As said, not fully tested in the latest versions but just something to think about which language should be loaded with these "moving parts" (mainly related to tools requests).

Well, having two active locales (one for the concrete5 interface and one for the current page) may require switching between the two during page load (for instance English when building the toolbar and Finnish when rendering the page content).
This may be implemented, but something will still be in the wrong locale (there's some locale-dependent defines, that we should really get rid of).

Furthermore, we need to set the correct locale when loading dynamic content (for instance, the page properties should be in English, the block actions would be in Finnish).

That's a good subject, but requires active feedback from the core team...

As you can imagine, I completely agree with what you're saying.

I'm fine with a "warmUp" method but share your concerns. However, at the end I'm happy with a working solution, even if it's 10ms slower..

I recently had a customer who wanted to have the toolbar in German, but the actual site translations (form block etc.) in the language matching the site section. I tried to come up with a solution but I had to stop at some point as it goes deep into the core..

Yes exactly and other case even if both of the translations match (site and toolbar target language is the same):
When you edit a block or add a new one, I think the translations are swapped into English (e.g. the "Submit" button in the form block).

Again, not sure whether this still exists in 5.6.2 / latest i18n but this has certainly generated some confusion among our customers. Even a customer who was afraid of publishing their changes after editing the form block, they called me and asked "What did I do wrong because the button text does not translate anymore?".

Of course, I just guided them to publish or preview their changes but it's still a bit of a drawback in usability if these issues make people afraid of using the system. Not in a nagging way as much as it might sound like it, just pointing out.

Best,
Antti

It's the same in the github version..

This is probably a good spot to say this one as I can't find a specific forum about it, but we talked about the automatic translation of translation files and felt like it was pretty risky to tie into a the core installer a 3rd party service like transifex.

But... we think that it might be the time to start including a set of most used languages(maybe 10-12) in the core.

Best Wishes,
Mike

Great news, Mike!
We have 15 locales translated at about 100% (and 20 above 90%).
How would you choose a subset of these locales? Which criteria will you adopt?
If you bundle all the translations at 100%, we have a download size that raises a lot.

Wouldn't it be preferable to download translations at install time?
We'd have a smaller download size and translations always up-to-date.

I agree that we shouldn't implement Transifex-related stuff in the c5 distribution. We could implement Transifex-related stuff in concrete5.org website, and the c5 installations talks only to concrete5.org.
In that direction I suggested the stuff I discussed here:http://www.concrete5.org/developers/pro-accounts/community-leaders-...
In other words, concrete5.org takes care of getting translations from Transifex, and concrete5 installations retrieve the translations from concrete5.org.

Indeed it will get a bit larger. But I think that's still a much safer route. We can delete them if they aren't used at installation or something similar. That said, I guess we should figure out how much bigger 12 most used languages would be and take it from there.

Best Wishes,
Mike

Here's the sizes of the translation files for the languages that are at 95% or better:

Chinese (China): 255K
Czech (Czech Republic): 284K
Danish (Denmark): 278K
Dutch (Netherlands): 288K
Finnish (Finland): 288K
French (France): 301K
German (Germany): 293K
Italian (Italy): 294K
Japanese (Japan): 325K
Persian (Iran): 335K
Portuguese (Portugal): 187K
Romanian (Romania): 275K
Russian (Russia): 378K
Spanish (Argentina): 295K
Spanish (Peru): 295K
Spanish (Spain): 295K
Swedish (Sweden): 284K
Turkish (Turkey): 287K

It's a total of about 5.2 MB (but zipped they are 1.7 MB)

I think that in the above list we should soon include Greek: see the translation progress of it herehttp://concrete5.github.io/concrete5-translations/...

Vasilis (bas_abla on Transifex) is doing a really great work.

As a general rule, we should check the languages that are at 95% or better before every release. Maybe this could be included automatically in a "deploy" task in Gruntfile.