Developer challenge: Ideas for hardcode a stack or block with asset loading (js/css) in themes

Permalink
Hey Devs!

I stumbled over a concrete5 issue which is freaking me out right now. There must be a way to solve this in a nice and efficent way. I’m not a concrete5 guru, so please correct my statements below because some of them are based on assumptions.

The problem:

Hardcoding a stack or a block in a theme page type will not load the associated js/css-assets. The issue is mentioned here for 5.5.2.1:
http://www.concrete5.org/developers/bugs/5-5-2-1/stacks-programatic...

Sample code would be for /themes/<my_theme>/default.php:

// hardcode a block
$navBT = BlockType::getByHandle('autonav');
$navBT->render('view');
// hardcode a stack 
$stack = Stack::getByName('Test Stack');
$stack->display();


The cause of the problem (please correct me if i’m wrong):

Concrete5 Application Flow is the following:
http://www.concrete5.org/documentation/introduction/dispatcher-and-...
The relevant code is here: /concrete/libraries/view.php

Concrete5 checks the database for page areas and the associated blocks. Based on that it will get all associated assets (in a blocks js/ and css/-folder automatically). This also includes assets which are added manually via the addHeaderItem()-method.

// Now, if we're on an actual page, we retrieve all the blocks on the page
// and store their view states in the local cache (for the page). That way
// we can add header items and have them show up in the header BEFORE
// the block itself is actually loaded


The assets are stored in a local cache and the asset “parsing” is complete before we even got to the theme. Comment in code on line 820:
// finally, we include the theme (which was set by setTheme and will automatically include innerContent)


If we now call a hardcoded block in our theme the asset loading will not be called because the view rendering processing has completed that before. This is reasonable because “Loader::element('header_required');” will most definitely be called before any stack/block will appear hardcoded and so any change in headerItems would not cause any change if $stack->display() would handle that.

To sum this up: The view does not care right now if there are any hardcoded blocks or stacks, because blocks and stacks are stored in areas in the database in general.

How do we solve this?

The ideas in my head are:

a) The view has to some kind of “parse” the theme files and check if there are any display()-calls for blocks and stacks. This has to be done before the following lines (789):

foreach($_pageBlocks as $b1) {
$btc = $b1->getInstance();
// now we inject any custom template CSS and JavaScript into the header
if('Controller' != get_class($btc)){
      $btc->outputAutoHeaderItems();
   }
   $btc->runTask('on_page_view', array($view));
}


Parsing would mean some more loading time and memory consumption, but maybe there is another solution? The whole process could be hooked to the on_before_render-Event I guess. So maybe we could also wrap this in a optional library.

b) Add a custom block/stack-Method (displayHardcoded()) which will get the assets and include them dynamically via jquery. This would be a workaround. I think it is possible because we can “simulate” a view render process with the php output buffer and check header/footerItems afterwards.

I think option a) would be the nicest (of course). Do you have suggestions or other approaches? I would be really really glad if we could get this fixed, because the workflow for concrete5 devs/designers would be so much easier.

Why is this so important?

I love concrete5 for being easy to handle for the customer if he wants to change content, because he knows best about the content of his store or business. This was the main reason that I decided using concrete5, because my colleague and I were tired of being content managers for customers.
Providing hardcoded header and footer stacks is a nice way to avoid customer mistakes (e.g. deleting the footer stack or somehow mess with it). Page defaults and non-edit-permissions on stacks are ok, but they won’t update existing pages, so that is no real option for me as a dev, because i want the most flexibility I can get. And concrete5 is just a small step away from it. ;)

So please share your thoughts about this! Thanks!

Cheers,
Matthias

programmieraffe
 
mkly replied on at Permalink Reply
mkly
There are some "tricks" to deal with this.

The easiest one if it's just javascript is to use this in your blocks on_page_view()
$html = Loader::helper('javascript');
$this->addFooterItems($html->javascript('whateva.js'));


The other is to run the block controller methods manually.
// In the head above header_required
$block = Block::getByName('My Block');
if(is_object($block)) {
  $bc = $block->getController();
  $bc->on_start();
  $bc->on_page_view();
}

Then in the body
$block = Block::getByName('My Block');
if(is_object($block)) {
  $block->display();
}

I'm not sure if I forgot something. I won't be moral of the story guy here, but avoiding this is preferred as you don't get much caching benefits.

Also, sometimes you need to block the output of the top part with some output buffering.

EDIT: oops display not render
marcandre replied on at Permalink Reply
marcandre
Isn't it too late for
addHeaderItem
when the block is hard coded into the theme file?

I'd rather load always the CSS/JS through the theme header file, or try to load the content of those 2 files before the render of the block (maybe possible with
$this->inc()
? Not tried yet.).
mkly replied on at Permalink Reply
mkly
If you do it in the head of the theme before
// This is what outputs the header items
<?php Loader::element('header_required'); ?>

So from what I've found is that you can run those methods first to add the header items. But I rarely if ever do this so I'm not sure how often it works or if there are problems.
programmieraffe replied on at Permalink Reply
programmieraffe
Few short suggestions to mkly's first post:

I want to achieve this for css and js files, so adding only to footer items is not really an option.

Regarding the cache and your second suggestion - the best solution (still) would be if that would be handled by the view, because than this could be added to cache.
programmieraffe replied on at Permalink Reply 3 Attachments
programmieraffe
I ran some quick test with a quick & dirty block, see attachment. The block works with ID's, so only add one at a time to a page.

Edit: One thing is wrong in the screenshot - the additional scripts and stylesheets (beforrender/pageview) are in the extra/-folder, so concrete5 does not autoload them because of js/ and css.
andrew replied on at Permalink Reply
andrew
I don't think your .zip file came through.
programmieraffe replied on at Permalink Reply
programmieraffe
Uploaded it again, should work now.
programmieraffe replied on at Permalink Reply 5 Attachments
programmieraffe
Okay, so I tested this with stacks and the result is the same as with hardcoded blocks, which makes sense. ;) See attachments for full results.

If you want to hardcode a block or a stack with dynamic assets right now, you have to make sure the following:

1) Add all (!) javascript files of your block via on_before_render event to the footerItems in the block controller. (All means even the files in the js folder)
public function on_before_render(){
$bt = BlockType::getByHandle("test_asset_loading");
$uh = Loader::helper('concrete/urls');
$html = Loader::helper('html');
$this->addFooterItem($html->javascript($uh->getBlockTypeAssetsURL($bt).'/js/script.js'));
$this->addFooterItem($html->javascript($uh->getBlockTypeAssetsURL($bt).'/extra/extra-pageview-footer.js'));
}


2) Theres no way to add css files to the head php-style. So either you do this via jquery or you just use inline css.
programmieraffe replied on at Permalink Reply
programmieraffe
Regarding the original topic and my intentions:

There are at least 3 cases for this issue:

1) We want to hardcode an existing block (Block->getByName / Block->getByID) which is already in the database. I don't care about doing this because this is way to insecure, because the block could be deleted on the page he was added and then we're f... ;) (If I understand the docs correctly)

2) We want to hardcode a block on-the-fly (common use-case would be the auto-nav example) - This works by now because no additional files are required.

3) We want to hardcode a stack (which is in the db already):

Hardcoding stacks is my main goal because a common use case for me is that the customer has a slider in his header (which requires additional js and css files). And I think it would be a common use case for other sites.

---

But -> this is what came to my mind after doing the quick&dirty testing:

What would be needed is an additional "Page defaults" interface which can add stacks to areas of page types. Let's call it "Page master stacks". The main difference is that if we change the page master stacks, all pages are "changed automatically". It would be enough if the stacks added in this interface appear at the first position of each area.

So the rendering process for areas on a page would be:
1) is there a stack which is connected in master stacks to this area and this page type? - if yes add the blocks of this stack to the top of the area
2) render the page as usual...

It's the same as with power point masters.This would also enable caching and does not require any hardcoding at all.

So, I'm hoping this is not a totally-random-moment where Andrew steps in and "destroys" my dream because of the complexity. ;) Let me hear your thoughts, I appreciate it.
JohntheFish replied on at Permalink Reply
JohntheFish
I have only been following this loosely, rather than in detail. Have you tried the Global Areas block for this?

http://www.concrete5.org/marketplace/addons/global-areas/...

The reason I ask is because I have been working on an adaptation of the Global Areas block (Parent Areas, currently in PRB) and it seems to cope with stacks in areas pulled in by Global Areas (or my Parent Areas derivative). So maybe coding a Global Areas block into a template (where the Global Areas block pulls in an area with a stack) would work in a similar way to coding any other block into a template.

Just a wild idea. I haven't a clue if it would work, but if it does, it may be an easy solution.
programmieraffe replied on at Permalink Reply
programmieraffe
Hey John!

Thanks for the addition, but I think this will not work. I guess the global areas block also works because concrete5 does know about its blocks (it's referenced in the db) and so it is loaded at the beginning of the view render process.
Hardcoding this into the template would most likely have the same effect as hardcoding stacks or blocks. It's just too late for the block to add something to the head.
mkly replied on at Permalink Reply
mkly
I did post a fairly rambling entry about dealing with css here. Not sure if it would help. But if you are writing your own blocks
http://www.concrete5.org/community/forums/customizing_c5/addheaderi...
programmieraffe replied on at Permalink Reply
programmieraffe
Mh okay. I could also wrap the whole css file with style tags so that it will appear inline in my view. Would be a block-based workaround for css-files. We could also work with @import here, but that would be not good regarding performance (at least what I heard).

I did also a workaround last week where I just did the whole thing manually with output buffering and a custom page type controller:

// controller/page_types/base.php
<?php
// ugly workaround...
// because stacks do not load assets properly_http://www.concrete5.org/developers/bugs/5-5-2-1/stacks-programatic...                
class BasePageTypeController extends Controller {
    public function on_start() {
        // workaround for flexslider...
        $stack = Stack::getByName('Header-Slider');
        if($stack===NULL)
            return;
        ob_start();
        $stack->display();
        $content = ob_get_contents();
        // stop output buffering
        if (ob_get_level() > OB_INITIAL_LEVEL) {


// controller/page_types/news_page.php
<?php
require 'base.php';
class NewsPagePageTypeController extends BasePageTypeController {}


But I think this is just a workaround and it's not really nice. And the c5 view-render process in general works like charm and handles all the assets correct if the blocks are stored in the database referenced to the page.

I'm not sure if it's possible but I'm aiming to a more general solution to this. Maybe Andrew or another core dev can give their opinion about this too. Option A would be the solution with page master stacks mentioned above which is kind of a conversion of page defaults. Or Option B would be a modified view-render-process which will take care of hardcoded stacks.

In my opinion this would be really useful for a lot of c5 users.
programmieraffe replied on at Permalink Reply
programmieraffe
Okay, so good news:

After stumbling about this line
// concrete/libraries/view.php:767
$_pageBlocksGlobal = $view->getGlobalBlocks();


I found the forum post here, describing how global stacks are working:
http://www.concrete5.org/community/forums/customizing_c5/global-sta...

Awesome! Almost exactly what i wanted. Why isn't this documented in the dev guide yet?

There are a few improvements possible regarding this code:
// concrete/models/collection.php
public function getGlobalBlocks() {
         $db = Loader::db();
         $v = array( Stack::ST_TYPE_GLOBAL_AREA );
         $rs = $db->GetCol('select stName from Stacks where Stacks.stType = ?', $v );
         $blocks = array();
         if (count($rs) > 0) {
            $pcp = new Permissions($this);
            foreach($rs as $garHandle) {
               if ($pcp->canReadVersions()) {
                  $s = Stack::getByName($garHandle, 'RECENT');
               } else {
                  $s = Stack::getByName($garHandle, 'ACTIVE');
               }
               if (is_object($s)) {


The globalBlocks are loaded every time, doesn't matter which page-type or if the area is actually used on the page. This is causing 2 problems. (Use case would be two page types. One with global stack "Header-Slider" and one without)

1) performance / page load: this depends on what you actually use in the global stacks, but the browser cache will also take care of this. So that is not really problem.

2) block-incompatibility: If you use a javascript-file (e.g. view.js), you have to make sure you check first if the block (view) really exists otherwise there will be errors. E.g. if the script uses a var from view.php (blockID or something). So not every block type is automatically compatible to global stacks but as a dev you could sort this out of course.

Solution to these problems could also be the proposed "page type => (global) stack" relation in database and a rewrite of getGlobalStacks(). But then this would only be really useful if you do it the way I mentioned it above. Only binding global stacks to page types would not make much sense, because this does not automatically mean that the global stacks are really hardcoded in the page type template.

If you get rid of hardcoding and bind stacks to areas of page types via dashboard (database), then you would get a real benefit, because you're sure that a block is loaded and will be actually rendered.

But for now I can work with this.
arcanepain replied on at Permalink Reply
arcanepain
Hi,

First off, thanks for all your work on this! I've spent a crazy amount of time going through the process on page render with these hardcoded bits, and that whole Stacks thing has really caught me out on a couple of occasions.

That said, yep...i've ended up where you are at the moment - just use GlobalAreas. I have to say, i'm REALLY not that happy with the inclusions of relevant stack-contained block assets on each and every page load (with caching, yes, the impact on the server is reduced after first load, but no everyone uses the caching functionality, not all blocks support it, etc, etc...) but, from a more front-end performance point of view, we're talking extra HTTP requests left, right and center, and these do start to rack-up. Haha...and this is all assuming we don't get any conflicts between stylesheets and javascripts when the global stuff is always being included, even on pages that are not expecting those assets.

In short, can't imagine it's ideal it works this way but, given the way pages are rendered in C5, perhaps just something we've got to work with and work around for now.

Did also really bug me when I had dashboard-created Stacks (ie. created via the dashboard interface and showing up under 'Other Stacks' instead of 'Global Areas') with stuff in, that I just COULDN'T hardcode. Fortunately, if you inset the stack into a page area via the UI, there is a method that changes the stack into a global area ('0' -> '20' in the database) so, again, this kindof works around that issues (although still the assets on every page thing!) I've successfully implemented a isSystemPage() check in the view library which only includes the Global Area stuff when viewing the front-end, and this has worked well...unfortunately i'm not THAT bothered by the HTTP requests of the dashboard though (as it already loads LOADS of stuff).

Haha...mini-rant aside, I am at the same place you are with the 'for now I can work with this'. Not ideal, but perhaps worth the quirks for the functionality they provide.
programmieraffe replied on at Permalink Reply
programmieraffe
Hey arcanepain!

Definitely agree to all of your points. Great that i'm not standing alone at this point. ;)

I think that the improvements regarding our problems could be made and that they would not be super-heavy-complex, because database structure (stack type 0/20) would be suitable for more stack "types". Let's say 10 for Stack::ST_TYPE_PAGE_TYPE_STACK. called "pageTypeStacks":
// warning - pseudocode! ;)
//db-schema:
pageTypeStacks => id | stackID | pageTypeID/(cID) | pageArea
//view.php (around line 767 when the blocks are collected):
$view->getPageTypeStacks()
collection/page -> 
public function getPageTypeStackBlocks() {
    $db = Loader::db();
    $v = array( $this->getCollectionTypeID(), Stack::ST_TYPE_PAGE_TYPE_STACK ); // could be a normal stack, no problems with that
    $rs = $db->GetCol('select id from PageTypeStacks where PageTypeStacks.cID = ? and Stacks.stType = ?', $v );
    // 2DO: join and get stack name
    // same as getGlobalBlocks()
    return blocks;
}
// area render/display code in edit mode:


I don't know if there would be other problems. Versioning is not a problem, because that is handle by stacks interface.

There is one problem I could see: You can't be sure that the area is really hardcoded in the theme, so pageTypeStacks would also not provide 100% security. But that would be a misconfiguration I think and the fault of the admin. ;) And it is way better than just to load it always.

But I'm not that experienced in c5 code, so maybe there are some other problems with this. Have you some considerations for that?
arcanepain replied on at Permalink Reply
arcanepain
Yep...sounds like there might be some mileage here. To be honest, i'm hardly a c5 code ninja myself (I wish!)...I can just about understand what i'm reading, where I need to be, and what I can get away with editing/tweaking.

Sort of bugs me that there are these two different types of stacks - one which works (by including EVERYTHING, whether it needs to or not) and one which just doesn't. Why not just have ONE type of stack, and call it Global Area?! Just confusing when it starts misbehaving, and the documentation as far as it goes just seems to suggest that you can you both more or less interchangeably (hardcode as GlobalArea() or as Stack::getByName()). As far as I can tell, they are pretty much the same except for that Global Stack identifier '20' in the database (and the very comprehensive asset inclusion this causes/allows).

Of course, if this asset things IS the one and only difference between the two, it should be made clear in the Dashboard -- if not via a separate Global Areas page (instead of both just in Stacks) then will a little note to say that one will allow out of the box allow Block assets (CSS/JS) and one will not, and allow the editor/developer to create/select the type they need accordingly. If there's a whole other reason for the two existing side by side that i'm missing...then fine! I'm just not so sure there is.

The problem is that, until a page is first rendered and C5 has a chance to make a proper note of every block - stack contained or otherwise - that is part of that page 'collection', we'll never know for SURE that it's going to render things + functionality correctly. As conscientious developers and editors, yes, we could totally add in a page 'opt-in' step to our workflow to selectively control where extra stack block assets are going to be required but, thing is, a huge amount of non-technical people use C5, and i'm very sensitive to this. I reckon, in the long-run (and this is perhaps the view the C5 team have taken on it) it causes fewer problems to shove all Global Area assets into every page just in case someone gone and added a load of stuff using this cool functionality, only to find the toolbar breaks or their site mega-menu fails to drop or [ insert nifty Javascript widget name here ] just doesn't do its job. I think there is DEFINITELY a performance penalty for this (especially if people go to town on javascript/CSS heavy blocks in multiple Global Areas) and a risk of conflicts, but more often than not I think it probably does the trick, and only irks people like us who bother inspecting the <HEAD> of our pages! :D

Way back when 5.5 was coming out, I submitted a bug complaining about Global Area blocks on my site (was the Header Nav, I think, and the Mega Menu add-on) not including javascript/CSS on first page load...only after rendering the page once (broken) and refreshing did everything show up. At the time, think I tracked this down to the $_arIsGlobal variable (think it was something like that) that needs to trigger and be set on page render before C5 knew the page it was dealing with included a GlobalArea (hence had to go and ferret out extra assets). It couldn't do this until 2nd page load, because page had to be fully loaded before this was checked and variable (and 'isGlobal' setting in the db set). Someone picked up the bug with me (Admad, I think?) and went ahead and submitted a pull which Andrew accepted. Things worked fine after that BUT I think it was at that point that things started getting included EVERYWHERE. I never checked in detail, but I think Admad automatically opted every page/collection into each new/created GlobalArea ('arIsGlobal' column in the db to '1'), basically meaning that C5 didnt have to check on first page render and set this...everything automatically had it.

Long story short, and a chunk of C5 history out of the way, perhaps we don't need to un-do this, but maybe approach everything the other way around? Put the casual users / editors first. Maybe we make it so that each and every page can be opted-OUT of GlobalAreas assets (maybe via simple 'performance' attribute, similar to the caching setting) so that, if overhead is getting noticeable or there are conflicts on particular pages which for whatever reason don't use the offending GlobalArea/stacks, we can bypass the getGlobalBlocks() call in the view library?

Viola! Average users unaffected...and savvy, conscientious developers can easily tune performance and pages to their needs based on a solid understanding of what the Stacks (or Global Areas) are actually doing. In fact...this way, we could probably just the 'Other Stacks' thing all together, and just consolidate on 'Global Areas' (or the other way around). This needn't actually be that scary...just tweak the 'getGlobalBlocks' call in view.php (and in the stacks library or wherever it is) to not filter by that GlobalAreas constant ('20').

Another essay over...what do you think? Would allow those in the know to tweak and customise stack behaviour, won't break anything for those who've just gone ahead and created/used GlobalAreas and now expect them to work, AND wouldn't involve a huge overhaul to the dashboard pages, view library or stacks setup. :)
programmieraffe replied on at Permalink Reply
programmieraffe
Thanks for the long reply. Right now I'm kind of stuck if opt-out would be nicer than opt-in, but I see your point. If I have time I will think about this more. Maybe some other devs in the community have suggestions?