Symfony, XDebug and the maximum nesting level

Here you are, developing your code based on the Symfony2 framework. Creating a form here, add a Twig template there, until suddenly, boom! Your site doesn’t work anymore, and all the info you can find in your PHP logs is:

PHP Fatal error:  Maximum function nesting level of '100' reached, aborting! in Unknown on line 0

What just happened? Did I create some kind of recursive function I wasn’t aware of, did somebody commit code that I accidentally pulled? Did Jupiter align with Mars and somehow this is causing issues in my code. Who knows? Fortunately for us developers, there is a quick way to deal with this: google it..

A quick query on google actually shows us that we are not the first ones to encounter this issue, and a click on the first link (not entirely unexpected, and actually quite ironic as we see later, is a post on stack overflow), gives us quickly a solution: “You are using XDebug and must set the xdebug.max_nesting_level to a higher value”. A fix that is quickly enough implemented in our php.ini, a restart and we are happily coding again.

Even though this might satisfy many developers, I dear say, the overwhelming majority of developers, but not me. Somehow, this “just increase a number” sounds a bit as a hack to me. And if something looks like a hack, talks like a hack, and walks like a hack, i’m pretty confident about it being a hack. So let’s take a look at the actual error: where does the error come from, and how is it actually caused.

Function calls and stack traces

To understand this, we have to talk about how (PHP) functions work: when you call a function (or method) named A(), and in that function we call another function named B(), somehow PHP must know that when function B() is finished, it should continue with function A() at the place it left off. This information is stored onto a “stack”: when we call a function, PHP pushes information onto a stack, and when we return from that function, it pops that information off the stack again so it knows what to do (this is actually not quite what happens, but let’s keep things as simple as possible).

This stack - besides making PHP actually work - gives us as a developer a good insight when something goes wrong: the stack gives us detailed information in which function the problem occurred, but also which function called what other function, and which function before that etc etc. This is called a stack trace, and is basically a visual representation of the stack:

PHP Stack trace:
PHP   1. {main}() /wwwroot/recipes/web/app_dev.php:0
PHP   2. Symfony\Component\HttpKernel\Kernel->handle() /wwwroot/recipes/web/app_dev.php:29
PHP   3. Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle() /wwwroot/recipes/app/bootstrap.php.cache:2444
PHP   4. Symfony\Component\HttpKernel\HttpKernel->handle() /wwwroot/recipes/app/bootstrap.php.cache:3222
PHP   5. Symfony\Component\HttpKernel\HttpKernel->handleRaw() /wwwroot/recipes/app/bootstrap.php.cache:3071
PHP   6. call_user_func_array:{/wwwroot/recipes/app/bootstrap.php.cache:3109}() /wwwroot/recipes/app/bootstrap.php.cache:3109
PHP   7. Rainbow\FormBundle\Controller\Recipe6Controller->indexAction() /wwwroot/recipes/app/bootstrap.php.cache:3109
PHP   8. Symfony\Bundle\FrameworkBundle\Controller\Controller->render() /wwwroot/recipes/src/Rainbow/FormBundle/Controller/Recipe6Controller.php:55

But a stack, just like any other resource, is not unlimited. Every time you call another function, more information must be added to the stack, and ultimately, this will eat up all your available stack memory. In applications written in for instance C, this will cause a “stack overflow” (this is where the famous site got its name from), but in PHP, it depends on what happens: it’s possible that you hit the memory limit set in your php.ini. This happens in most cases, if your functions uses variables, which is very likely. But, it might be possible that you recurse a function that does not use any variables (or only very small variables, like a single integer or something), or you might have set your PHP memory limit to a very high number (say, 512M or even higher). In those cases, it might not be your PHP memory limit that will run out, but the actual stack memory that PHP uses: since each PHP function call also consists of a few C function calls. In C, there is actually only room for a limited number of nested function calls. Unfortunately, there is no real detection on when you reach this limit. Since every PHP function call takes around 3-4 C function calls, and if there was room on the C stack for say 1000 function calls, you could only do around 250 PHP function calls, before hitting the actual stack limit of the PHP core.

If the PHP memory limit is the limiting factor, your application will nicely terminate with an error message, something like:

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 131072 bytes)

If the C stack is the limiting factor: you would get something like this:

Segmentation fault

There are tricks and ways around this, but that is beyond the scope of this blog post. Fortunately, you don’t see this often: as said before, your PHP memory limit will be often the limiting factor anyway.

The XDebug limit

So, we’ve seen that nested function calls are limited, but unfortunately, there is no standard PHP safeguard against these stack overflows. Only when your PHP memory limit is low enough, you might hit that before hitting a stack overflow. And here is where XDebug comes in: XDebug DOES set such a limit. The reason - most likely - is because it allows you to easily catch runaway recursive functions (functions that will call themselves, which in turn call themselves etc etc). In regular applications, the amount of functions that call other functions should be fairly low. If you take a look at the stack trace above, you see there are 8 levels of function calls.

XDebug by default set this limit to a maximum of 100 nested calls. This means we still have room for 92 deeper calls before XDebug will throw the fatal error we showed in the beginning of the blog post. But why the limit of 100? It seems very arbitrary. But this limit, although arbitrary, is good enough: how on earth would you have an application that calls a function, which calls a function, which calls a function, and this 100 levels deep!? Surely, such code would be insanely complex, not maintainable and should be refactored into decent code as soon as possible.

But we deal with a new generation of PHP: where applications are based on frameworks, and frameworks on abstractions upon abstractions upon abstractions. That’s a “good thing”(tm) though, it allows to create very complex systems in a maintainable way, but on a downside, we have many layers of functionality that calls other functionality. For instance, if you take a look at the stack trace above again, you’ll notice that it takes 7 layers before we hit the indexAction method of the Recipe6Controller, which is where we actually do the things we need to do. In “old-school” applications, this would be done on the first function.

But again: this is not a bad thing. It just means we need to evaluate the way we deal with things, including function nesting. And even then: even the most deeply layered code in frameworks are not even close to this 100 level limit, so even with XDebug, every framework should work fine and without problems on hitting the stack limit.

Twig and hitting the nesting limit

So, if 100 is a good enough limit, why do we keep hitting it on occasion when dealing with Twig?

To understand this, we must take a look at what Twig actually does. Twig can use blocks to define content that can be re-used or overwritten by other template files. This makes it easy to create a base template and define a block named body inside it:

<html>
    <head>
        <title>My title</title>
    </head>
    <body>
        {% block body %}{% endblock %}
    </body>
</html>

Every template now can “extend” this base template file, and define their own “body” block:

{% extends "basetemplate.html.twig" %}

{% block body %}
  <h1>Hello world</h1>
{% endblock %}

Somehow, Twig needs to seek, find and render each block and make sure the output is placed correctly in the extended template.

Within Symfony2, it takes about 10-11 nested functions before we even get to Twig rendering, provided we simply use the render() method within a controller. Furthermore, each nested Twig template we display, it takes another 2 function calls (Twig_Template->display() and Twig_Template->displayWithErrorHandling()), PLUS the actual function that outputs the HTML code (doDisplay() method that is found in your cached twig templates). That is a total of 3 nested functions. In our simple example template above, we have the overhead of 11 nested functions before we actually start the rendering, 3 methods for our actual template that only contains the body block, but which extends the base template, and 3 methods for the base template. Which makes it a total of 17 nested functions. Every time Twig actually needs to display a block (in our case, between our <body> and </body> tags), that too consists of another two extra nested function calls. A total of 19 nested calls are needed, for a seemingly simple example.

Also note that by definition it is common practice that Twig template are separated into at least three different templates: a base template generic for your application, a layout template for your controller / bundle, and the actual controller template. That’s a lot of nested functions by default.

But still, even if we round everything up, we still get to at most 25 nested functions. Not even close to our limit of

1. Next thing we need to take a look at, are Symfony forms.

Twig and forms, a powerful, but nested combination

Have you ever tried to customize a form? Noticed that it consists of all kind of blocks, within blocks, within blocks? You can simply customize a single form field by specifying its name like: form_username_widget, or customize all email fields by creating / overriding the email_widget block, or even every label on every form, by overriding the form_label block.

In fact, this structure brings out the power of both Twig and Forms. If you ever wondered how Symfony forms are rendered as tables, just take a look in Symfony’s form_table_layout.html.twig file. You’ll notice that it “uses” the form_div_layout.html.twig file, which contains the main form elements for displaying forms in a <div> layout, but it will override only 4 small blocks. That’s all it takes to change from <div> layout to a <table> layout. Dividing your templates up in blocks, just like dividing your code up in functions gives us a lot of power and flexibility.

Rendering forms with Twig in Symfony works with a few simple Twig functions like form_row, form_widget, form_label etc. These functions are defined in Symfony\Bridge\Twig\Extension\FormExtension class. If you take a look in this class, you’ll notice that these twig functions mostly are simple calls to Symfony\Bridge\Twig\Node\SearchAndRenderBlockNode. Without going into too much detail, these functions will call the form renderer’s searchAndRenderBlock() method with the correct name. So the Twig function form_label will call the form renderer searchAndRenderBlock with the argument label. This function will search for the correct form block based on the current context, and will take into consideration the form theme, the default form themes, block inheritance etc. Ultimately, it will find the correct block to render, and call the renderBlock() method. This method contains now the correct Twig template file where the block resides in, the actual block name and Twig variables. At this point, it calls the displayBlock method of the twig renderer to actually render the block (funny enough, there is a comment in this function that warns about the XDebug and nesting levels).

So, all in all, it will add a few more nested calls to our already inflating stack. But that still would be ok, if it weren’t for the actual setup of forms. Take a look on how the block form looks like (this block can be found in form_div_layout.html.twig):

{%- block form -%}
    {{ form_start(form) }}
        {{- form_widget(form) -}}
    {{ form_end(form) }}
{%- endblock form -%}

This is the actual twig block that will be rendered when calling the twig function form(). Notice that it only calls other Twig form functions. The form_widget() function, will find and render the form_widget block, which in turn renders the form_widget_compound twig block, which renders the form_rows block, which in turn calls the form_row Twig function. Considering that each twig function creates a few nested function calls, plus displaying each block does so too, it should not be hard to understand how this way we reach our 100 nested function limited.

These days, Symfony2 has a neat little Twig profiler in its web debug toolbar, which gives you a good insight in which blocks are called. Basically it’s a stack-trace on Twig blocks instead of PHP functions:

main 13.44ms/100%
└ RainbowFormBundle:Recipe6:index.html.twig 10.61ms/79%
│ └ ::base.html.twig 10.32ms/77%
│   └ ::base.html.twig::block(title)
│   └ ::base.html.twig::block(stylesheets)
│   └ RainbowFormBundle:Recipe6:index.html.twig::block(body) 9.64ms/72%
│   │ └ form_div_layout.html.twig::block(form) 8.65ms/64%
│   │   └ form_div_layout.html.twig::block(form_start)
│   │   └ form_div_layout.html.twig::block(form_widget) 7.21ms/54%
│   │   │ └ form_div_layout.html.twig::block(form_widget_compound) 7.10ms/53%
│   │   │   └ form_div_layout.html.twig::block(widget_container_attributes)
│   │   │   └ form_div_layout.html.twig::block(form_errors)
│   │   │   └ form_div_layout.html.twig::block(form_rows) 5.72ms/43%
│   │   │   │ └ form_div_layout.html.twig::block(form_row) 1.64ms/12%
│   │   │   │ │ └ form_div_layout.html.twig::block(form_label)
│   │   │   │ │ └ form_div_layout.html.twig::block(form_errors)
│   │   │   │ │ └ form_div_layout.html.twig::block(form_widget)
│   │   │   │ │   └ form_div_layout.html.twig::block(form_widget_simple)
│   │   │   │ │     └ form_div_layout.html.twig::block(widget_attributes)
│   │   │   │ └ form_div_layout.html.twig::block(form_row) 1.53ms/11%
│   │   │   │ │ └ form_div_layout.html.twig::block(form_label)
│   │   │   │ │ └ form_div_layout.html.twig::block(form_errors)
│   │   │   │ │ └ form_div_layout.html.twig::block(form_widget)
│   │   │   │ │   └ form_div_layout.html.twig::block(form_widget_simple)
│   │   │   │ │     └ form_div_layout.html.twig::block(widget_attributes)
│   │   │   │ └ form_div_layout.html.twig::block(button_row)
│   │   │   │ │ └ form_div_layout.html.twig::block(submit_widget)
│   │   │   │ │   └ form_div_layout.html.twig::block(button_widget)
│   │   │   │ │     └ form_div_layout.html.twig::block(button_attributes)
│   │   │   │ └ form_div_layout.html.twig::block(hidden_row)
│   │   │   │   └ form_div_layout.html.twig::block(hidden_widget)
│   │   │   │     └ form_div_layout.html.twig::block(form_widget_simple)
│   │   │   │       └ form_div_layout.html.twig::block(widget_attributes)
│   │   │   └ form_div_layout.html.twig::block(form_rest)
│   │   └ form_div_layout.html.twig::block(form_end)
│   │     └ form_div_layout.html.twig::block(form_rest)
│   └ ::base.html.twig::block(javascripts)
└ @WebProfiler/Profiler/toolbar_js.html.twig 2.84ms/21%
└ @WebProfiler/Profiler/base_js.html.twig 1.36ms/10%

This is just the block calls for a very simple form with just a first name and last name text field. Note that the length does not matter. We are only interested in the depths. One form_widget_simple block is already nested 13 twig block deeps. Depending on the fact if the block is called directly from the twig template, or through a twig form function, it takes between 2 and 4 nested PHP function calls for each level. It turns out, that rendering this simple form takes us to a maximum nesting is 41 levels deep. And this is just a very simple form! Adding a simple date widget gets us even over 50 nested functions! Halfway to our 100 limit!

Going beyond

It’s quite easy to understand how we reach our 100 nested functions: all we need to do is add some deeper nesting in our forms by calling blocks within blocks, preferable by using the twig form functions. A great way to achieve this is through collections. A simple “collection within a collection” is enough to make so many nested forms, which all call blocks and twig form functions, which ultimately all call PHP functions, all nested together. Hitting the 100 limit is almost too easy in such setups.

So, back to the original question: is just increasing the memory limit option in your XDebug configuration a hack? Well, yes and no. First of all, we must ask the question: is a limit of 100 nested functions enough for everybody? Practically the answer is no, as Twig and the Form component has showed us. But could another (better?) architecture of both Twig and the Form component not fix this? The answer: I don’t know. I do not think coming up with a better design for both Twig and the Form component are in fact possible (at least, not structurally). If this would be changed, we would loose a lot of power and it that would definitely not be worth it. On the other side, what do we actually need to fear when we increase the limit? Because, without XDebug, there is no limit at all, and even worse: there is no real way of PHP detecting that you are reaching any boundaries. XDebug tries to protect us from this boundary, and it does this maybe a little bit too fanatically. It’s ok to move the limit a bit up: there is no speed-penalty, not even a real memory penalty. Increasing the limit will not cause any harm. We are still protected against infinite recursive functionality, and we can still use twig and forms without problems. It really is a hack solution, as it does not solve the “problem”, but since there isn’t really a problem to begin with, I personally consider this hack to be a-ok..

A final note: the maximum nesting limit of 100 has been increased to 256 from XDebug 2.3 onwards. This will solve most issues, except for very - very deeply nested forms.