Have you debugged your code with the debugger to see what happens?
But most of the times just an illegal instruction exception is thrown out.
If you can catch this under the debugger and see if the instruction is actually in the binary, then we can talk more constructively about the problem e.g. compiler generates invalid instruction for your target.
I would suggest you start by debugging the application.
Debugging shows that different optimization flags make different stack sizes.
The unoptimized code gets a bigger stack and, thus, works good.
Well, the stack size is always the same.
But it looks like the stack end is aligned to the end of the page.
Thus, the space available for the stack depends on the the page offset of the end of the bss segment.
So the problem was that the stack size is not sufficient.
But this kind of problem is very hard to investigate if the generated code does not check for available stack space.
I made an assumption that the following compiler flags are sufficient to generate a 'left-stack-space' aware code:
But it does not look like it has any effect on the code generated by my toolchain for this platform.
How do I make the code aware of the free stack space?
BTW: the utility "flthdr" does not complain if you specify stack size equal to, for example, "8K",
but the outcome is not what one should expect.
The stack checking support in the compiler only serves to detect when you run out of stack and allow you to recover from that situation, but this won't help if you consistently allocate more stack than you have.
The use of alloca, and malloc will both return error codes when they run out of stack and they know exactly where the limit lies, but you must check their return values.
The ColdFire port doesn't have a special stack checking implementation (for example one that checks to see if the function call use of the stack might overflow the stack) and uses the default probing mechanism which means that the compiler may probe for stack overflow once ever 2^12 instruction bytes, which in the case of a small device might be once or not at all depending on the layout.
Therefore for a small ColdFire part there is no easy way, except through manual checking, to see if you've overflowed the stack at any given point in time.
You can do this yourself by using `-finstrument-functions` and writing a hook function that checks to see where the stack is at during entry and exit of a function.
Hopefully that helps you out.