For the last few weeks I’ve had a problem with a client site not indexing all content. Cron would run as normal, then redirect to a page in the website. Indexing would never get past 26%, and I couldn’t find anything in the Drupal log or the Apache logs to indicate what was happening.
A clue was found when I realised that the page to which cron was redirecting to contained some PHP in the node. But removing the PHP from the node wasn’t an option – there were several nodes containing PHP, and it was required. Drupal should accept PHP in nodes without any issue.
The solution was provided on the #drupal-support IRC channel by user dp1984; he suggested that I turn off the PHP filter, run cron, and then switch it back on. This sounded (to me) like a workaround, but he assured me (and I proved it on the development version of the site) that this was actually a permanent fix – once the PHP filter was switched back on, cron could run again at will without any problems.
I don’t profess to know why this fix works – only that it does. I’d be interested to know more details of what’s going on, but some days it’s more a case of solving the clients problems without wondering too much why or how you did it..