This is a bigger question than it appears to be. You will get many opinions about this. The classic LEDE design (and the modified RFZ Designs that we use) are based on a certain amount of time passing after the direct sound from the speakers hits your ears, before any reflected or diffused sound returns to the listening position. Generally this ITG Initial Time Gap is accepted to be between 20 -25msec. This would call for a distance Delta of at least 22 feet (speed of sound x time).... In other words, you would want the ROUND TRIP to be at least 22 feet for the sound to travel past your ears, hit the diffusors (or other reflective/diffuse surfaces) and travel back to your ears... So, any diffusors much closer than 11 feet behind you can be too close...
Now, that's not to say we've never put diffusors into smaller time intervals. If I have a smallish (control) room and I want to put diffusors behind the listening position less that 10 feet from my head, I use a moderate amount of 2D diffusions, such as Omnifussors or Skylines, instead of the 1D QRD style. THe reason is you get more scattering with the 2D guys, and as such a little more attentuation (not as much level) coming back. AND, it Sound Good To Me (SGTM --- I'll use that a lot).
Hope this helps.