I am currently working with Silverlight and Silverlight Streaming and had an issue over the weekend I am sure I will not be the last one to have so:

Don’t try to load a Silverlight application from the file system, I mean double clicking an html file, prefer to have a little web application or even use “Starting ASP.NET Development Server from a right click in explorer“ to test your application.

The security used in both cases is different so it makes the difference.

Thanks to Mathieu for the support in the development and Jon Galloway for the support in troubleshooting the issue.