How compatible are Sybase and SQL Server clients connecting to the alternate back end? Given the shared history of the Sybase and SQL Server products, and their use of a similar (identical?) wire protocol in TDS, what definite statements can be made, if any, about cross compatibility.

If I have a legacy system (binary, no source) that was written against a Sybase 11 back end, is it likely to run against SQL Server 2008?

Similarly, if I have a Windows binary, linked against sybase libs, would it just work against a Sybase server, in general or of specific vintage/versions?
It's true that Sybase and the Microsoft SQL Server databases share common roots, but they are not any more compatible.

Yes, both engines support T-SQL, they have very similar syntax, even many functions looks and works the same. But this is not enough, between the two products have appeared too many differences at this moment. If you want to migrate between them you need to review code, it's easier because many similarities, but must be done.

Microsoft recommended to use SQL Server Migration Assistant (SSMA) for Sybase - it can help you migrate tables, views, indexes, triggers, and stored procedures. In most cases you need to perform just a few manual patches.

Microsoft provided many helpful resources within subject of Sybase and SQL Server:


Similarities and Differences description 
Guide to Migrating to SQL Server 2008
Guide to Migrating to SQL Server 2012