Working with ssh via TortoiseSVN on Windows is not endearing to ssh. Of course in this case it's my own fault for not having an https access method to the repository... which is to say, it's my fault for not using the superior server support Subversion provides via HTTP. Anyway, ssh doesn't make for a very good, complete experience. SSH servers and clients are not as easy to abstract or build upon as with other protocols, so actually building ssh support into the server and client is rather hard, and you end up with hacky (IMHO) command-line solutions.
I think you're talking more about bad ssh use than problems with ssh, itself. If you're building your own protocol, layering it on top of SSH takes care of security. If you just want access to files, sftp comes with SSH and is capable-- much nicer than http for filesystem access.