Which MIME type to use for a binary file that's specific to my program?

I'd recommend application/octet-stream as RFC2046 says "The "octet-stream" subtype is used to indicate that a body contains arbitrary binary data" and "The recommended action for an implementation that receives an "application/octet-stream" entity is to simply offer to put the data in a file[...]".

I think that way you will get better handling from arbitrary programs, that might barf when encountering your unknown mime type.

you could perhaps use:

application/x-binary

what is MIME types

list of mime types

see explanation

mimetype headers are recognised by the browser for the purpose of a (fast) possible identifying a handler to use the downloaded file as target, for example, PDF would be downloaded and your Adobe Reader program would be executed with the path of the PDF file as an argument,

If your needs are to write a browser extension to handle your downloaded file, through your operation-system, or you simply want to make you project a more 'professional looking' go ahead and select a unique mimetype for you to use, it would make no difference since the operation-system would have no handle to open it with (some browsers has few bundled-plugins, for example new Google Chrome versions has a built-in PDF-reader),

if you want to make sure the file would be downloaded have a look at this answer: https://stackoverflow.com/a/34758866/257319

if you want to make your file type especially organised, it might be worth adding a few letters in the first few bytes of the file, for example, every JPG has this at it's file start:

https://i.stack.imgur.com/gj30t.jpg

if you can afford a jump of 4 or 8 bytes it could be very helpful for you in the rest of the way

:)

According to the specification RFC 2045 #Syntax of the Content-Type Header Field application/myappname is not allowed, but application/x-myappname is allowed and sounds most appropriate for your application to me.