the first problem:
"RTSP Source" i've found a lot of, where can i get it ?
I need to reproduce this on my end to see which mediatypes it gives on output pin. It could be that it has many mediatypes. And muxer gets the first one.
i see the file was muxed with "V_MS/VFW/FOURCC" codecId string (for complete list see matroska codec specs
That's why i suppose mediatype was FORMAT_VideoInfo, because there is no any codecs private data.
And that's why this file is playing only in vlc because others expect this private data.
for correct muxing AVC1 video mediatype should be formatType == FORMAT_MPEG2Video with MPEG2VIDEOINFO structure with correct dwSequenceHeader - which will contain avc1 codec private data.
if in graphedit SMM_MKV_Muxer and SMM File Writer connects fine, I think the problem is in the code that connect it.
Are you sure that you connect Output pin of muxer with filewriter, because the muxer has 2 pins ?
We can assist you if the problem will be reproduced in graphedit.
Waiting for your comments.