When you put the matrix between $` `$ delimiters, GitHub will render it inline.
$` `$
$` \begin{bmatrix} X \\ Y \end{bmatrix} `$
I created a test page to see how different Markdown flavors render math expressions, and what are their limitations.