When you put the matrix between $` `$ delimiters, GitHub will render it inline.

$` \begin{bmatrix} X \\ Y \end{bmatrix} `$

I created a test page to see how different Markdown flavors render math expressions, and what are their limitations.