avcodec_ functions return -12 value when the AVERROR_INVALIDDATA error is encountered. Therefore, you don't use the AV_CODEC_ID_ASS to open the context for the libass subtitle, but the AV_CODEC_ID_MOV_TEXT identifier is used instead. The mov_text codec seems incompatible with the graphical subtitle format you feed to it.