MongoDB is not an RDBMS database so you can't add a datasource using RDBMS type. Only option is to use custom datasource. For that you can use the implementation as in [1] and copy the jar to the MI_HOME/lib folder. You can see sample datasource configurations from [2]

[1] https://github.com/wso2-attic/wso2-dss-connectors/blob/master/mongodb/src/main/java/org/wso2/dss/connectors/mongodb/MongoDBDataSource.java

[2] https://github.com/wso2/product-ei/blob/master/distribution/src/conf/custom-datasources.xml