Pytorch is generally the go to library to implement new architectures. It's very good for prototyping which is perfect for research. So you should start with Pytorch imo. Also, writing an algorithm from scratch is a lot of work and will probably be less efficient than Pytorch because you either don't use GPUs or use them but do not possess the optimized CUDA kernels Pytorch has access to.