[D] Training StarCoder using 3D parallelism.

Recently I have been working on newly released model StarCoder, I would like to implement 3D parallelism(megatron-deepspeed) on this model to train on custom dataset. I see they have implemented 3D parallelism for GPT model.

Is there a way that I can implement it for StarCoder, if so please provide any reference.

One question, Is the implementation of gpt in huggingface and megatron lm both same?

I have 2 40GB gpus.

dataset deepspeed gpt machinelearning megatron reference starcoder training