[P] Self-Hosting a 16B LLAMA 2 Model in the Banking Sector: What Could Go Wrong?

I've received a freelance job offer from a company in the banking sector that wants to host their own LLAMA 2 model in-house.

I'm hesitating to accept the gig. While I'll have access to the hardware (I've estimated that an A100 80GB will be required to host the 16B parameter version and process some fine-tuning & RAG), I'm not familiar with the challenges of self-hosting a model of this scale. I've always relied on managed services like Hugging Face or …

a100 banking hardware job llama llama 2 llama 2 model machinelearning sector