all AI news
Supporting Mixtral in gpt-fast through torch.compile - faster decoding than any non-Groq endpoint(!)
Feb. 27, 2024, 2:27 a.m. | /u/programmerChilli
Machine Learning www.reddit.com
Since then, we also recently added a Mixtral implementation to gpt-fast as well. Check it out here: https://github.com/pytorch-labs/gpt-fast/tree/main/mixtral-moe
Featuring
- (!) no custom kernels
- int8 and tensor-parallelism support
- still very simple (<150 LOC to support)
- faster decoding than any (non-Groq) API endpoint, at up to 220 tok/s/user.
I also wrote a longer-form explanation of the …
api decoding faster groq loc machinelearning merging simple support tensor work
More from www.reddit.com / Machine Learning
[D] Does DSPy actually change the LM weights?
1 day, 2 hours ago |
www.reddit.com
[D] Culture of Recycling Old Conference Submissions in ML
1 day, 5 hours ago |
www.reddit.com
Jobs in AI, ML, Big Data
Software Engineer for AI Training Data (School Specific)
@ G2i Inc | Remote
Software Engineer for AI Training Data (Python)
@ G2i Inc | Remote
Software Engineer for AI Training Data (Tier 2)
@ G2i Inc | Remote
Data Engineer
@ Lemon.io | Remote: Europe, LATAM, Canada, UK, Asia, Oceania
Artificial Intelligence – Bioinformatic Expert
@ University of Texas Medical Branch | Galveston, TX
Lead Developer (AI)
@ Cere Network | San Francisco, US