[D] Benchmarking retrieval across context lengths

The following “needle in a haystack” test for GPT 4 went viral earlier this year, showing 100% retrieval for the first 64k tokens:


https://github.com/gkamradt/LLMTest_NeedleInAHaystack


Is this considered a valid test among machine learning experts? If it is valid, then has it been replicated anywhere else? It seems unlikely that this would be the only public implementation of the test if it was valid.


If it is not valid, what method might be?


Finally, overall how many tokens of context do …

benchmarking context experts gpt haystack machine machine learning machinelearning public retrieval test tokens viral