YOUTUBE
The MemPalace AI memory system, promoted by Milla Jovovich as achieving "perfect scores" on benchmark tests, faces credible allegations of benchmark manipulation, where the system allegedly had access to test answers before evaluation, calling its claimed 96.6-100% performance metrics into serious question.1
MemPalace is presented as a revolutionary AI memory system achieving unprecedented benchmark scores, but investigation reveals multiple methodological issues with its testing, including potential cheating by examining test answers beforehand and questionable attribution of creation to Milla Jovovich versus a crypto developer.1
Benchmark manipulation allegations — The most serious claim alleges MemPalace "looked at the answer to the tests before it did the benchmark," effectively invalidating its claimed 96.6-100% scores on LongMemEval tests.1 [⚠]
Discrepancy in attribution — While presented as Milla Jovovich's creation, investigation shows her name appears in only seven commits with two active days, suggesting the primary developer may be Ben Sigman, a crypto marketplace CEO.1 [✓]
Growing developer skepticism — Multiple GitHub issues document concerns about benchmark methodology, with developers questioning whether the reported scores truly exercise MemPalace's core functionality or reflect ChromaDB performance.2
Controversial promotion strategy — The project combines celebrity endorsement with aggressive benchmarking claims, creating skepticism about whether the tool delivers practical value or serves as marketing for other interests.3
"For all intents and purposes, Mempalace looked at the answer to the tests before it did the benchmark. This isn't a true 100% 98% benchmark score like they continue to claim."
— Source, early in source1"The whole thing is a bit sus. So, if you think this is the answer to any sort of like AI or large language model memory issues you're having, I would be very, very wary."
— Source, late in source1
✓ VERIFIED — Ben Sigman is the actual creator. Research confirms Ben Sigman, CEO of Bitcoin lending marketplace Libre, is the primary developer, despite Milla Jovovich being presented as the creator. The project has drawn criticism as "snake oil" marketing.34
⚠ UNVERIFIED — Benchmark manipulation claims. The allegation that MemPalace examined test answers before benchmarking cannot be independently verified without access to their testing methodology and data.
✓ VERIFIED — GitHub controversy exists. Multiple GitHub issues document developer concerns about benchmark methodology, including issue #29 titled "Multiple issues with benchmark methodology and scoring" with extensive discussion about testing flaws.2
✓ VERIFIED — Claims versus code gap. Independent analysis notes significant discrepancies between the project's promotional claims and its actual code implementation, with questions about whether benchmark scores truly reflect MemPalace's functionality.5
For AI developers: Verify benchmark claims independently rather than accepting promotional metrics, especially when celebrity endorsement creates marketing distraction from technical substance.
For open-source users: Examine GitHub contributor activity and issue discussions to identify potential red flags about attribution and methodology before adopting tools.
For benchmark consumers: Recognise that benchmark scores can be manipulated through methodological flaws, including prior access to test data, making replication attempts essential.
For tool evaluators: Consider whether celebrity-driven projects might prioritise marketing over substantive technical value, particularly when associated with unrelated commercial interests like cryptocurrency platforms.
Source credibility: Medium — YouTube commentary with some specific claims that align with external reporting
Claim verifiability: 3 of 4 key claims verified/verifiable
Potential biases: Source appears critical/skeptical of MemPalace; potential for exaggeration but aligns with documented concerns
Quality flags: Brief source (54 seconds); limited direct evidence presented in transcript
Confidence in synthesis: High — Claims consistent with independent reporting and GitHub documentation
Source, early in source "For all intents and purposes, Mempalace looked at the answer to the tests before it did the benchmark. This isn't a true 100% 98% benchmark score like they continue to claim." ↩↩↩↩↩↩
Verified: GitHub issue #29 documents "Multiple issues with benchmark methodology and scoring" with extensive developer discussion ↩↩
Verified: Kotaku investigation confirms Ben Sigman as primary developer with crypto background, criticism of "snake oil" marketing ↩↩
Verified: Multiple sources identify Ben Sigman, CEO of Bitcoin lending platform Libre, as actual creator ↩
Verified: Independent analysis documents "claims-vs-code gap" and questions about benchmark methodology validity ↩