Project Luna: First Beta Results

After three months of closed beta testing with our first cohort of language learners, we have enough data to share some preliminary findings. The results are encouraging.

The Test Setup

We had 47 beta testers using Luna for Mandarin Chinese learning over a 12-week period. Each user completed at least 3 sessions per week, with sessions averaging 20 minutes. This gave us substantial data on long-term adaptation patterns.

Key metrics we tracked:

• Vocabulary retention rates over time
• Error patterns and correction trajectories
• Session engagement and completion rates
• User-reported satisfaction and "feels like it knows me" scores

Finding 1: Adaptation is Real and Measurable

The most important finding: Luna's recommendations genuinely improve over time, and this improvement is measurable.

We compared Luna's performance to a control condition where recommendations were based only on in-session context (no cross-session memory). By week 8, Luna's vocabulary recommendation accuracy was 34% higher than the control.

The improvement curve follows the pattern we expected from the consolidation model - rapid gains in the first few weeks as the system builds a user model, then slower but steady improvement as semantic patterns solidify.

Finding 2: Error Pattern Recognition Works

Luna's ability to identify and address systematic error patterns was particularly effective. When the system detected that a user consistently confused certain tone pairs, it automatically adjusted its curriculum to provide more targeted practice.

Users who received pattern-based interventions showed 42% faster improvement on their identified weak points compared to users who received standard spaced repetition.

Finding 3: The "Feels Different" Effect

Perhaps the most interesting finding was qualitative. In exit interviews, 38 of 47 users (81%) reported that Luna "felt different" from other language learning apps they had tried. Common descriptions included:

"It's like it actually knows what I'm struggling with"

"I don't have to re-explain myself every session"

"It remembers my mistakes and helps me fix them"

This subjective experience of being "known" by the system correlates strongly with engagement metrics. Users who reported high "feels like it knows me" scores had 2.3x higher session completion rates.

What We're Improving

The beta also revealed areas for improvement:

• Cold start problem: The first 2-3 sessions feel generic before enough data accumulates. We're working on better initialization strategies.
• Explanation transparency: Users want to understand why Luna is making certain recommendations. We're adding more "because you struggled with X" explanations.
• Cross-language transfer: Some learning patterns should transfer between languages, but our current model treats each language independently.

What's Next

We're expanding the beta to include Spanish and French, and opening up applications for our second cohort. If you're interested in being part of this research, get in touch and select "Project Luna / Beta Access" as your subject.

The underlying technology (MemoryCore) remains open source. If you want to build your own applications with temporal memory, check out the repo.