Rendered at 21:19:17 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
andai 22 hours ago [-]
Great work. Have you shared this with the Math Academy folks? I bet they'd love to copy your homework :)
I did a little deep dive into this kind of stuff over the summer. I haven't written it up yet, but I'll share some highlights below.
I realized Anki was optimized for the opposite of the problems I actually had. Anki's optimized to minimize study time. It does this by maximizing difficulty. For me, this means maximizing pain!
I'm not low on time; I'm low on willpower. I need it to be fun and easy! Otherwise, I quit and memory goes to zero. Gwern implies I am typical in this regard; most people who try SRS don't stick with it, even if they want to. So "Minimize Pain" seems like a worthy endeavour here.
So, by necessity, I made the reps easy. By making the reps easy, I realized I had accidentally become much more fluent. [0]
(The "automaticity" you mention, comes from practicing more frequently.) In other words, working hard (suffering more) was giving me worse results. Imagine! (I had a similar experience with complex motor skills, but that's a separate post!)
At some point I realized I could replace my entire flashcard app with a lightly modded Anki: increase target retrievability (to increase frequency), and use latency as the score (easy with a tiny addon, and/or custom card type).
As far as the math goes, I got stuck on generating the cards (past arithmetic). How did you do that? :)
(Also, I'd love to see the knowledge graph! I've been trying to reverse engineer the old one from Khan Academy...)
Speaking of fluency, I think "overlearning" is a misnomer. Our standards are too low and we are chronically "underlearning". Similarly, our "Mastery Learning" is actually Temporary Basic Competence...
andai 22 hours ago [-]
Oh, a worthy postscript: Ebbinghaus (of the Forgetting Curves) also discovered the relearning curves. If you've learned something before, it becomes much easier to learn again later.
This is relevant because you can't necessarily apply Very Frequent Reps to your entire knowledge base. There's just not enough time to maintain fluency in everything.
But occasionally, you will want to be fluent in something, which you were once fluent in. Then, you simply shift it into Tier A, spam reps, and rapidly regain what was lost -- for as long as it's relevant to you.
gmays 21 hours ago [-]
Thank you! ANd thanks for sharing, that was a good overview and I had a similar experience.
To answer your question, I'm a product guy, so it was easier for me to start with the user app/mobile UX I wanted + constraints rather than starting with the cards.
Then I created a problem view in the dev version of the app that let me see every type of problem to see how it rendered, how input worked, etc.
Then with that plus the system around it (details below) gave me higher confidence in generating the 40,000+ problems across all the topics. It still has room for improvement, but happy with how it came out so far.
So, for the problems, they were programmatically generated, but NOT LLM-generated/trusted, so the process would give me the confidence:
1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions itd produces the same corpus. so there's no AI inventing random questions at runtime .
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it doesnt generate a LaTeX string and cross fingers its interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
So, starting out I assumed doing it programmatically would be liability. But rather confidence comes BECAUSE it's programmatic. For a large class of problems deterministic generators + exact/exhaustive validation was easier to audit than tens of thousands of hand written questions. So could leverage AI for all that, just had to define the rules/review the system.
I'm very happy with the outcome, but in terms of process it far exceeded my expectations in what I learned about approaching problems like this.
I'm a very heavy AI coding using (I burn hundreds of billions of tokens a year!) so this was a fun way to validate my approach and test some new ones to build high quality experiences, particularly on mobile which is more of a taste thing. The decade plus of product experience really came in handy in guiding the AI here, which took a lot of iteration on the experience and trying different things. It was a blast, and I was generally surprised at how fast it came together. Thank you again.
andai 20 hours ago [-]
Wow, that's very comprehensive! I think this would also make a great article.
>1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
Yeah, this is the "knowledge graph", right? (Or does each node in the knowledge graph have multiple problem families?)
How did you come up with the graph itself? The curriculum.
And did you say you're shipping SymPy on the mobile client?
Very cool, thanks!
gmays 18 hours ago [-]
Thanks. Yeah, related but problem families are more separate from the knowledge graph.
To be clear, the knowledge graph here is different than the Math Academy sense. What they have is far more impressive since it's for learning and convers prerequisites, related topics, etc. Where mine is more specifically focused on automaticity, with the assumption you've already learned it and just want to maintain/improve recall.
A recall target is basically the smallest piece of knowledge that gets its own spaced repetition state. So, for example, `7 × 8` can be one target. `8 × 7` is just another presentation of that same target, while something like `56 ÷ 7` is a separate inverse retrieval target.
A "problem family" is more of an authoring/generation construct layered onto that. E.g. is defines a bounded class of problems w/exact operand ranges, mathematical rules, expected answer forms, exclusions, presentation rules, etc. It can then deterministically produce valid problems for the relevant targets. So its not a 1:1 mapping between graph nodes <> problem families.
For the curriculum I tried to keep mathematical identity separate from curriculum placement. The internal graph is organized around coherent mathematical concepts and independently meaningful retrieval skills. Then grade/course views are mappings over that graph. It should roughly follow common core, but with some gaps since I only wanted to cover stuff doable in your head. I also covered add'l memorization topics to supplement the Math Academy courses I plan to do since I'll need those myself. I didn't get them all, but I know the MA team plans to add automaticity stuff, so I assume by the time I get to those they may already cover it anyway.
To clarify SymPy does not ship in the iOS app. It;s only used offline during the content build/validation process for the classes of symbolic math where it's useful. The app only ships the validated content. Runtime grading is bounded + local. E.g. there's no Python, SymPy, runtime AI, or unrestricted CAS running in the app.
I had to iterate a lot to get it performant, working 100% offline and at a manageable size with so much content. There were some compromises but it works reasonably well so far.
codepoet80 9 hours ago [-]
I chose Algebra II, which I'm currently re-visiting as an adult learner starting a new direction in college. It immediately thrust me into problems I've not seen before. I skipped multiple times and was never presented with something I recognized. I went into settings an turned on Algebra I, thinking that would start with more basics, but had the same issue. Gave up after 5-6 tries, feeling just as frustrated and dumb as I do in class.
Programmer of 27 years, doing math by other names. I continue to be frustrated by how bad math-as-a-programming-language is. This tool did nothing to improve that feeling.
gmays 6 hours ago [-]
Thanks for the feedback.
The goal of the product is to give practice and help build automaticity in topics you already know, which makes learning harder stuff (e.g. through Math Academy) easier later to raise your ceiling.
So, I'm not sure if maybe you're less familiar with the material (below the threshold) and you should take a course focused on teaching the material OR that the material in Mathy is too hard for the selected topic. Because if you're familiar, it should be helping you rather than frustrating you.
It's an offline app so I don't capture that kind of data. But you make a good point, and at a minimum for each topic I can have it start with easier problems then increasingly get harder as you do better.
I'll think more about that today and maybe get a fix out once I have some time.
Thanks again for the feedback!
rahimnathwani 4 days ago [-]
This is great, Gabe.
One small piece of feedback: on my Pixel, the Android keyboard obscures the green button. So when I'm answering a numerical question, I use your number pad to enter the digits, and then use the enter key on my Android keyboard. This feels weird, but I guess with regular use it wouldn't.
gmays 24 hours ago [-]
Thank you, I will look at that bug and get something deployed tonight! I only tested on Mac/iPhone, so I appreciate the feedback.
fn-mote 22 hours ago [-]
Even on iOS the keyboard activates in a very distracting way.
gmays 21 hours ago [-]
Thanks for letting me know, will get this fixed tonight.
tehnoslow 4 days ago [-]
A good project. It’s great that the tasks are generated deterministically, without using AI-generated answers
gmays 24 hours ago [-]
Thanks! Yeah, that was a dealbreaker for me since I didn't want to bother with AI slop. So it took a while and a ton of testing/validation, but happy with how it came out.
zhivota 10 hours ago [-]
How did you end up creating the questions? I have a half finished thing like this (more like math academy actually as it's for my kids who I homeschool) in a private repo but I found that generating questions was difficult for several reasons:
- It turns out good math questions have a ton of forms! Multiple choice, fill in the blank equations, word problems, diagram labeling, etc, and a good math book or online tool varies the images and forms used so students don't get bored.
- AI just isn't great at generating these forms directly, so I ended up having to create these heavy question form adapters to try to get the output of AI to work.
At the time I did it, about six months ago, the LLM I could afford to use for the content just wasn't good enough, and I was having a lot of issues with accuracy, adherence to the curriculum item, and a lack of variability in the numbers and words chosen. Though writing this comment actually gives me a few new ideas...
I shelved it but I'm tempted to try again seeing this and knowing how much models have improved.
gmays 7 hours ago [-]
Thanks. I added some structure that made it easier for me to QA, then focused on the process.
I was initially wary of doing it programmatically since the slop risk was high. But in the end, thanks to the process, making doing it programmatically made me MORE confident. E.g. for many probs deterministic generators + exact/exhaustive validation was easier to audit than thousands of manually written ones (there are over 40k problems).
So I could leverage AI mostly after defining the rules and reviewing the system (which did have a lot of manual QA upfront especially for the different types, but went quickly after I built the QA tool). And that let me treat it more like a problem I'm better at solving as a product/systems guy, rather than a math expert.
TL;DR the problems were programmatically generated, but NOT LLM-generated/trusted. Rough process in case it helps you:
1. Define the problem families explicitly. Each has bounded inputs, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions it'd produce the same corpus. So there's no AI inventing random questions at runtime.
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it does NOT generate a LaTeX string and then just hope it's interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc.
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
One thing I'm especially loving about AI is that it lets you convert increasingly more problems to something you're good at, which makes it easier to approach/solve in novel ways. It let me do this with Mathy, which would have been untenable before. And I've been doing the same with robotics, which is exciting as a software/product guy.
hackermailman 22 hours ago [-]
Expii did something similar but abandoned the AI judge that gave you more practice though I don't know why
drivebyhooting 16 hours ago [-]
How did you come up with the curriculum? I tried the third grade curriculum and it’s hard to know if it’s calibrated.
Have you compared the curriculum with Duolingo math, IXL, Kahn academy, etc?
gmays 6 hours ago [-]
Nope, I haven't compared those. It roughly follows common core, so it should be somewhat similar, but I'm not sure how each of those based their curriculum.
Plus I added automaticity related topics from some Math Academy courses I want to take to help raise my learning ceiling.
drivebyhooting 5 hours ago [-]
When you say it follows common core, how did you get granular skills and competencies?
I’ve been doing similar things for math and language arts and I’m looking for curriculum creation hacks.
The best I’ve come up with is plagiarizing directly from books and wrap a dopamine loop around it.
gmays 2 hours ago [-]
Yeah, common core doesn't really give you the graph at the granularity I'm using. I mostly use it as a curriculum/placement authority.
The more granular decomposition is something I built separately. I take a standard/topic and break it down into coherent concepts, then into the smallest independently useful retrieval skills I want to track. For Mathy that's especially constrained because I only care about things that make sense for automaticity, not everything you'd need to teach the subject.
So for example a standard might imply multiplication/division fluency, but internally that gets decomposed into individual mathematical relationships and retrieval directions. `7 × 8`, `56 ÷ 7`, and `56 ÷ 8` are related, but they're not necessarily the same learning state.
Then for procedural material I define bounded problem families rather than copying individual questions. A family specifies the operand domain, mathematical rule, intended strategy, answer form, exclusions, hardest cases, etc, and the questions are generated deterministically from that.
So unfortunately I don't have a magical curriculum-generation hack :) It's been a combination of standards/course outlines for scope + a lot of decomposition + AI helping propose/check things + deterministic validation afterward. It was a lot of experimentation, and I still have room to improve.
I also intentionally don't copy questions from textbooks. The standards and public course outlines tell me what belongs where, but the actual problems/content are independently authored or generated. That makes it much easier to reason about correctness, coverage, etc.
For the domaine, yea, it's important. I built-in steaks and XP/levels since I could borrow the general approach from a previous habit tracking app I made (maincharacter.game). I also added medals for achievements. The website version has a leaderboard, which I may also add to the iOS app, but would take some add'l work since it's currently offline only ATM.
AlchemistCamp 23 hours ago [-]
This is a great project to see. I'm also an early fan and customer of Math Academy and am glad to see something like this focused on micro skills. As much as memorization is dismissed, having automatic recall base skills lets you get much further, much faster in harder ones.
gmays 23 hours ago [-]
Thank you! Yeah, Justin Skycak's writing really turned me on to the benefits of automaticity. Once you're aware of it you start to notice everywhere you lack it and how much it'd help in raising your ceiling.
yurimo 18 hours ago [-]
This is very cool, thank you. I understand the choice of no account, but would have loved if my progress could be linked between devices, maybe icloud sync?
gmays 16 hours ago [-]
You know, I hadn't actually considered iCould sync, thank you! I didn't like the idea of not syncing devices either, but couldn't justify just eating the storage costs since it's a free app.
But using iCloud would let users cover their own storage costs for syncing practice/progress/medal/challenge data.
I'll take a look at implementing this when I get some free time so I can have high confidence in the approach. Thanks again!
fn-mote 22 hours ago [-]
I did some geometry/trig.
Too much fine print. Trig would be better if it showed the picture of a right triangle with all sides labelled. Giving only the essential information doesn’t help with issues like confusing sine and cosine, which a memorization drill should address.
Basically: add more pictures.
Consider that in applications the names of the variables are different. Not sure that I would change this in a memorization app, but there’s no sign the author is aware of it.
gmays 22 hours ago [-]
Thanks, that's good feedback. I will look at options, thank you.
rglover 1 days ago [-]
Dude, hell yeah! I was just shopping for this exact thing not that long ago [1]. I've never been great with math and wanted something I could practice with away from my laptop. Love that it works in the browser, too. Well done, this is really nice.
Enjoyed your post too. Hadn't heard of Math Academy until today, appreciate the in-depth review.
captn3m0 1 days ago [-]
I made something similar for my partner a while ago for math drills but this is way better (and more comprehensive). I gave up after 3-5 modules. Kudos.
gmays 23 hours ago [-]
Thank you!
v9v 11 hours ago [-]
When testing I got a question asking what -3² is, with the intended answer being -9 instead of 9.
gmays 6 hours ago [-]
Thanks. You might be thinking of the order of operations wrong. Example:
−3² = −(3 × 3) = −9
(−3)² = (−3) × (−3) = 9
If there are no parentheses, you first square 3 and then negate the result. But if there are, then you square -3 first as you explained.
I took a look and Mathy correctly accounts for these. But if you're sure your problem had parentheses instead of what you shared, please let me know!
Either way, thank you for the feedback, this was a good thing for me to double check.
v9v 3 hours ago [-]
Sorry, I really was thinking of the order of operations wrong!
oezi 17 hours ago [-]
Can you make this multi-lingual? Say top 20 languages?
gmays 6 hours ago [-]
Thanks for the suggestion. I've considered this, but since it's a free app I didn't want to expand the maintenance burden beyond what I can keep up with.
There are some other things I want to improve first, but I'll definitely consider this if I see more demand/requests for it.
I did a little deep dive into this kind of stuff over the summer. I haven't written it up yet, but I'll share some highlights below.
I realized Anki was optimized for the opposite of the problems I actually had. Anki's optimized to minimize study time. It does this by maximizing difficulty. For me, this means maximizing pain!
I'm not low on time; I'm low on willpower. I need it to be fun and easy! Otherwise, I quit and memory goes to zero. Gwern implies I am typical in this regard; most people who try SRS don't stick with it, even if they want to. So "Minimize Pain" seems like a worthy endeavour here.
So, by necessity, I made the reps easy. By making the reps easy, I realized I had accidentally become much more fluent. [0]
(The "automaticity" you mention, comes from practicing more frequently.) In other words, working hard (suffering more) was giving me worse results. Imagine! (I had a similar experience with complex motor skills, but that's a separate post!)
At some point I realized I could replace my entire flashcard app with a lightly modded Anki: increase target retrievability (to increase frequency), and use latency as the score (easy with a tiny addon, and/or custom card type).
As far as the math goes, I got stuck on generating the cards (past arithmetic). How did you do that? :)
(Also, I'd love to see the knowledge graph! I've been trying to reverse engineer the old one from Khan Academy...)
--
[0] The ancients knew this principle well...
https://en.wikipedia.org/wiki/Precision_teaching
Speaking of fluency, I think "overlearning" is a misnomer. Our standards are too low and we are chronically "underlearning". Similarly, our "Mastery Learning" is actually Temporary Basic Competence...
This is relevant because you can't necessarily apply Very Frequent Reps to your entire knowledge base. There's just not enough time to maintain fluency in everything.
But occasionally, you will want to be fluent in something, which you were once fluent in. Then, you simply shift it into Tier A, spam reps, and rapidly regain what was lost -- for as long as it's relevant to you.
To answer your question, I'm a product guy, so it was easier for me to start with the user app/mobile UX I wanted + constraints rather than starting with the cards.
Then I created a problem view in the dev version of the app that let me see every type of problem to see how it rendered, how input worked, etc.
Then with that plus the system around it (details below) gave me higher confidence in generating the 40,000+ problems across all the topics. It still has room for improvement, but happy with how it came out so far.
So, for the problems, they were programmatically generated, but NOT LLM-generated/trusted, so the process would give me the confidence:
1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions itd produces the same corpus. so there's no AI inventing random questions at runtime .
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it doesnt generate a LaTeX string and cross fingers its interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
So, starting out I assumed doing it programmatically would be liability. But rather confidence comes BECAUSE it's programmatic. For a large class of problems deterministic generators + exact/exhaustive validation was easier to audit than tens of thousands of hand written questions. So could leverage AI for all that, just had to define the rules/review the system.
I'm very happy with the outcome, but in terms of process it far exceeded my expectations in what I learned about approaching problems like this.
I'm a very heavy AI coding using (I burn hundreds of billions of tokens a year!) so this was a fun way to validate my approach and test some new ones to build high quality experiences, particularly on mobile which is more of a taste thing. The decade plus of product experience really came in handy in guiding the AI here, which took a lot of iteration on the experience and trying different things. It was a blast, and I was generally surprised at how fast it came together. Thank you again.
>1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
Yeah, this is the "knowledge graph", right? (Or does each node in the knowledge graph have multiple problem families?)
How did you come up with the graph itself? The curriculum.
And did you say you're shipping SymPy on the mobile client?
Very cool, thanks!
To be clear, the knowledge graph here is different than the Math Academy sense. What they have is far more impressive since it's for learning and convers prerequisites, related topics, etc. Where mine is more specifically focused on automaticity, with the assumption you've already learned it and just want to maintain/improve recall.
The Mathy knowledge graph is roughly:
Domain > Topic > Concept > recall target > prompt variants
A recall target is basically the smallest piece of knowledge that gets its own spaced repetition state. So, for example, `7 × 8` can be one target. `8 × 7` is just another presentation of that same target, while something like `56 ÷ 7` is a separate inverse retrieval target.
A "problem family" is more of an authoring/generation construct layered onto that. E.g. is defines a bounded class of problems w/exact operand ranges, mathematical rules, expected answer forms, exclusions, presentation rules, etc. It can then deterministically produce valid problems for the relevant targets. So its not a 1:1 mapping between graph nodes <> problem families.
For the curriculum I tried to keep mathematical identity separate from curriculum placement. The internal graph is organized around coherent mathematical concepts and independently meaningful retrieval skills. Then grade/course views are mappings over that graph. It should roughly follow common core, but with some gaps since I only wanted to cover stuff doable in your head. I also covered add'l memorization topics to supplement the Math Academy courses I plan to do since I'll need those myself. I didn't get them all, but I know the MA team plans to add automaticity stuff, so I assume by the time I get to those they may already cover it anyway.
To clarify SymPy does not ship in the iOS app. It;s only used offline during the content build/validation process for the classes of symbolic math where it's useful. The app only ships the validated content. Runtime grading is bounded + local. E.g. there's no Python, SymPy, runtime AI, or unrestricted CAS running in the app.
I had to iterate a lot to get it performant, working 100% offline and at a manageable size with so much content. There were some compromises but it works reasonably well so far.
Programmer of 27 years, doing math by other names. I continue to be frustrated by how bad math-as-a-programming-language is. This tool did nothing to improve that feeling.
The goal of the product is to give practice and help build automaticity in topics you already know, which makes learning harder stuff (e.g. through Math Academy) easier later to raise your ceiling.
So, I'm not sure if maybe you're less familiar with the material (below the threshold) and you should take a course focused on teaching the material OR that the material in Mathy is too hard for the selected topic. Because if you're familiar, it should be helping you rather than frustrating you.
It's an offline app so I don't capture that kind of data. But you make a good point, and at a minimum for each topic I can have it start with easier problems then increasingly get harder as you do better.
I'll think more about that today and maybe get a fix out once I have some time.
Thanks again for the feedback!
One small piece of feedback: on my Pixel, the Android keyboard obscures the green button. So when I'm answering a numerical question, I use your number pad to enter the digits, and then use the enter key on my Android keyboard. This feels weird, but I guess with regular use it wouldn't.
- It turns out good math questions have a ton of forms! Multiple choice, fill in the blank equations, word problems, diagram labeling, etc, and a good math book or online tool varies the images and forms used so students don't get bored. - AI just isn't great at generating these forms directly, so I ended up having to create these heavy question form adapters to try to get the output of AI to work.
At the time I did it, about six months ago, the LLM I could afford to use for the content just wasn't good enough, and I was having a lot of issues with accuracy, adherence to the curriculum item, and a lack of variability in the numbers and words chosen. Though writing this comment actually gives me a few new ideas...
I shelved it but I'm tempted to try again seeing this and knowing how much models have improved.
I was initially wary of doing it programmatically since the slop risk was high. But in the end, thanks to the process, making doing it programmatically made me MORE confident. E.g. for many probs deterministic generators + exact/exhaustive validation was easier to audit than thousands of manually written ones (there are over 40k problems).
So I could leverage AI mostly after defining the rules and reviewing the system (which did have a lot of manual QA upfront especially for the different types, but went quickly after I built the QA tool). And that let me treat it more like a problem I'm better at solving as a product/systems guy, rather than a math expert.
TL;DR the problems were programmatically generated, but NOT LLM-generated/trusted. Rough process in case it helps you:
1. Define the problem families explicitly. Each has bounded inputs, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions it'd produce the same corpus. So there's no AI inventing random questions at runtime.
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it does NOT generate a LaTeX string and then just hope it's interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc.
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
One thing I'm especially loving about AI is that it lets you convert increasingly more problems to something you're good at, which makes it easier to approach/solve in novel ways. It let me do this with Mathy, which would have been untenable before. And I've been doing the same with robotics, which is exciting as a software/product guy.
Have you compared the curriculum with Duolingo math, IXL, Kahn academy, etc?
Plus I added automaticity related topics from some Math Academy courses I want to take to help raise my learning ceiling.
I’ve been doing similar things for math and language arts and I’m looking for curriculum creation hacks.
The best I’ve come up with is plagiarizing directly from books and wrap a dopamine loop around it.
The more granular decomposition is something I built separately. I take a standard/topic and break it down into coherent concepts, then into the smallest independently useful retrieval skills I want to track. For Mathy that's especially constrained because I only care about things that make sense for automaticity, not everything you'd need to teach the subject.
So for example a standard might imply multiplication/division fluency, but internally that gets decomposed into individual mathematical relationships and retrieval directions. `7 × 8`, `56 ÷ 7`, and `56 ÷ 8` are related, but they're not necessarily the same learning state.
Then for procedural material I define bounded problem families rather than copying individual questions. A family specifies the operand domain, mathematical rule, intended strategy, answer form, exclusions, hardest cases, etc, and the questions are generated deterministically from that.
So unfortunately I don't have a magical curriculum-generation hack :) It's been a combination of standards/course outlines for scope + a lot of decomposition + AI helping propose/check things + deterministic validation afterward. It was a lot of experimentation, and I still have room to improve.
I also intentionally don't copy questions from textbooks. The standards and public course outlines tell me what belongs where, but the actual problems/content are independently authored or generated. That makes it much easier to reason about correctness, coverage, etc.
For the domaine, yea, it's important. I built-in steaks and XP/levels since I could borrow the general approach from a previous habit tracking app I made (maincharacter.game). I also added medals for achievements. The website version has a leaderboard, which I may also add to the iOS app, but would take some add'l work since it's currently offline only ATM.
But using iCloud would let users cover their own storage costs for syncing practice/progress/medal/challenge data.
I'll take a look at implementing this when I get some free time so I can have high confidence in the approach. Thanks again!
Too much fine print. Trig would be better if it showed the picture of a right triangle with all sides labelled. Giving only the essential information doesn’t help with issues like confusing sine and cosine, which a memorization drill should address.
Basically: add more pictures.
Consider that in applications the names of the variables are different. Not sure that I would change this in a memorization app, but there’s no sign the author is aware of it.
[1] https://graybearding.bearblog.dev/how-im-teaching-myself-mat...
−3² = −(3 × 3) = −9 (−3)² = (−3) × (−3) = 9
If there are no parentheses, you first square 3 and then negate the result. But if there are, then you square -3 first as you explained.
I took a look and Mathy correctly accounts for these. But if you're sure your problem had parentheses instead of what you shared, please let me know!
Either way, thank you for the feedback, this was a good thing for me to double check.
There are some other things I want to improve first, but I'll definitely consider this if I see more demand/requests for it.