But HRM and TRM were designed for puzzles, not language. I wanted to answer a simple question: can these hierarchical reasoning models generate coherent text at extremely small scales?

Spoiler: Yes, but with some important caveats.

What I Built

I adapted TRM for autoregressive language modeling and trained several variants on TinyStories—a dataset of simple children’s stories. All models were trained at roughly 1M parameters to keep experiments manageable.

I decided to run all of these experiments on my Macbook Pro. It’s slow, but workable at very tiny scales, and allowed me to have some fun with model research on a machine I already owned. If you check out the github you should be able to reproduce what I did here in a few hours on a newer Macbook Pro. I plan to keep pushing the limits of my laptop and writing about my experiences.

The experiments:

• Transformer baseline: Standard transformer mostly as a comparison
• TRM with dense layers: Various recursion configurations with dense + wide layers
• TRM with sparse layers: Various recursion configurations with a sparse MoE at parameter equality
• Recursion with carry: For both dense and sparse configurations, allow a 4 iteration refinement on carry

Each model saw ~2.1M training sequences at 512 length over 31,250 steps with identical hyperparameters (lr=0.01, batch_size=64, warmup + cosine decay).

For TRM with sparse layers, I implemented the approach described in MoEUT which shows MoE models can reach parameter equal outcomes by using recursion when compared to dense models without recursion. The claims largely hold up inside the TRM harness at small scales.

There are many pieces of research including the MoEUT paper which show that 2 layers are required for transformer generalization, and the MoEUT paper found that a group size of 2 with additional recursion worked better than larger groups of layers with less recursion. I followed this and similar findings and trained all TRM models with just 2 layers and recursion. This meant wider than standard expansion for dense models, and many experts for our MoE variants.

Training Results: Simpler is Better

At 1M parameters on a language modeling task, architectural complexity doesn’t help:

Final loss rankings (best to worst):

1. Transformer baseline
2. (Tied) TRM sparse MoE with compute similar recursion
3. (Tied) TRM with dense + wide layers
4. Carry refinement variants

The transformer baseline achieved the lowest training loss though all “bests” were quite close and resulted in usable models. Deep recursion and carry refinement added overhead and additional compute without improving perplexity or generation quality compared to the baseline. Spending more compute did still improve results compared to other TRM variants, but none were able to reach baseline transformer performance.

Dense TRM Models Generate Coherent Text

Sample from a H-cycle=3, L-cycle=6 no carry refinement training run (greedy, temp=0.0):

Prompt: "Once upon a time, there was a little girl named Lily"

Output: ". She loved to play outside in the park. One day, she
saw a big, red ball on the ground. She wanted to play with it,
but it was too high for her.

Lily's mom saw her struggling and asked, "What's wrong, sweetie?"

"I can't reach the ball, but I can't reach it," her mom said.

Lily was sad, but she knew she had to help the ball. She went
to her mom and said, "I can help you, mommy. I can help you."

Her mom was happy to help and they went back to the park
together. They played on the swing and had lots of fun. When it
was time to go home, they went home and had a happy day at the
park. The end."

But don’t get too excited! Remember, the transformer baselines reach slightly lower loss at similar size on this dataset. TinyStories dataset is known to allow for reasonable generations at very low parameter sizes and all of our models and variants produce similar output.

Temperature and top-k sampling work as expected, producing more diverse outputs at higher temperatures.

Sample from same model (temp=0.5, top_k=20):

Prompt: "The dog ran very fast to"

Output: " the park. He was so excited to see the other kids. He
ran around the park and saw a big, red ball. He wanted to play
with it, but he was too small. He was too small and he couldn't
reach it. He tried to get it, but he couldn't. He was so sad.
He started to cry..."

MoEUT at 1M Parameters

I implemented MoEUT following the original paper with one modification to add loss-free bias balancing as shown in the DeepSeek paper.

• Sigmoid-based expert routing
• Loss-free bias balancing
• 8 FFN experts with k=2 active
• 2 attention experts with k=1 active

The training was stable. Loss-free bias balancing kept expert loads balanced without auxiliary losses. Compared to the dense TRM models, training loss was comparable, and generation quality was comparable. This is expected from the MoEUT paper which shows us MoE models can be made to reach parameter-equal outcomes by employing recursion as described in Universal Transformers.

Implementation note on Apple Silicon: My MoEUT implementation isn’t truly compute-equal to the dense models due to a hardware-specific constraint. On Apple Silicon (MPS backend), sparse tensor operations caused unbounded memory growth during training—likely due to memory fragmentation. To work around this, I had to compute dense projections for all experts and then filter to the top-k, rather than computing only the selected experts. This means MoEUT variants used more compute than they theoretically should. This limitation is specific to the MPS backend and likely wouldn’t occur on CUDA hardware with better sparse operation support.

The MoEUT paper demonstrated advantages at larger scales and would be an interesting path for future work, particularly on hardware that can efficiently handle sparse computations.

Carry Refinement

HRM and TRM allow the model to refine previous outputs, improving quality with more compute at inference time. In theory, this should help with difficult tokens.

In practice, at 1M parameters on TinyStories, carry refinement variants consistently lagged behind fully connected recursion in training loss. The carry refinement mechanism added complexity and compute without improving generation quality.

This doesn’t mean carry refinement is flawed—just that for simple narrative generation at small scale, the overhead isn’t justified.

Key Takeaways

What I observed:

The hierarchical reasoning mechanisms that excel at puzzles don’t automatically transfer to language modeling—at least not on simple narrative tasks at small scales. The recursion that helps TRM “think through” an ARC-AGI puzzle doesn’t provide comparable benefits when generating children’s stories.

Similar Work

There are a number of papers about reasoning in latent space or scaling up test compute dynamically. Hierarchical reasoning models appear to operate similarly. In generation of stories, it was possible to perform more or fewer refinements than models were trained with, receiving different outputs. Anecdotally, some story generations appeared to be slightly improved when running 4 carry refinement steps or additional L-cycles at inference. It’s possible I was just looking for this result, but this likely deserves additional follow-up on larger models to further study latent space reasoning on language modeling tasks.

Future Work

Several important questions remain unanswered:

Scale: Does MoEUT show advantages at 40M+ parameters where the original paper demonstrated benefits?

Task complexity: Would recursive models excel on reasoning-heavy language tasks like GSM8K or other mathematical/logical problems where deeper computation might matter?

I focused on TinyStories because it’s fast to train and well-understood. But hierarchical reasoning might shine on tasks that actually require… reasoning. That’s the obvious next experiment.

Conclusion

Hierarchical Reasoning Models can generate text. At small scales on simple tasks, they’re not better than standard transformers, but they’re not really worse either (when configured correctly). They seem quite similar to universal transformers, and are likely relevant to research into latent space reasoning. At least at the scales I tested, architectural sophistication needs task-appropriate justification. Deep recursion and carry refinement are tools, not magic. They excel where they’re needed and add overhead where they’re not.

Code and configs available at: github.com/jhspaybar/TinyRecursiveModels

Before we dive deeper, if you’re just interested in seeing the code, head over to Github and checkout out ecs_state.

While there are many cluster management systems on the market today, I’ll be focusing on EC2 Container Service(ECS) from AWS. I’m very familiar with it as I was a founding member of the team and wrote large portions of the cluster management and task placement systems. It is a “shared state” cluster management system informed by the learnings shared by Google in their Omega paper. The core concept of a system built in this manner is that the cluster manager will share what is happening in the cluster to anyone or anything that asks. In ECS, this means that anyone, or anything which calls the List and Describe APIs can inspect what is happening not just with their own resources, but with all resources available in the cluster. Armed with this information, a decision can then be made about how to modify this state, either by starting or stopping a Docker container somewhere in the cluster. A human performing these actions can be a bit time consuming, but a machine performing these actions is referred to as a “scheduler”. The scheduler reads in data about cluster state, and then stops and starts Docker containers based on a set of rules. In the case where multiple schedulers race for the same resources, the cluster manager must choose a winner and reject the other request for placement in the cluster.

The most fundamental way to place a task in an ECS cluster is to use the List and Describe APIs, apply some logic, and then call StartTask which takes as arguments a TaskDefinition(a manifest containing the Docker image to run, the resources to use, and other configuration), as well as the identifier for the machine where it should be started. By allowing for direct placement, even applications with complex requirements that may only be capable of running on a very specific machine can pick that spot through this process. Many applications however don’t need this form of control, and ECS provides us with some concrete examples of what a scheduler might look like for these applications. In ECS, there is an API called RunTask which is described in the documentation as “Start a task using random placement and the default Amazon ECS scheduler.” When calling this API we can see it returns quickly with details about the task that was placed, or provides us with an error like reason: RESOURCE:CPU meaning it could not find a location for the task because the CPU resource constraint could not be satisfied. This is currently a simple scheduler that inspects the state of the cluster, finds which machines in the cluster can accept the task placement, and then randomly places up to 10 tasks on the available machines. It’s simple, but for a quick job like a build task, or image processing, the location of placement may not matter as much as just quickly finding somewhere for the task to run.

Another scheduler that is part of the ECS APIs is the Service Scheduler. It is built to run long running tasks within the cluster, restart them if they stop(possibly if the machine they’re running on crashes), and optionally manage the lifecycle necessary to place those tasks behind an Elastic Load Balancer. It even allows for rolling zero downtime updates to your containers while properly draining connections from the load balancers. There’s really no magic involved though, and nothing is happening that couldn’t be done by a user external to ECS. When starting a service, ECS expects a role to be provided which allows for describing and registering machines behind an Elastic Load Balancer. Currently, if you start a service with a load balancer and have Cloudtrail Logs enabled, you can see the DescribeInstanceHealth and RegisterInstancesWithLoadBalancer calls being made by the ECS scheduler as tasks are started and stopped. Just as with the RunTask API, the Service Scheduler is inspecting cluster state, and then making decisions about where and when to start and stop tasks, the logic is more complex, but is still just List, Describe, Start, and Stop.

One of the benefits of this shared state model is that each of these schedulers can be developed and released independently. In previous types of cluster management systems, the state was often stored in a single location which would then add more and more specialized logic culminating in a giant ball of spaghetti that was slow to modify as new customer requests arrived. One drawback however is that this logic to inspect state and query it is now duplicated in many separate schedulers. In order to reduce this effort I’m releasing ecs_state which is a small Go library that uses the ECS List and Describe APIs to store information about running tasks and available resources in memory in sqlite. There are a set of APIs to allow control over when to refresh state, as well as an API to search for machines with the resources available to accept the task. Further logic and filtering can then be applied in memory before finally calling the StartTask or StopTask APIs. The ECS forums have seen quite a few requests for schedulers that run a task once on every machine, or run tasks at specific times like cron, I’m hoping with a little bit of a headstart that these schedulers and others will become simpler and quicker to create.