Snapshot does not ignore previous events

Hello,

I’m trying to make a “cut” in the event queue to improve the loading time at startup.I want to make a semantic snapshot, which uses the last State of the Fish as the new state after the reset event has been received. I followed the guide: guides - snapshots

For this i gave the fish a new version:

    export const MyFish= (id: string): Fish<State, Event> => ({
        fishId: FishId.of('ax.myfish', id, 3), 
        initialState: initialState(id),
        onEvent,
        where: myTags.single(id),
        deserializeState: (snapshot: unknown) => { 
            console.debug('Deserializing snapshot of size ' + snapshotSize(snapshot))
            return snapshot as State
        },
        isReset: (event) => event.type === EventType.snapshot,
  })

The handler of the snapshot events looks like this:

  export const SnapshotHandler = (_state: State, event: SnapshotEvent): State => {
      return event.state
  }

For each fish i did observe the last state and then emitted an event with it:

pond.emit(myTags.single(id), { type: EventType.snapshot, state: lastState })

This did not stop the fishes from running through all previous events when woken up. Also the snapshot sizes in MB did not change at all.
Am i missing something here?

best regards
Sebastian

Hello @SebastianS,

That the size of the snapshot does not change feels ok. Currently, it feels, that you kind of double the snapshot mechanism. Both are running in parallel and the original mechanism should not behave any way differently.

Of course, you can define the rate of the snapshots now and you can distribute the snapshots between the peers (what is really nice) but the single note will not get faster anyhow.

As soon you have a snapshot-Event the isReset looks correct. But It could be, that you got caught from the build-in snapshot mechanism first.

What confuses me a lot is the fact that you changed the version number and your snapshot event did not set your state.

I will quickly build a fish and test it on my PC. I’ll come back to you soon

BTW: think about a version in your snapshot events. Probably you’ll come to a situation where you like to increase it

Hi Sebastian!

One important thing to keep in mind with semantic snapshots is that they really must mean “this is the state at this point of the event log”. If you take the current fish state on one node and persist it as an event that matches isReset, then any outstanding time travel from past events that haven’t reached this node yet will be silently discarded — your fish will “lose” events in the sense that they don’t get applied to the state. This is why our built-in state snapshot mechanism works differently, not via events; state snapshots will be invalidated and discarded when time travel would “go past them”, i.e. affect earlier states.

Not sure if this answers your question, but the above is an important consideration. Please let me know if it is not fully clear.

Regards,

Roland

Thank you for your answer it was helpful to further understand snapshots.
I am aware of that i could potentialy lose events when my executing node has not yet received all events from every other node. Therefor i would only create such a semantic snapshot when i’m sure that all nodes are in sync.

But on my development machine i am the only node at the moment and there should happen no time travel in the first place. In the current state i only emit the reset events. So there are no events after the semantic snapshot should have happened. And therefor the state snapshot should not overwrite the semantic snapshot because there are no incoming events refering to older states.
Please correct me if i missunderstood anything, i’d appreciate it.

Regards
Sebastian

Okay cool, I just wanted to make sure you know the tradeoff!

Regarding the state snapshot: that mechanism is completely transparent, provided you increase the Fish version whenever required. So the state snapshot shouldn’t overwrite anything, it is just an optimisation.

When the last (i.e. youngest) event is a semantic snapshot, you should see onEvent invoked only once, with initial state and that one event — do you see it being called multiple times?

Yes after the semantic snapshot event, the onEvent handler is called as many times as before the event.

To solve this topic, the isReset is only disabling the time-travel when inserted events are appearing before the isReset-event/condition

The normal snapshot did not take place here, because each entity fish did not consume more than 1024 events. (See update)

When you like to do snapshot events, please be aware, that you creating an event from a state. You don’t know if it misses some information from other peers.
We are currently working on a couple of ideas on how to improve here.

Thanks for bringing this up.

Update:
Since pond version 2.6, you are able to define your snapshotThreshold yourself. You can define it application-wide in the PondOptions or individually in your fish.