Improve performance of Error.inLastDays fish by applying tags (dynamically)

Hi,

I´m trying to improve the performance of my Error.inLastDays fish but I don’t know whether my solution is a good one.

The fish should give me the errors that occured in the last n days. I filtered the events by timestamp before in the onEvent function. When I do so, to many events are passed to the onEvent function and it takes a long time to calculate the state.

As discussed with @wwerner before I can use a tag to improve the performance.
So I created a date tag:

const dateTag = Tag<Event>('Date')

As id I would like to pass the date in this format:

dateTag.withId("2021-07-30")

The challenge is to apply the date tag with id to my fish’s where dynamically because i want to support multiple days.

I have written the follwing function to create date tags dynamically by passing the number of days as parameter:

const appendDateTagsOfLastDays = (baseTag: Tag<Event>, days: number) => {
  let where: Where<Event> | null = null
  for (let i = 0; i < days; i++) {
    const date = sub(Date.now(), { days: i })
    const dateTagId = format(date, 'yyyy-MM-dd')
    if (!where) {
      where = baseTag.and(dateTag.withId(dateTagId))
    } else {
      where = where.or(baseTag.and(dateTag.withId(dateTagId)))
    }
  }
  return where as Where<Event>;
}

Im using this in the inLastDays fish´s where:

const errorTag = Tag<Event>('Error')


inLastDays: (days: number): Fish<InLastDaysState, Event> => ({
    fishId: FishId.of('Errors.inLastDays', `${days}`, 0),
    initialState: {
      errors: [],
      visibleDays: days,
    },
    where: appendDateTagsOfLastDays(errorTag, days),
    onEvent: (state, event, metadata) => {
      if (event.eventType === 'error_occured') {
        state.errors.push({
          errorId: event.errorId,
          errorTS: metadata.timestampMicros,
        })
      }

      const thresholdUS = Date.now() - days * 24 * 60 * 60 * 1000
      state.errors = state.errors.filter((e) => e.errorTS > thresholdUS * 1000)
      return state
    },
  })

where.toString() outputs:
“‘Error’ & ‘Date’ & ‘Date:2021-07-30’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-29’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-28’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-27’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-26’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-25’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-24’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-23’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-22’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-21’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-20’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-19’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-18’ | ‘Error’ & ‘Date’ & ‘Date:2021-07-17’”

My Question is now is this a recommended way to do that or is there an easier or better solution?

TL;DR: Actyx Query Language (AQL) allows for building powerful event queries. It supports time-based filtering. Use Actyx v2 and AQL for this case.


Hey @jokr,

thanks for posting We’re well aware that filtering events based on their emission time has been quirky to implement so far. So we’ve come up with a better way.

As to your approach:
The more specific Tags you’re able to use in your twin’s subscription (where), the fewer events onEvent has to process, so that’s why you can ‘improve performance’. Your approach is generally valid. You could use just the week number and the year to have less tags to deal with, in case weekly resolution is fine grained enough for you.

However, since we’ve released the Actyx Query Language AQL, there now is much more straightforward way to go about it. It’s currently not supported within twins, but you can use the events API directly using pond.events().

So, given you’re on Actyx v2 and use the most current version of the Pond you can use AQL to query for the tags you need, restrict the result to a time range and bind it to your React UI as shown in the example below.

const Errors = () => {
  // Use the pond
  const pond = usePond()
  // This state will hold our list of events
  const [events, setEvents] = useState<AqlEventMessage[]>([])
  // Perform an AQL query and set the state
  useEffect(() => {
    pond
      .events()
      .queryAql("FEATURES(timeRange) FROM 'a' & from(2021-07-20T13:30:48Z)")
      .then((events) => {
        // The query will return different types of objects (events, diagnostics,
        // and offset maps). In this case all we want are the events so we filter
        // out anything that is not an event using the `type` property
        setEvents(events.filter((e): e is AqlEventMessage => e.type === 'event'))
      })
  }, [])
  return (
    <ul>
      {events.map((event) => (
        <li key={event.meta.eventId}>{JSON.stringify(event.payload)}</li>
      ))}
    </ul>
  )
}

But there’s more: You can play around with your queries interactively using the Node Manager as described in Query Events using the Node Manager | Actyx Developer. Further details on AQL’s capabilities are described in the reference documentation.
AQL is continuously being expanded upon. If you want to stay in the loop, discuss upcoming features, and share your requirements, have a look at the Actyx Insiders topic in the forum. We’d love to hear what you think and how we can make it even more useful for you.

We’d strongly recommend migrating to v2 and using AQL for these use cases.
If you need additional support with that or need to stick to using twins for that, please hop over to our Discord channel to discuss more deeply.

Hope this helps!