# Lambda expressions

## Overview

DFS uses lambda expressions to define transformations on data points as well as define Boolean conditions that act as filters. Lambda expressions wrap mathematical operations, Boolean operations, internal function calls or a combination of all three.

All lambda expressions in DFS begin with the `lambda:` keyword.

``````|where(lambda: "topic" == 'ttop/grap/prec')
``````

In the above example `"topic"` is used to access the value of a field called `topic` from the current data_point and compared against the string `'ttop/grap/prec'`.

Note here that literal string values are declared using single quotes, while double quotes are used to access the values of tags and fields within the current data_item.

## Field paths

As field and tag values can be deeply nested maps and lists, it is possible to use a `JSON-path` like syntax to reference them:

Valid examples:

``````"averages"

"axis.z.cur"

"value.sub.data"

"averages.emitted"
``````

## Built-in functions

### Type conversion

With a few exceptions every type can be converted to every other type.

Bool

``````bool(a_value) -> true|false
``````

Integer

``````int(value) -> integer
``````

Float

``````float(value) -> float
``````

String

``````string(val) -> string
``````

### Time functions

Every data_point in faxe contains a field called `ts`, which holds a UTC timestamp in milliseconds.

Function Description
`now()` -> integer returns an utc timestamp in milliseconds
`dt_parse(ts, formatstring)` -> integer used to parse a datetime string to the internal format, see datetime-parsing for details
`to_iso8601(ts)` -> string converts the timestamp to an ISO8601 datetime string
`to_rfc3339(ts)` -> string converts the timestamp to an RFC3339 datetime string
`millisecond(ts)` -> integer milliseconds within the second [0, 999]
`second(ts)` -> integer second within the minute [0, 59]
`minute(ts)` -> integer minute within the hour [0, 59]
`hour(ts)` -> integer hour within the day [0, 23]
`day(ts)` -> integer day within the month [1, 31]
`day_of_week(ts)` -> integer the weekday with week [1, 7] 1 is monday
`week(ts)` -> integer isoweek-number within year [1, 53]
`month(ts)` -> integer month within the year [1, 12]
`time_align(ts, duration)` -> integer align the given timestamp to duration

Examples:

``````lambda: hour("ts") >= 8 AND hour("ts") < 19
``````

The above expression evaluates to true if the hour of the day for the data point falls between 08:00 and 19:00.

``````lambda: time_align("ts", 3m)
``````

Will align every timestamp (ts) to a multiple of the 3 minutes within an hour (ie: 00:00, 00:03, 00:06, ...)

### Math functions

Function Description
`abs(x)` -> number
`acos(x)` -> float
`acosh(x)` -> float
`asin(x)` -> float
`asinh(x)` -> float
`atan(x)` -> float
`atan2(y, x)` -> float
`atanh(x)` -> float
`ceil(x)` -> float
`cos(x)` -> float
`cosh(x)` -> float
`exp(x)` -> float
`floor(x)` -> float
`fmod(x, y)` -> float
`log(x)` -> float
`log10(x)` -> float
`log2(x)` -> float
`max(x, y)` -> number
`max(list)` -> number
`min(x, y)` -> number
`min(list)` -> number
`pi() -> float` gives pi
`pow(x, y)` -> float
`round(x)` -> integer round a number to an integer
`round_float(x, precision)` -> float round a float (x) with the given precision
`sin(x)` -> float
`sinh(x)` -> float
`sqrt(x)` -> float
`tan(x)` -> float
`tanh(x)` -> float

### String functions

Function Description
`str_at(x, pos)` -> string/undefined Returns the grapheme in the position of the given utf8 string. If position is greater than string length, then it returns undefined. Negative offsets count back from the end of the string.
`str_capitalize(x)` -> string Converts the first character in the given string to uppercase and the remaining to lowercase
`str_contains(x, contents)` -> bool Check if string contains any of the given contents
`str_downcase(x)` -> string Convert all characters on the given string to lowercase
`str_ends_with(x, suffix)` -> string Returns true if string ends with suffix, otherwise false.
`str_ends_with_any(x, suffixes)` -> string Returns true if string ends with any of the suffixes given, otherwise false.
`str_eqi(x,y)` -> bool Compares strings case insensitively
`str_first(x)` -> string/undefined Returns the first grapheme from an utf8 string, undefined if the string is empty
`str_last(x)` -> string/undefined Returns the last grapheme from an utf8 string, undefined if the string is empty
`str_length(x)` -> int Returns the number of unicode graphemes in an utf8 string
`str_lstrip(x)` -> string Returns a string where leading Unicode whitespace has been removed
`str_lstrip(x, char)` -> string Returns a string where leading char have been removed
str_normalize/2
`str_replace(x, patt, repl)` -> string Returns a new string based on subject by replacing the parts matching pattern by replacement.
str_replace_leading/3 Replaces all leading occurrences of match by replacement of match in string.
str_replace_trailing/3 Replaces all trailing occurrences of match by replacement of match in string.
`str_replace_prefix(x, match, repl)` -> string Replaces prefix in string by replacement if it matches match. Returns the string untouched if there is no match. If match is an empty string (""), replacement is just prepended to string.
`str_replace_suffix(x, match, repl)` -> string Replaces suffix in string by replacement if it matches match. Returns the string untouched if there is no match. If match is an empty string (""), replacement is just appended to string.
`str_reverse(x)` -> string Reverses the given string.
`str_rstrip(x)` -> string Returns a string where trailing Unicode whitespace has been removed
`str_rstrip(x, char)` -> string Returns a string where trailing char have been removed
`str_slice(x, start, len)` -> string Returns a substring starting at the offset given by the first, and a length given by the second param, if offset is negative, count back from end of string.
str_split/1
str_split/2
str_split/3
str_split_at/2
str_split_by_any/2
str_split_by_any/3
str_split_by_re/2
str_split_by_re/3
`str_starts_with(x, pre)` -> bool Returns true if string starts with Prefix
`str_starts_with_any(x, prefixes)` -> bool Returns true if string starts with any of the prefixes given, otherwise false.
`str_strip(x)` -> string Returns a string where leading/trailing Unicode whitespace has been removed
`str_strip(x, char)` -> string Returns a string where leading/trailing char have been removed
`str_upcase(x)` -> string Convert all characters on the given string to uppercase

### Misc

Function Description Example
`defined(Key)` -> bool whether the given Key is defined in the current data-item
`undefined(Key)` -> bool whether the given Key is NOT defined in the current data-item
`topic_part(TopicString, PartIndex, [Seperator])` -> string extract a part from a topic string, Separator defaults to '/', the index of the first part is `1` not 0 topic_part('this/is/mytopic', 2) -> 'is'
`from_json_string(String)` -> faxe map or list convert a valid json string into faxe's internal data-structure from_json_string('{"a":"b"}')
`to_json_string(String)` -> string convert faxe's internal data-structure into a json string to_json_string("data.struct")
`size(ListOrMap)` -> integer get the number of entries in a list or map size("data.lines")
`member(Ele, ListOrMap)` -> bool check for list/set membership of a value, or when used with a map, check if Ele is a key in the map
`not_member(Ele, ListOrMap)` -> bool
`map_get(Key, Map)` -> any get a value from a map, 'undefined' is returned, if the key is not present in map map_get("topic", ls_mem('stream_lookup'))
`random(N)` -> integer generate a random integer between 1 and N
`random_real(N)` -> float generate a random float between 0.0 and 1.0, that gets multiplied by N
`mem` values are set with the mem node
`ls_mem(Key)` -> any get the single value associated with Key from the flow-memory
`ls_mem_list(Key)` -> any get the list value associated with Key from the flow-memory
`ls_mem_set(Key)` -> any get the set value associated with Key from the flow-memory

### Conditional functions

If

Returns the result of its operands depending on the value of the first argument. The second and third arguments must return the same type.

Example:

``````|eval(lambda: if("field.val1" > threshold AND "field.val1" != 0, 'true', 'false'))
.as('value')
``````

The value of the field `value` in the above example will be the string `true` or `false`, depending on the condition passed as the first argument.

The if function’s return type is the same type as its second and third arguments.

``````if(condition, true expression, false expression)
``````
{"ft_lang_label":"__label__en","ft_lang_prob":0.61635554,"math_prob":0.9542615,"size":8648,"snap":"2022-27-2022-33","text_gpt3_token_len":2288,"char_repetition_ratio":0.17480333,"word_repetition_ratio":0.11843079,"special_character_ratio":0.27382055,"punctuation_ratio":0.092203386,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.97769207,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-08-11T02:19:29Z\",\"WARC-Record-ID\":\"<urn:uuid:ff8ea4d7-268a-44ec-bcc5-f2dd5842b66b>\",\"Content-Length\":\"38746\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:02586e1a-97ea-4b3e-8b22-594a92c8910b>\",\"WARC-Concurrent-To\":\"<urn:uuid:4fa6a789-3278-488d-9fc1-4620d51152f5>\",\"WARC-IP-Address\":\"104.17.33.82\",\"WARC-Target-URI\":\"https://faxe-docs.readthedocs.io/en/latest/dfs_script_language/lambda_expressions.html\",\"WARC-Payload-Digest\":\"sha1:M5KPNKKC4Q75SGKBNYWQAXVVPLKCZORG\",\"WARC-Block-Digest\":\"sha1:LPJKYYYP7OKKCTYW632QRF5IR7FSNYQG\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-33/CC-MAIN-2022-33_segments_1659882571232.43_warc_CC-MAIN-20220811012302-20220811042302-00592.warc.gz\"}"}