Virtual Fields

Virtual Fields allow you to derive new values from your data in real time

One of the most powerful features of Axiom are Virtual Fields. With Virtual Fields, there is no need to do any up-front planning of how to structure or transform your data. Instead, send your data as-is and then use Virtual Fields to manipulate your data in real-time during queries.

The feature is also known has "derived columns/fields", however Axiom's Virtual Fields have some unique properties that make them much more powerful.

In this guide, you'll be introduced to Virtual Fields, their features, how to manage them, and how to get the best out of them.

Creating a Virtual Field

Once a dataset is selected in either the Analytics or Stream views, clicking on the virtual fields icon in the toolbar will open the virtual field slide-out that displays all the virtual fields for a dataset:

virtual fields tool buttonvirtual fields slideout

Click "Add Virtual Field" to open the virtual field editor dialog.

virtual fields editor

The editor dialog has the following components:

  • Name & Description - help your team understand what the virtual field is doing
  • Expression - this the expression that is run on every event to derive the virtual field. The expression would produce some output (from a boolean check like the example, to strings, numbers, objects or more)
  • Preview - use the preview window to test your virtual field expression against live data

The power of virtual fields is in letting you manipulate data 'on read' instead of 'on write', allowing you and your team members to adjust and update virtual fields over time as well as easily add new ones without worrying that the data has already been indexed.

Usage

Visualizations

Virtual Fields will be available as parameters to visualizations but, as the type of a virtual field can be any of the supported types, it is important to make sure that you are using a virtual field that produces the correct type of argument.

Filters

Virtual Fields will be available in the filter menu and all filter options will be presented. It is important to ensure that you are using a supported filter operation for the type of result your virtual field produces.

Group By

Virtual Fields can be used for segmentation in the same way as any standard field.

Reference

Virtual Fields use a rich expression language that is easy to grasp but powerful in use. This section documents the language, and the functions available.

In the 'reference' section of the Virtual Fields, just point to the APL Scaler function docs, as we don't want to duplicate everything

To make things easier, Virtual Fields are just APL expressions (e.g. extend alias1 = expression1, alias2 = expression2, alias3 = ... ) , and therefore Virtual Fields share all the same functions and syntax as APL expressions.

You can find a list of APL functions below:


Literals

FunctionsDescription
stringssingle and double quotes are supported.
numbers101, 101.1
booleanstrue and false
arrays["one", "two", "three"]
maps`{ region: "us-east-1" }
nil -nil

Arithmetic Operators

OperatorDescription
+addition
-subtraction
*multiplication
/division
%modulus
**pow

Comparison Operators

OperatorDescription
==equal
!=not equal
<less than
>greater than
<=less than or equal to
>=greater than or equal to

Logical Operators

Operator
and or &&
or or ``
not or !
success ? 'yes' : 'no' - ternary

String Operators

OperatorDescription
+concatenation
matchesregex match
containsstring contains
startsWithhas prefix
endsWithhas suffix

Info
To test the negative case of not matching, wrap the operator in a `not()` operator:
`not ("us-east-1" contains "us")`
You must use parentesis because the unary operator `not` has precedence over the binary operator `contains`.

Numeric Operators

In addition to the arithmetic operators:

  • .. - numeric range
Example
`age in 18..45`
Tip
The range is inclusive: `1..3 == [1, 2, 3]`

Membership Operators

OperatorDescription
incontains
not indoes not contain
Example
Arrays: metadata.region in ["us-east-1", "us-east-2"]
Maps: 'region' in { region: 'us-east-1 } // true

Builtins

OperatorDescription
lenlength of an array, map, or string
allwill return true if all element satisfies the predicate
nonewill return true if all element does NOT satisfies the predicate
anywill return true if any element satisfies the predicate
onewill return true if exactly ONE element satisfies the predicate
filterfilter array by the predicate
mapmap all items with the closure
countreturns number of elements what satisfies the predicate
Ensure all comments are less than 280 chars
all(comments, {.Size < 280})
Ensure there is exactly one private repo
one(repos, {.private})

Closures

  • {...} - closure

Closures allowed only with builtin functions. To access the current item, used the # symbol.

Example
`map(0..9, {# / 2})`

If the item of array is struct, it's possible to access fields of struct with omitted # symbol (#.Value becomes .Value).

Example
filter(comments, {len(.body) > 280})

Slices

  • myArray[:] - slice

Slices can work with arrays or strings

Example
The variable `myArray` is `[1, 2, 3, 4, 5]`
`myArray[1:5] == [2, 3, 4] myArray[3:] == [4, 5] myArray[:4] == [1, 2, 3] myArray[:] == myArray`