Collect Device Attributes

Task Id

Save Run Reset

On Page Load

Run Task Id

Literal Constant Query String Param

Save Run Reset

Visual Field

  Drop your CSV File here to Import and Browse

Explore, Process and Visualize your data right in your browser.
visualfield.org

Import File

  Click or drop your delimited File here

Preview

Charset:

Separator Comma , Semicolon ; Pipe | Space Tab \t Custom     Custom

Delimiter Double Quote " Single Quote ' Custom     Custom

Skip first lines.    Header row     Empty Fields as Null

Destination Table Temp

After Import, start Task Id   Automate Import next time

Import Import + Browse Save (No Import) List Reset

Import URL

URL

Preview Fetch (preview only)

Charset:

Import Import + Browse Save (No Import) List Reset

Device Attributes and On Page Load are included in Config settings

Save Reset

Enter SQL Statement

Define Run Sequence

Define Custom Function

Define Prompt Input

Define Run Menu

Browse

Table Visualization

Save List Reset

Chart Visualization

Save List Reset

Map Visualization

Save List Reset

Media Visualization (Slide Show)

Save List Reset

Offline Map Tile Store

Current Map Zoom Level:

Current Tiles in Offline Store:

Delete Tile Store Close

Export Metadata

Import Metadata

Reset Metadata

Reset

Reference Guide

Home (Menu)

If you have an RFC 4180 standards based CSV file with a single header row, just drag and drop it on to the file input area on this pane. Your CSV will be parsed, loaded into a table having the same name as the filename, and then switched to the Browse pane so you can readily view the contents.

---

Table (Import File)

This pane enables you to import a delimited file into the Visual Field WebSQL database. To start, just drag and drop your file onto the file input area and a preview will be displayed. A destination table will need to be nominated (which will default to the deduplicated filename). If your delimited file is in RFC 4180 CSV standards format and contains a single header row then you should not need to change any of the other setting and can proceed to import your file by using the Import buttons at the bottom of the pane. In the case you need to change settings for a different formatted file, these are described as follows;

Charset

Unused in this version - will always be UTF-8

Separator

Specify the file field separator from the select list. If the separator used in your file is not shown in the select list, then choose "Custom" and enter the separator character in the Custom input field.

Delimiter

Specify the file field escape delimiter from the Select list. If the delimiter used in your file is not shown in the select list, then choose "Custom" and enter the delimiter character in the Custom input field.

Skip first lines

If the delimited content in your file does not begin on the first line, then enter the number of lines to skip before parsing your data. Please note that leading empty lines will automatically be omitted during file read.

Header Row

Indicates that the file contains a header row that lists the column names. If this is unchecked, column names will be assigned as; F1, F2, F3, etc.

Empty Fields as Null

With this checked, empty fields encountered in the file will be assigned null when inserted into the database table. If this is unchecked an empty string will be inserted.

Destination Table

Enter the table name to be created in the Visual Field database for this import. Will default to the deduplicated filename.

Temp

If this is selected, the table will be created as "TEMP" - meaning that it wont persist after a subsequent page reload (opening of the database).

After Import Start Task Id

Various actions within Visual Field can have an associated Task Id. Enter the Task Id, that you wish to start automatically after you file loads here. Eg, you could put an SQL, Custom Function or Visualization task immediately after your file is loaded.

Automate Import Next Time

With this checked, the next time, you drop a file with the same filename, the import will run automatically without further action required.

Import

Press this to commence the parse and import of your file. A database table will be (re)created by this import.

Import + Browse

Press this to commence the parse and import of your file. Once your file is loaded, the Browse pane for the corresponding table will be activated so you can immediately see the contents

Save (No Import)

Use this to save the configuration specified above but do not actually run an import. Useful if you need to change something about the import.

List

Will give a mini list with actions in this pane. Limited to 100 rows maximum - if you have more than this number of tables, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the import configuration of the particular table/filename

Drop

Drop the import configuration of this particular table/filename. However, the actual database table, if it exists, will not be dropped.

Reset

Will clear this form.

---

Table (Import URL)

This pane enables you to import a web service that is serving delimited content into a Visual Field WebSQL database table. If you are using the file:// protocol to use Visual Field, or if the web service is being served from a different domain, the web service may need to include CORS headers or the import fetch request may fail. A destination table will need to be nominated. If the web resource is delimited in RFC 4180 CSV standards format and contains a single header row then you should not need to change any of the other setting and can proceed to import once a destination table is specified. In the case you need to change settings for a different formatted resource, these are described as follows;

Charset

Unused in this version - will always be UTF-8

Separator

Specify the file field separator from the select list. If the separator used in your file is not shown in the select list, then choose "Custom" and enter the separator character in the Custom input field.

Delimiter

Specify the file field escape delimiter from the Select list. If the delimiter used in your file is not shown in the select list, then choose "Custom" and enter the delimiter character in the Custom input field.

Skip first lines

If the delimited content in your file does not begin on the first line, then enter the number of lines to skip before parsing your data. Please note that leading empty lines will automatically be omitted during file read.

Header Row

Indicates that the file contains a header row that lists the column names. If this is unchecked, column names will be assigned as; F1, F2, F3, etc.

Empty Fields as Null

With this checked, empty fields encountered in the file will be assigned null when inserted into the database table. If this is unchecked an empty string will be inserted.

Destination Table

Enter the table name to be created in the Visual Field database for this import. Will default to the deduplicated filename.

Temp

If this is selected, the table will be created as "TEMP" - meaning that it wont persist after a subsequent page reload (opening of the database).

Save as Task Id

This will associate a Task Id with this particular import action. The task can then be invoked at a later point to automate the ingestion of the web service.

Cache Max-Age

If this import has a Task Id and that if the import is being invoked as part of that task, then if this number of seconds has not yet elapsed since the previous import of this service, then this task will be skipped. That is, you can specify the number of seconds to elapse before this resource should be re-imported when invoking this action as a task. Will default to zero meaning always import. If this import is being run manually by one of the buttons below, then this setting is ignored.

URL Header Name & Value

You can optionally specify upto 2 http headers to attach as part of the URL request.

Fetch (preview only)

This will attempt to retrieve the webservice URL nominated but not import it.

Import

Press this to commence the fetch, parse and import of your web resource. A database table will be (re)created by this import.

Import + Browse

Press this to commence the fetch, parse and import of your web resource. Once your file is loaded, the Browse pane for the corresponding table will be activated so you can immediately see the contents

Save (No Import)

Use this to save the configuration specified above but do not actually run an import. Useful if you need to change something about the import.

List

Will give a mini list with actions in this pane. Limited to 100 rows maximum - if you have more than this number of tables, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the import configuration of the particular table/filename

Run

Will perform a fetch, parse, Import and switch to the Browse Pane once loaded

Drop

Drop the import configuration of this particular resource. However, the actual database table, if it exists, will not be dropped.

Reset

Will clear this form.

---

SQL Statement

Use this pane to execute an arbitrary SQL statement in the Visual Field WebSQL database. This could be either a select statement or a DML statement against your own created objects. Some care should be taken in that your database will also house the Visual Field dictionary which consists of a series of tables that begin with VF_ - and you should generally not apply DML against these dictionary tables (although there are some specific exceptions to this guideline - see General Help).

Enter SQL Statement

Enter your SQL statement in the textarea.

Statement Description

If you choose to save your statement, you will need to specify an arbitrary description.

Task Id

A Task id can be associated with your saved SQL Statement so that it can be invoked as part of a Task.

Execute

Press this button to execute your SQL Statement, results will appear below.

Save (No Import)

Will save your SQL Statement

List

Will give a mini list of your saved SQL Statements. Limited to 100 rows maximum - if you have more than this number of saved SQL statements, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved SQL statement

Run

Will run the saved statement

Drop

Will drop the saved statement.

Reset

Will clear this form.

Will allow you to download the SQL statement result set as CSV.

---

Custom Function

Use this pane to specify Custom Function definitions. Custom Functions are a series of some predefined javascript functions that extend what is available within native WebSQL. Custom Functions operate against a WebSQL table and apply the javascript function specified using parameter characteristics (mappings). A Custom Function will drop and recreate a result table of the function results. For all Custom Functions, the structure of the result table is the same - which means that Custom Function outputs can be automated (as part of a task) and can either be linked (joined) back to the source table or used for further processing (as a task). This is because the structure of the output table is predictable. Custom Functions have a function Id that is associated with them and an arbitrary "library" that is a general grouping (described below). If you are a web developer, the Custom Function metadata is extensible - please search the source of this page for "// Example of adding your own Custom Function definition" approximately line 14,502 for instructions.

Input Table

Specify the input table the Custom Function will be run against.

Function

Select the function that you wish to apply. Functions are broken down by a "library" which is just an arbitrary grouping. Please refer to the API section for further details of each specific Custom Functions.

Library

Math

A collection of javascript Math functions

Stats

A small collection of mathematical statistical functions

Misc

A collection of functions written by the author that do not largely fall within another library. This may contain large parts of functions from other libraries (such as Turf and Metaphone) but have been substantially reworked to function within Visual Field.

Turf

A collection of spatial analytical functions from the Turf.js library

As mentioned, when a custom function runs it will drop and recreate the result table which is of a predefined structure. This structure is;

SRC_TABLE

This will be populated with the source input table used for the function.

SRC_ROWID

If there is a direct mapping of the custom function row back to a corresponding row in the SRC_TABLE, then this field will be populated with the ROWID of the SRC_TABLE. This gives a definitive way to link the output of Custom Function results back to the source input table. Not all functions produce a result set that is directly mappable. And some functions produce grouping results where these rowid's may be repeated.

GROUPING_RESULT

This value is a secondary result value. It is either populated when the Custom Function is some sort of aggregation function or the Custom Function produces a secondary result. In the case of an aggregation value, this can typically be used with repeating ROWID's to aggregate the results back to the original table. In the case it is used for secondary results, this will be specific to the particular Custom Function.

RESULT_STATUS

Either 'true' or 'false'. For each row, this is the result status of the execution of the javascript Custom Function. If the function completes successfully (normally) then this will be set to 'true'. If the function threw an exception, then this will be set to 'false'

RESULT

This is the actual output of the invoked Custom Function. This is a normally a scalar result and may or may not map back to the source input table by way of SRC_ROWID.

ERR_MESSAGE

If the Custom Function threw an exception, this is populated with any associated error message

Custom Functions operate against a single input table (and not joined tables). If you need to apply a custom function using input data from 2 or more tables, the source tables will need to be joined prior to passing to the joined table to the Custom Function. Please refer to an example in General Help guide. When joining input tables, typically a cross join or Cartesian Product is to be generated to be passed to the Custom Function. In order to minimise the number of input rows in the join table, it is suggested to apply any additional relational filtering predicates as possible prior to joining tables. Relating the results of Custom Function output back to the source input tables can then still be achieved via the SRC_ROWID values from the Result table - but in the case of joined table input may involve a select statement of multiple tables to achieve this. Initially, using the ROWID's in this way may seem awkward until you get understand how to easily apply this. Because ROWID's are used in this way, the inputs to Custom Functions must be a table and not a database view.

Some Custom Functions require an array of input values. For such functions, arrays can be passed as a delimited list of values. Such functions will typically accept a list parameter and a list separator and delimiter parameter. This is how arrays can be passed to Custom Functions. Generating delimited lists from SQL, for input to Custom Functions, might seem difficult at first but thankfully WebSQL (sqlite) provides the fantastic group_concat function that can make the generation of such lists very simple. Please refer to the General Help section for some examples of using group_concat.

The fields on the Define Custom Function pane are;

Input Table

Nominate the source input table here.

Function

Select the Custom Function to be applied.
Next
Press this to retrieve the Custom Function Parameters and nominate the Parameter mapping values. For the parameter type - "Literal Constant" will pass a constant value, "Column Value" will pass the table column and "Query String Param" will pass the nominated Query String Parameter. Note in the case of "Query String Param" only 1 query string value for a given parameter name will be passed (ie, if your query string parameter is an array (eg; ?A=1&A=2) then only 1 single value (the first) will be passed to the function.

Result Table

Specify the Result table to be created when this Custom Function runs.

Temp

Select if the result table should be created as a TEMP table.

Description

Enter a description for this Custom Function.

Task Id

Enter an optional Task Id that will be associated with this Custom Function so it can be run as part of a task.

Run

This will execute your Custom Function

Save

Will save your Custom Function Definition

List

Will give a mini list of your saved Custom Functions. Limited to 100 rows maximum - if you have more than this number of functions, use the Browse functions. In the mini list, the actions that can be undertaken are;

Browse

Browse the output table of a Custom Function

Edit

Edit the saved Custom Function

Run

Will run the saved Custom Function

Drop

Will drop the saved Custom Function

Reset

Will clear this form.

---

Table Visualization

A table visualization is a tabular presentation of a nominated table or view. The actual visualization produced will be based on the table/view, have a filter input, and paginated tabular output. The fields in this pane are used to define a Table Visualization.

Table/View

Specify the Table or View for the visualization.

Description

Specify a description to be associated with this visualization.

Title

This is the title heading that will be displayed at the top of the page when this table visualization runs. Titles can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself. For the case of "Column Value", the value of the first row of the table column will be used.

Task Id

A Task Id to be associated with this table visualization

Link Target

If any presented data in the table/view contains valid hyperlinks, then this value will be passed to the "target" attribute of the generated hyperlink. For example, this may be useful if you are invoking Visual Field from within an iframe and require linking to a parent page.

Self Drilldown

Self Drilldown is a mechanism that will enable you to create a hyperlink from your presented data that will link back to the Visual Field page (ie, vf.html). This is provided to enable the creation of mini data driven applications.

Drilldown Column (Query String)

Nominate a column in the Table/View that has a Query String value that can be used for the self drilldown. The Drilldown Column will be passed as a Query String to the vf.html invocation. Note that this needs to be in the form of a valid web GET Query String name value pairs. Eg; param1=value1&param2=value2. Query Strings may also need to be encoded if the parameter name or value contains spaces for example. Please refer to the Misc:encodeURIComponent Custom Function as a possible mechanism that can be used to assist the encoding of such Query Strings (that could be called as a prior task to creating these links). The Query String value should NOT include a leading Question Mark.

Drilldown Column Replacement Label

This is just a constant string value that will be used in replacement of the Drilldown Column itself (name/value pair) and used as the hyperlink anchor text.

Append Original Metadata

If this page was invoked with a passed METADATA parameter then by selecting this checkbox, that original METADATA parameter will be attached to the URL string in the Drilldown Column. Eg; name1=value1&name2=value2&METADATA=... Note that it may not be essential to reattach the METADATA - as you application could potentially rely on existing saved offline state data. However, if you intend to provide hyperlinks that do not rely on any saved offline state, then as a general guideline, the METADATA (which defines your mini-app) parameter should be appended to your hyperlink.

Save

Will save your Table Visualization Definition

List

Will give a mini list of your saved Table Visualizations. Limited to 100 rows maximum - if you have more than this number of Table Visualizations, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Table Visualization

Run

Will run the saved Table Visualization

Drop

Will drop the saved Table Visualization

Reset

Will clear this form.

Will exit from a Table Visualization and return to the Visual Field Home (Menu) pane

Will toggle between Fullscreen mode for this visualization

Will toggle filtering of individual table/view column values. Note that when enabled, filter values must be exact column value matches, whereas Search Input is a wildcard search. When the filter is enabled this button is displayed in blue; = Filter is Enabled

Will allow you to download the Table Visualization data.

Usage Notes

1. If a Table Visualization is used as a Task in a Sequence, the sequence will be terminated as part of this Task (even if there are other tasks defined after this task in the Sequence).
2. Columns used in the definition of the Table Visualization - eg, Title or Drilldown Column will be omitted from the Table Visualization when it runs.

---

Chart Visualization

A Chart Visualization is a Pie, Bar or Line chart presentation of a nominated table or view. The fields in this pane are used to define a Chart Visualization. A chart of upto 4 data series can be generated.

Table/View

Specify the Table or View as the input source for the visualization.

Description

Specify a description to be associated with this visualization.

Chart Background Color

Specify a background color for the chart.

Title

This is the title heading that will be displayed at the top of the page when this chart visualization runs. Titles can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself. For the case of "Column Value", the value of the first row of the table column will be used.

Task Id

A Task Id to be associated with this Chart Visualization

Link Target

If Drilldown Columns are specified in the chart definition, then upon invocation of such links, the "target" attribute of the generated hyperlink will be set to this value. For example, this may be useful if you are invoking Visual Field from within an iframe and require linking to a parent page.

Drilldown Append Original Metadata

If this page was invoked with a passed METADATA parameter then by selecting this checkbox, that original METADATA parameter will be attached to any chart drilldown links. Note that it may not be essential to reattach the METADATA - as you application could potentially rely on existing saved offline state data. However, if you intend to provide hyperlinks that do not rely on any saved offline state, then as a general guideline, the METADATA (which defines your mini-app) parameter should be appended.

Pie Chart Display Theme

Pie Charts use this input field as a list of hex triplet color codes to be used in the generation of the pie chart. This value does not apply to Bar or Line charts. As the colors are internally converted to RGBA values, please only specify hex triplet color codes and not HTML named colors and ensure they are separated by commas.

X-Axis Column

Specify the X-Axis or, in the case of a Pie chart, the reference column. For Line and Bar charts, if the X-Axis Column is a parsable date (typically adhering to either ISO 8601 or RFC 2822 Date time formats), then the chart will be created as a Time-Series chart. In this case, the next field (X-Axis Order-By-Column) should also be set to this X-Axis Column Value or may otherwise result in curious (distorted) Chart Visualizations being generated.

X-Axis Order By Column

Optionally specify the column to order the result set by within the chart. If the Chart is a Time Series (ie, the X-Aixs Column is a date/time), then this should be set to the same field. For non-Time-Series Line and Bar charts, the ordering of the X-Axis is perhaps less important in relation to possibly distorting the resultant chart image.

X-Axis Order by Direction

Select either 'asc' or 'desc' for the order by direction.

Stacked

Applies only to Bar and Line charts and specifies whether the series are to be stacked in the chart. Note that Bars and Lines are stacked as groupings of Bars and Lines (and not combined).

Series - Type

Select Line, Bar or Pie for the type of chart to be displayed. Note that if this is set to Pie, then only Series 1 is used and Series 2-4 are ignored.

Series - Y-Axis Orient

Select "Left" or "Right" for the Y axis that will be associated with this series. Allows charts to be generated with up to 2 independent Y axis.

Series - Drilldown Column (Query String)

Nominate a column in the Table/View that has a Query String value that can be used for the self drilldown. The Drilldown Column will be passed as a Query String to the vf.html invocation. Note that this needs to be in the form of a valid web GET Query String name value pairs. Eg; param1=value1&param2=value2. Query Strings may also need to be encoded if the parameter name or value contains spaces for example. Please refer to the Misc:encodeURIComponent Custom Function as a possible mechanism that can be used to assist the encoding of such Query Strings (that could be called as a prior task to creating these links). The Query String value should NOT include a leading Question Mark. In relation to the hyperlink itself - a click event upon the associated chart element itself will invoke the hyperlink.

Series - Series Color

Select the color to be used for this series

Save

Will save your Chart Visualization Definition

List

Will give a mini list of your saved Chart Visualizations. Limited to 100 rows maximum - if you have more than this number of Chart Visualizations, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Chart Visualization

Run

Will run the saved Chart Visualization

Drop

Will drop the saved Chart Visualization

Reset

Will clear this form.

Will exit from a Chart Visualization and return to the Visual Field Home (Menu) pane

Will toggle between Fullscreen mode for this visualization

Will toggle Line Chart point markers between 3 set radii size values. Possibly useful for; 1) when viewing on a touch device that requires larger points to interact with, and 2) when downloading a Chart as an image it may be desirable to remove all point markers from the chart.

Will download a chart in an image format.

Usage Notes

1. If a Chart Visualization is used as a Task in a Sequence, the sequence will be terminated as part of this visualization task (even if there are other tasks defined after this task in the sequence).
2. Chart Visualizations have an internal limit of the maximum number of table/view rows that can be displayed. Charts with a greater number of rows will automatically fallback to a Table Visualization. Please refer to the General Help for details of this.
3. Pie Chart Visualizations have an internal limit of the maximum number rows that will include a chart header legend. In charts with a greater number of rows the header legend will automatically be omitted. Please refer to the General Help for details of this.

---

Map Visualization

A Map Visualization is a Leaflet Map presentation of a nominated table or view. The fields in this pane are used to define a Map Visualization.

Table/View

Specify the Table or View as the input source for the visualization.

Description

Specify a description to be associated with this visualization.

Title

This is the title heading that will be displayed at the top of the page when this map visualization runs. Titles can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself. For the case of "Column Value", the value of the first row of the table column will be used.

Task Id

A Task Id to be associated with this Map Visualization

Link Target

If a Drilldown Column is specified in the map definition, then upon invocation of such links, the "target" attribute of the generated hyperlink will be set to this value. For example, this may be useful if you are invoking Visual Field from within an iframe and require linking to a parent page.

Drilldown Column (Query String)

Nominate a column in the Table/View that has a Query String value that can be used for the self drilldown. The Drilldown Column will be passed as a Query String to the vf.html invocation. Note that this needs to be in the form of a valid web GET Query String name value pairs. Eg; param1=value1&param2=value2. Query Strings may also need to be encoded if the parameter name or value contains spaces for example. Please refer to the Misc:encodeURIComponent Custom Function as a possible mechanism that can be used to assist the encoding of such Query Strings (that could be called as a prior task to creating these links). The Query String value should NOT include a leading Question Mark.

Drilldown Column Replacement Label

This is just a constant string value that will be used in replacement of the Drilldown Column itself (name/value pair) and used as the hyperlink anchor text. The hyperlink itself will be displayed within the map geometry element onclick popup.

Append Original Metadata

If this page was invoked with a passed METADATA parameter then by selecting this checkbox, that original METADATA parameter will be attached to the URL string in the Drilldown Column. Eg; name1=value1&name2=value2&METADATA=... Note that it may not be essential to reattach the METADATA - as you application could potentially rely on existing saved offline state data. However, if you intend to provide hyperlinks that do not rely on any saved offline state, then as a general guideline, the METADATA (which defines your mini-app) parameter should be appended to your hyperlink.

Geometry Column(s)

Enter the table/view geometry (WKT) column that the Map Visualization will be based upon. In the current version of Visual Field, only a single geometry column can be entered.

Tooltip

This is the mouseover tooltip that will be displayed upon hover of map geometries. Tooltips can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself.

Custom Style

Maps with no Custom Style defined will use the Leaflet Mapping Library default path styling options. Please refer to the Leaflet Mapping Library Path options reference. Maps can have up to 6 custom styles defined that override the default Leaflet styling.

Custom Style - Option

Enter the Leaflet Mapping Library Path Option value. That is, the Path option relating to the map geometry.

Custom Style - regular/hover

Select either "regular" or "hover". "regular" means the styling will be used in place of the Leaflet default styling. "hover" means the styling will be applied upon mouseover hover of the map geometry.

Custom Style - Value

This is the value that for the styling that will be applied. Style values can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself. Applying styles from a "Column Value" will enable you to create data context driven map styling.

Save

Will save your Map Visualization Definition

List

Will give a mini list of your saved Map Visualizations. Limited to 100 rows maximum - if you have more than this number of Map Visualizations, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Map Visualization

Run

Will run the saved Map Visualization

Drop

Will drop the saved Map Visualization

Reset

Will clear this form.

Will exit from a Map Visualization and return to the Visual Field Home (Menu) pane

Will toggle (enable/disable) your device Geolocation. To enable this will require user input. If a position can be determined, the map will be zoomed to that location and a marker overlaid on the map. In addition, the VF_DEVICE_ATTRIBUTES table will be updated with the current position attribute information.

Will toggle between Fullscreen mode for this visualization

Will enable the fetch and store of Map Tiles that can be used later if the map is viewed offline (without an internet connection). Requires Map Tile feeds to be CORS enabled. Please note that your Tile provider may have usage policies relating to the fetching of Map Tiles for offline use.

Usage Notes

1. If a Map Visualization is used as a Task in a Sequence, the sequence will be terminated as part of this visualization task (even if there are other tasks defined after this task in the sequence).
2. Map Visualizations have an internal limit of the maximum number of table/view rows (features) that can be displayed. Maps with a greater number of rows will automatically fallback to a Table Visualization. Please refer to the General Help for details of this.
3. Table columns that are used to generate the map - eg; Map Title, Geometry or Custom Styling fields will be omitted from the Map Visualization geometry onclick popup attribute listing.
4. As mentioned above - the fetching of Map Tiles for later offline use may be subject to usage policies of your tile provider.

---

Media Visualization

A Media Visualization is a forward and reverse sequential display of a table with a media (Image, Audio or Video) column. It is much like a Slide Show Presentation with some additional context functions. The fields in this pane are used to define a Media Visualization.

Table

Specify the Table as the input source for the visualization. Note that this must be a table and not a view.

Description

Specify a description to be associated with this visualization.

Title

This is the title heading that will be displayed at the top of the page when this media visualization runs. Titles can either be a constant ("Literal Constant") or come from a table column ("Column Value") itself.

Task Id

A Task Id to be associated with this Media Visualization

Media Source Column

Enter the table media column that the Media Visualization will be based upon.

Sequence Order By Column

Enter a table column that will be used to order the Media Visualization Sequence.

Sequence Order by Direction

Select either 'asc' or 'desc' for the order by direction.

Link Target

If a Drilldown Column is specified in the media definition, then upon invocation of such links, the "target" attribute of the generated hyperlink will be set to this value. For example, this may be useful if you are invoking Visual Field from within an iframe and require linking to a parent page.

Drilldown Column (Query String)

Nominate a column in the Table/View that has a Query String value that can be used for the self drilldown. The Drilldown Column will be passed as a Query String to the vf.html invocation. Note that this needs to be in the form of a valid web GET Query String name value pairs. Eg; param1=value1&param2=value2. Query Strings may also need to be encoded if the parameter name or value contains spaces for example. Please refer to the Misc:encodeURIComponent Custom Function as a possible mechanism that can be used to assist the encoding of such Query Strings (that could be called as a prior task to creating these links). The Query String value should NOT include a leading Question Mark.

Drilldown Column Replacement Label

This is just a constant string value that will be used in replacement of the Drilldown Column itself (name/value pair) and used as the hyperlink anchor text. The hyperlink itself will be displayed within the Media Visualization Context Info Popup.

Append Original Metadata

If this page was invoked with a passed METADATA parameter then by selecting this checkbox, that original METADATA parameter will be attached to the URL string in the Drilldown Column. Eg; name1=value1&name2=value2&METADATA=... Note that it may not be essential to reattach the METADATA - as you application could potentially rely on existing saved offline state data. However, if you intend to provide hyperlinks that do not rely on any saved offline state, then as a general guideline, the METADATA (which defines your mini-app) parameter should be appended to your hyperlink.

Save

Will save your Media Visualization Definition

List

Will give a mini list of your saved Media Visualizations. Limited to 100 rows maximum - if you have more than this number of Media Visualizations, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Media Visualization

Run

Will run the saved Media Visualization

Drop

Will drop the saved Media Visualization

Reset

Will clear this form.

Will exit from a Media Visualization and return to the Visual Field Home (Menu) pane

Will open a context Info Popup of the remaining table attributes and values for this table row.

Advance to Previous Slide.

Advance to Next Slide.

Toggle between Fullscreen mode.

Usage Notes

1. If a Media Visualization is used as a Task in a Sequence, the sequence will be terminated as part of this visualization task (even if there are other tasks defined after this task in the sequence).

---

Define Prompt Input

A prompt input allows for the creation of a simple javascript prompt to retrieve a single line of input text from the user. This text will then be saved to the VF_QUERY_STRING table as a simple name/value parameter. It may overwrite any existing Query String Parameters. More detailed mechanisms for retrieving input fields from users are not currently provided.

Prompt Text

Specify the Prompt Greeting text that is to be displayed when the Prompt runs.

Query String Parameter Name

Specify the Query String Parameter Name for the value that will be saved to the VF_QUERY_STRING table.

Description

Specify a description for this Prompt.

Task Id

Specify a Task Id that will be associated with this Prompt.

Save

Will save your Prompt Input Definition

List

Will give a mini list of your saved Prompt Definitions. Limited to 100 rows maximum - if you have more than this number of Prompt Definitions, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Prompt Definition

Run

Will run the saved Prompt Definition

Drop

Will drop the saved Prompt Definition

Reset

Will clear this form.

---

Define Run Sequence

A Sequence is simply a Task that is made up of other tasks. When the sequence is run, the component task list is expanded to all leaf tasks and then they are run in order.

Sequence Task Id

Specify the Task Id for this Sequence.

Component Task Id's (comma separated)

This is a list of Task Id's that make up this sequence. The list should consist of comma separated numeric values.

Description

Specify a description for this Sequence.

Save

Will save your Sequence Definition

List

Will give a mini list of your saved Sequence Definitions. Limited to 100 rows maximum - if you have more than this number of Sequence Definitions, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Sequence Definition

Run

Will run the saved Sequence Definition

Drop

Will drop the saved Sequence Definition

Reset

Will clear this form.

Usage Notes

1. When executed, a Sequence is first expanded into a list of leaf Task Id's. In order to prevent possible infinite loops, a maximum sequence predefined limit is used. This limit can be adjusted somewhat under the Config settings; Maximum sequence task expansion setting.
2. In the current version of Visual Field, a sequence will terminate upon the rendering of any Visualization task even if there are other subsequent tasks defined as part of the Sequence. Such as a Table Visualization, Chart Visualization or a Map Visualization. This may be changed in subsequent versions of Visual Field.

---

Define Run Menu

The run menu is found in the upper left part of the Home (Menu) pane and can be identified by the cyan icon. Run menu entries are simple label entries that have a Task Id associated with them. The run menu gives a very simple way (shortcut) to run a task from the Home pane of Visual Field. This may be useful for presentations and quick summaries where a menu entry will be far easier to navigate to than locating the particular Task under under the general dropdown menus.

Menu Item Label

Specify the textual label that will make up the run menu entry.

Menu Item Title

This will be populated in the HTML title attribute of the Run Menu entry and may be useful for a simple tooltip. Optional.

Task Id

Specify the Task Id that should be run upon selection of this menu entry.

Relative Order

A numeric value that depicts the relative order (from lowest to highest) that this menu item should appear down in the menu list.

Save

Will save your Run Menu Item Definition

List

Will give a mini list of your saved Run Menu Definitions. Limited to 100 rows maximum - if you have more than this number of Sequence Definitions, use the Browse functions. In the mini list, the actions that can be undertaken are;

Edit

Edit the saved Run Menu Item Definition

Run

Will run the saved Run Menu Item Definition

Drop

Will drop the saved Run Menu Item Definition

Reset

Will clear this form.

---

Collect Device Attributes

Here you can specify a Task Id that will be associated with the collection of device attributes. Device attributes are a set of attributes that are specific to the client device. Instantaneous collected device attributes are saved in the VF_DEVICE_ATTRIBUTES table. No historical values are saved - only instantaneous values. Currently, only GeoLocation Device Attributes are collected and, with current browsers, typically will only work on file:// protocol or https:// protocol contexts. GeoLocation also typically requires an input action from the user.

Task Id

This is a Task Id that will be associated with the Collection of Device Attributes. The task will need to be run to collect the device attributes.

Save

Will assign the specified Task Id to the Collection of Device Attributes

Run

Will Run the Collection of Device Attributes.

Reset

Will clear this form and reset the Task Id.

---

On Page Load

Here you can specify a Task Id that will be run after the Visual Field page loads in the browser. This can either be a constant value or a value passed in as a Query String parameter.

Run Task Id

This is a Task Id that will be run upon when the Visual Field page loads in the browser. "Literal Constant" will run the specified Task Id, whereas "Query String Param" will run the value of the named Query String parameter passed in as a GET parameter as part of the URL to this page.

Save

Will assign the specified Task Id to the On Page Load

Run

Will Run the On Page Load Task Id.

Reset

Will clear this form and Reset the Task Id.

---

Browse

This pane enables you to readily browse objects (tables and views) in your Visual Field WebSQL database. In addition to your own created table, there are a set of Visual Field tables and views - which can be identified by the "VF_" prefix. Ordinarily, you should not need to browse or modify the "VF_" tables but there are some specific exceptions to this guideline - refer to the General Help section. Some "VF_" tables have table specific markup associated with them and will render with a certain set of functionality to allow further operations from the browse pane. For your own loaded tables, there will be no markup associated with the display of the table data. The Browse pane generates a paginated and sortable table listing. It can be used for listing of all tables, whereas the mini-lists within Visual Field are unsortable and are limited to 100 rows.

Table/View

Enter the Table or View you wish to browse. A datalist of values should be suggested to make easy identification of your loaded tables.

Search

You can filter your table result listing by search filter criteria. All tokens in the Search input must be present in any of the fields in your table.

Will Run your Search.

Will toggle filtering of individual table/view column values. Note that when enabled, filter values must be exact column value matches, whereas Search Input is a wildcard search. When the filter is enabled this button is displayed in blue; = Filter is Enabled

Reset

Will clear this form.

Will download the displayed table, with any filtering applied, as CSV format.

Delete

Will delete a single row from the table in your WebSQL database. Applies to tables only (not applicable to Views).

---

Browse VF Tables

A shortcut to Browse the Visual Field internal table VF_TABLES - which houses the metadata for any table you have created as part of a "Table import (File)" or "Table Import (URL)". Refer to the Browse section above for more detail.

---

Browse All DB Objects

A shortcut to Browse all Visual Field WebSQL database objects (tables, views and temp tables). Refer to the Browse section above for more detail.

---

Browse Media

A shortcut to Browse the VF_MEDIA table. The VF_MEDIA table is the table that raw media is loaded into using the media file import capability under the section. The displayed media can be resized by dragging the lower right corner of the media or can be displayed Fullscreen by clicking on the Media. Because Media can be very large in size (upto 40Mb), this pane is limited to a page size of 5 rows. This 5 row pagination will be used regardless of what you may have set in the "Table Page Size" setting in the Config section. This limited page size is intended to reduce the likelihood of browser memory issues (browser crash).

---

Show Query String

A shortcut to Browse Visual Field WebSQL database VF_QUERY_STRING table. This table is populated upon page load and prompt actions. Refer to the Browse section above.

---

Show Device Attributes

A shortcut to Browse Visual Field WebSQL database VF_DEVICE_ATTRIBUTES table. This table is populated upon the running of "Collect Device Attributes" Task Id. Refer to the Browse section above.

---

Photo Map

A Photo Map enables the quick creation of a Map Visualization of Geotagged Media contained in the VF_MEDIA table. This requires geotagged embedded Exif data within the loaded media. The Map display can be further filtered on File Modification Date as well as general searching against; Comments, File Name or All Exif tag entries. Because raw unprocessed Media may be potentially very large in size, there are some display limits on the resulting Map. These limits are as follows;

1. If the total size of Media to be displayed exceeds 300Mb, then the Media field will be omitted from the map and an attempt to use any embedded thumbnail instead will be done. Embedded thumbnails are Images that are contained within Exif tags within the Media themselves. Thumbnails will be used when the total original Media exceeds 300Mb and all the Media have a thumbnail Exif tag.
2. If the total size of the Media thumbnails exceeds 300Mb, or all the Media do not have an associated thumbnail Exif tag, then no Image will be displayed in the Map Media popup field.

From

Optionally specify a date that the Last Modified Date of the VF_MEDIA entry must occur on or after.

To

Optionally specify a date that the Last Modified Date of the VF_MEDIA entry must occur on or before.

Search

Optionally specify a text filter search expression to search within VF_MEDIA Comments, File Name or All Exif tag fields.

Reset

Will clear this form.

Will exit from the Photo Map and return to the Visual Field Home (Menu) pane

Will toggle (enable/disable) your device Geolocation. To enable this will require user input. If a position can be determined, the map will be zoomed to that location and a marker overlaid on the map. In addition, the VF_DEVICE_ATTRIBUTES table will be updated with the current position attribute information.

Will toggle between Fullscreen mode

Will enable the fetch and store of Map Tiles that can be used later if the map is viewed offline (without an internet connection). Requires Map Tile feeds to be CORS enabled. Please note that your Tile provider may have usage policies relating to the fetching of Map Tiles for offline use.

---

Export Metadata

This pane enables you to export the metadata of the essential Visual Field configuration that you may have defined. Exported metadata can then be re-imported either manually or automated by passing in on the Query String. So you can share your metadata with your colleagues or generate reports using your metadata. Metadata is made up of a simple JSON structure consisting of a core subset of the Visual Field configuration. So the metadata is human readable.

Encode as URI

If you intend to re-import your metadata in a file:// context as part of the Query String, then check this checkbox so that your JSON metadata is generated and then URI encoded. You can then make a reference to this metadata in the Query String that will be valid in a file:// context. Ie, if you want to make Visual Field reports without needing the use of a web server. The way you would reference this is to re-import the whole encoded metadata in a Query String URL upon page load as the METADATA parameter. Eg; file://vf.html?METADATA=<encodedURI>. This may be necessary because of browser security restrictions that prevent loading of file URL's over Ajax in a file:// context. Encoding of the Metadata as a URI is provided if you are not using a web server but wish to load in METADATA on the Query String. If you simply wish to generate a JSON representation of your Metadata, do not check this checkbox.

Reset DB

When Metadata is (re)imported it will be applied in a "merge" sense. In that existing Metadata will not be affected unless it is referencing by the same Id values. Metadata can be reset by the user - see Reset Metadata. But in a web context, whilst possibly unusual, this option can allow the reset of Visual Field Metadata objects upon pageload. Checking this option will simply include a tag; "RESET_DB:true" within the Metadata. When such metadata is loaded in a URL context on the Query String, then all Visual Field related WebSQL Objects will be dropped and recreated (as empty) upon initial page load. Normally, you should not need to check this checkbox (and for some actions where offline state is required to be maintained between web invocations this option should NOT be selected).

Export

Will export your Visual Field Metadata as either an Encoded URI text file or a JSON file.

---

Import Metadata

This pane enables you to import previously saved Visual Field Metadata. Simply drop your metadata file on this input field. Files can be JSON, or URI encoded JSON. Import of Metadata in this way by the user will apply it in a "merge" sense in that it will only overwrite objects with the same Id's. In this way you can share Metadata amongst your colleagues - and can do that non-destructively by using non-overlapping sets of Id's. Metadata imported in this way ignores any "RESET_DB" setting within the Metadata.

Usage Notes

1. To re-import Metadata by using the Query String, specify the parameter METADATA. When passed on the URL as a Query String parameter, the METADATA parameter will be evaluated as;
   1. If the METADATA parameter is an URI encoded string of JSON, it will be parsed and imported.
   2. If the METADATA parameter is a string (that fails to parse as JSON), it will be assumed to be a URL reference and fetched as a URL (remote web request) to retrieve the Metadata as JSON.

---

Reset Metadata

This pane enables you to reset Visual Field Metadata. Reseting Metadata means that all Visual Field WebSQL objects will be dropped and recreated as empty. If you have existing objects within the database, these should not be affected (however, any metadata references to them will be lost).

Reset

Will drop and recreate (empty) Visual Field Metadata WebSQL objects.

---

Export Dump

This pane enables you to create a Visual Field Export Dump of either an individual table or view, or of your entire database. A Visual Field Dump can then be re-imported at a later stage or imported into another HTML5 WebSQL enabled browser using Visual Field by using the "Import Dump" facility (see next section). A Visual Field Dump is made up of simple encoded variable width csv data. It is intended to be a robust method to enable the offline transfer of database objects, and data, between Visual Field WebSQL databases.

Max Dump File Size

This is the maximum file size that will be used when creating a Dump File. Dump Files will have a maximum of this size. During the creation of the Dump, if a write exceeds this file size, the Dump creation will be unsuccessful. This setting generally does not need to be changed. If a Dump uses less space than this limit - there should be no issues when creating the Dump. This limit is both browser and device dependent. At the time of writing, this limit was typically 2Gb on most devices. If your Dump will exceed this limit, you may need to break up your Dump by using the "Export Single Table/View" option.

Export Single Table/View

This will enable you to create a Dump of a single Table or View. A Dump will consist of encoded DDL drop and create instructions. And in the case of a table, index DDL creation and DML data load instructions. Dump Files can then be imported back into Visual Field using the "Import Dump" facility.

Export Entire Database

This will enable you to create a Dump of all of your Visual Field Database objects (tables,views,indexes) along with associated table data. Such Dumps can then be imported back into Visual Field using the "Import Dump" facility.

Omit Metadata Objects

In the case of a full Dump, you can omit Visual Field metadata objects by selecting this checkbox. Metadata objects are those database objects that begin with "VF_". Such metadata objects partly make up the sequencing, workflow, URL/File table definitions, etc, that you may have defined. It may be useful to not overwrite Metadata objects during an Import - especially perhaps if you are sharing or transferring data with your colleagues and you do not wish to overwrite their Visual Field Metadata for example.

---

Import Dump

This pane enables you to import a Visual Field Export Dump file. Simply select a file or drag and drop your Dump file here. Dump Imports will display importation progress and any failure lines encountered during the Import.

---

SQL Dump

This pane enables you to create an SQL Dump of either an individual table or view, or of your entire database. An SQL Dump is intended to be a portable mechanism to facilitate the easy import of such Dumps into another SQL Database such as sqlite or PostgreSQL. Due to various current limitations, SQL Dumps are perhaps not as robust as the "Export Dump" facility (discussed above) however SQL Dumps are perhaps more portable because they consist only of SQL statements. Please refer to the General Help regarding the differences in Export types and limitations. Currently, there is no SQL Import facility provided in Visual Field.

Max Dump File Size

This is the maximum file size that will be used when creating a Dump File. Dump Files will have a maximum of this size. During the creation of the Dump, if a write exceeds this file size, the Dump creation will be unsuccessful. This setting generally does not need to be changed. If a Dump uses less space than this limit - there should be no issues when creating the Dump. This limit is both browser and device dependent. At the time of writing, this limit was typically 2Gb on most devices. If your Dump will exceed this limit, you may need to break up your Dump by using the "Export Single Table/View" option.

Export Single Table/View

This will enable you to create an SQL Dump of a single Table or View. An SQL Dump will consist of SQL DDL drop and create instructions. And in the case of a table, index DDL creation and DML data load instructions. SQL Dump Files can then possibly be imported into another database such as sqlite or PostgreSQL.

Export Entire Database

This will enable you to create an SQL Dump of all of your Visual Field Database objects (tables,views,indexes) along with associated table data. Such Dumps can then possibly be imported into another database such as sqlite or PostgreSQL.

Omit Metadata Objects

In the case of a full SQL Dump, you can omit Visual Field metadata objects by selecting this checkbox. Metadata objects are those database objects that begin with "VF_". Such metadata objects partly make up the sequencing, workflow, URL/File table definitions, etc, that you may have defined. Visual Field Metadata objects are possibly of little use in a non browser based database (outside of Visual Field).

---

Config

This pane enables the setting of various application wide values.

Database Name

This will specify the WebSQL database name. Currently disabled and fixed as "vf"

Database Size

Specify the size of the WebSQL database in Mb.

Show Status Pane during Sequence Execution

The Status Pane shows a broad summary during Sequence Execution. It is set to be shown by default (with this setting) and it is generally useful once your report/workflow is deployed in a production sense so that some general feedback is given to the user. Unchecking this and running a sequence will switch between pertinent panes during sequence execution. This might be more useful during development so you can monitor the details of the sequence execution but could be distracting once a report is finalised. Once you finalise and debug your sequence, you should re-check this checkbox.

Truncate display of displayed fields after

This sets a limit on the maximum number of characters that will be displayed in Visual Field for any given result set. Eg, Browse, SQL output and Table, Chart and Map Visualizations. This helps mitigate against potential browser issues (memory crash) due to extremely large table fields as the display will be truncated after this length. Truncation is not applied to CSV downloads.

Table Page Size

This sets a limit to the pagination for SQL output, Table Browse and Table Visualizations. Setting a page size and pagination of output helps mitigate against potential browser issues (memory crash).

Map Base Layer Tile URL

Enter the Tile URL for use in any Map Visualization. Map Tile URL's should be in EPSG 3857 datum. To enable Offline Map Tile store, Tile URL's will need to be CORS enabled. A sample of Tile URL's is available by selecting a Sample Base Layers.

Map Base Layer Attribution

Enter the Map Base Layer Attribution that for the Map Base Layer Tile URL. This will be displayed on Map Visualizations.

Save

Save the specified Configuration.

Reset

Reset the Configuration to default Settings.

---

Import Media

This pane facilitates the import of media files into your Visual Field database. Media Files may be of type; Image, Audio or Video. Multiple files may be imported at a time. Simply drag and drop your media files onto the Import area. Individual file sizes are limited to a maximum of 40Mb. Files larger than 40Mb or non Image, Audio or Video files will be rejected. Files will be inserted into the VF_MEDIA table using the file name as the primary key. If an existing entry in VF_MEDIA exists for a file of the same name, then that will be replaced the the new media - otherwise a new entry will be made. If Exif data is present in your media files, then this will be extracted and inserted into a table called VF_IMG_EXIF. VF_IMG_EXIF is linked to VF_MEDIA by using the FILE_NAME column as the key. There may be multiple rows in VF_IMG_EXIF for a given file name. If the embedded Exif contains extended parameters such as XMP tags, etc (eg, from Drone Imagery), then these extended Exif values will also attempted to be extracted and placed into the VF_IMG_EXIF. Such extended Exif values can be distinguished by the CATEGORY field in VF_IMG_EXIF.

It is assumed that your browser will be capable of displaying your particular media. It may be necessary to convert your media into a browser enabled format prior to Import. Because of the 40Mb limit and otherwise, you may need to consider reducing the file size prior to Import. Whilst raw files upto 40Mb may be imported, you may achieve better browser response time by converting imported media prior to Import. Eg, converting Video to a format optimised for Browser viewing. Or resizing your Images for Browser viewing. This is optional however.

From within Visual Field, Media fields can be viewed just like regular fields from Browse functions and form SQL statements, etc.

Comments

You can optionally specify a simple Comments text as part of you Media Import. Comments will be inserted into the VF_MEDIA COMMENTS field and may be useful for filtering by at a later point in time.

API Guide

The METADATA that can be generated and imported back into Visual Field is essentially a simplistic JSON object with 4 key elements. This may be optionally URI encoded but underlying it is still a simple JSON object. This API page is included for those who may wish to automatically generate Visual Field metadata such as an automated report for example. The 4 elements in the metadata are;

VERSION

This is the Visual Field metadata version number. It is not the same as the Visual Field Release number but the Version number represents the structure of the metadata. The following describes the metadata structure for Version 1. This is currently the only version supported for import. This should be set to 1 for any generated metadata.

RESET_DB

Either "true" or "false". Indicates whether the WebSQL database Visual Field dictionary objects should be dropped and recreated upon page load. Applicable only when passed in using the METADATA parameter in the Query String and is ignored for manually imported metadata. This should normally be set to "false" or omitted. Should you need to reset the WebSQL database, then set this to "true".

CONFIG

An array of Config Item values for Visual Field. Config Values typically broadly affect the behaviour of Visual Field. Please refer to the config section displayed below for the specifics of what needs to be included in the CONFIG array. The CONFIG array needs to be included with the correct number of elements.

TABLES

The TABLES element is an object of a subset of Visual Field tables that are used to represent the core Metadata structured elements. This is a subset of all of the Visual Field tables and views. Please see the Tables section displayed below for a listing of these tables. TABLE elements are simply an array of table rows (another array). TABLES need to be included with the correct table elements and the correct number of elements for each table row to be imported successfully.

CONFIG

The CONFIG element is made up of an array with the following entries;

TABLES

The TABLES element is made up of an array of elements that are the Visual Field internal tables that are represented in the metadata. TABLE element themselves are then made up of an array of rows. The list of TABLES that can be included in metadata definitions is a subset of all of the Visual Field internal tables. Select a TABLE below to detail the row components to be included in a TABLES definition entry;

CUSTOM FUNCTION (ID)

Custom Functions are identified by a numeric ID within Visual Field. This is an extensible array should you wish to add your own function definitions (refer to the General Help section). To assist in identifying a Custom Function definition and its ID details, please choose from the following selection. Custom functions that are deterministic in that their output is solely dependent upon their input parameters, should be defined as "type=immutable".

General Help

Mandatory, Optional input fields and Editable Content

• This Styling Indicates a Mandatory Input Field
• This Styling indicates an Optional Input Field
• This Styling Indicates an Editable Data Content

File Import

If your Import File is in CSV standards format (RFC 4180) with a 1 row header containing field names, then you can generally accept the default settings and import your data without needing to change any settings. Many open data CSV datasets are in this format.

CSV parsing of multi line fields

The internal algorithm in which Visual Field parses import delimited files (eg, CSV) varies dependent upon the size of the input file. For files less than 20Mb, the parser will perform a full standards parse (RFC 4180) on the input file as a whole (internally done by the fantastic jQuery-csv parser). So any multi-line input fields (fields containing carriage returns or line feeds) should be parsed correctly. However, full file parsing of the input files in this way is not scalable for extremely large input data files. And this can cause memory utilisation issues within the browser (ie, crash) if this approach were to be adopted for all file sizes. To mitigate against this, there is an internal threshold set in Visual Field of 20Mb. For files larger than this 20Mb threshold, the input file will be split line-by-line and processed incrementally (ie, CSV parsed line-by-line). This enables Visual Field to ingest extremely large CSV, or other delimited, files (without having to parse the file as a whole). However for such large files (larger than this 20Mb threshold), multi-line input fields will typically cause a parse failure of those particular lines. In actuality, most of the very large delimited files, there are typically no multi-line fields. So in general, this 20Mb threshold for parsing approach - should go unnoticed and should not cause any issues - it also means that the Visual Field input file ingestion is very scalable for extremely large files with little to no inconvenience. Please refer to the Internal Limits section below.

Querying Columns containing spaces

If you are not accustomed to referencing database columns containing spaces when using SQL or similar, you may need to enclose those fields either in double quotes or square brackets. For example, suppose you have a table called SCHOOLS and one of its fields is "Upper Grade", then you could query this by something like;

select "Upper Grade" from SCHOOLS;

Or,

select [Upper Grade] from SCHOOLS;

Casting to correct datatypes

When importing delimited data (from the File Import or URL Import), for the created database table all of the field datatypes will be TEXT. No heuristics are applied to attempt to ascertain import field datatypes. Columns are simply all created as TEXT datatype and you may wish to CAST these to other datatypes. Casting can easily be done by various ways. You can cast in subsequent SQL queries. You may also wish to consider creating a task that runs after your table import and drops/(re)creates a database view that does the appropriate datatype casting. That is, the fields in the view could be cast to the appropriate datatypes. Casting is not strictly important in general - you can typically query numeric and string values in a similar way. However casting to the correct datatypes may become important when you start to relationally join tables together by SQL joins. Also, imported Query String Parameters placed in the VF_QUERY_STRING table are created as "TEXT" - and again they may need casting to appropriate datatypes if you relationally join these to other tables via SQL queries or views.

create view if not exists SCHOOLS_GRADE
as 
select cast("Upper Grade" as INT) "Upper Grade",
"Campus Name"
from SCHOOLS;

group_concat

There are several Custom Functions in Visual Field that require an array input by way of a delimited list. That is typically a list of values separated by a comma to form an input array of values. This may initially seem difficult to produce from SQL but thankfully, within WebSQL (sqllite), there is the excellent group_concat function. This is a string aggregation function with an optional delimiter. For example, if you wish to apply a statistical Custom Function against all lower grade values from the SCHOOLS table, you could prepare input to the Custom Function by a statement similar to the following;

create table if not exists SCHOOLS_LOWER_GRADES
as 
select group_concat("Lower Grade",',') as lower_grade_list
from SCHOOLS;

There are also several Spatial Custom Functions in Visual Field (courtesy of the fantastic Turf js library) that accept an array of geometries as input. Again, the group_concat function may be useful in producing a list (array) of WKT geometries for input. In this case however, WKT geometries may contain commas, so it is safer to enclose the grouped geometries in double quotes (delimiter). Eg, suppose our SCHOOLS table has a WKT geometry field called "Location 1", then to get a list (array) of geometries, you could use the following;

create table if not exists SCHOOLS_GEOMS_LIST
as 
select group_concat('"'||"Location 1"||'"',',') as geom_list
from SCHOOLS;

But what about the case where your input may contain double quotes or commas? The approach here is to take group_concat further with the replace function and replace any occurrence of a double quote with a double double quote. In this case you will essentially be producing a standards based CSV input list (array) from SQL! Eg, suppose we wish to make a listing of the "Campus Name" field in the SCHOOLS table and that field may contain either commas or double quotes, then the following approach is suggested;

create table if not exists SCHOOLS_NAME_LIST
as 
select 
group_concat('"'||replace("Campus Name",'"','""')||'"',',') 
as name_list
from SCHOOLS;

If for some reason you need the inverse function to group_concat, please see the Custom Function Misc:ListSplit - which takes a list (delimited array) and splits the results out to individual rows in the result table.

Editable Data Content

Editable Data Content enables you to easily change a database field value by simply clicking on the value and changing the content in page. Your WebSQL database will be updated with the new value. Slightly different styling is used to indicate an Editable Data Content and it may be displayed in;

• Browse Screens - directly in the displayed result table
• Table Visualizations - directly in the displayed result table
• Map Visualizations (including Photo Map) - in Map Feature Popups
• Media Visualizations - in Media Info Popups

There are several limitations that will cause a data value to not be able to be edited in page. These limitations are;

• The Content is based on a database View instead of a Table.
• The Content is not a simple scalar field. Eg, it cannot be edited if it is made up of;
  • A hyperlink
  • Media - eg; Image, Audio or Video
• The Content has had a formatting function applied.
• The Content has been truncated for display. Refer to the Config Section for setting length limits of displayed fields.

Exceptions to Guidelines relating to VF_ dictionary objects

As mentioned in the Reference Guide section, the Visual Field data dictionary is also maintained in the same WebSQL database that your tables will reside. The Visual Field dictionary tables and views are generally prefixed with "VF_" and there should generally be no need to query or run DML directly against these dictionary objects. There are a few specific exceptions to this guideline however, namely; VF_QUERY_STRING, VF_DEVICE_ATTRIBUTRES and VF_TILE_STORE.

VF_QUERY_STRING

This table contains the name value pairs of any Query String parameters passed to Visual Field in a GET context.

To obtain Query String expressions within SQL, it may be necessary to query the VF_QUERY_STRING table directly. No shortcut technique is provided - it is up to you to manually query this table. Some points worth noting also are;

1. The passed Query String values are of type TEXT. As per mentioned above, if you are joining such values to other tables, you may need to explicitly CAST such values correctly either as part of the join or otherwise.
2. Multi value parameters are acceptable and can be passed in so you may need to account for that also in any SQL referencing the VF_QUERY_STRING table.

Here are some SQL examples of querying VF_QUERY_STRING; Suppose our URL is; vf.html?A=-2&B=0&B=1

select "Campus Name"
from SCHOOLS
where "Lower Grade" =
(select cast(value as INT)
from VF_QUERY_STRING
where name = 'A');

And for a multi valued parameter;

select "Campus Name"
from SCHOOLS
where "Lower Grade" in 
(select cast(value as INT) 
 as "Value"
from VF_QUERY_STRING
where name = 'B');

VF_DEVICE_ATTRIBURES

Similar to VF_QUERY_STRING the VF_DEVICE_ATTRIBUTES will be populated with a Collect Device Attributes task invoked and acceptance by the user. Device Attributes are similar to VF_QUERY_STRING in that they may need to be explicitly queried and CAST to appropriate datatypes.

VF_TILE_STORE

The VF_TILE_STORE houses stored offline Map Tiles. This table is initially empty and gets populated if a user clicks the button in a Map Visualization and downloads Map Tiles for possible Offline use. Map Tile images must be CORS enabled and are simply stored in the VF_TILE_STORE table as an encoded array of Uint8 values. This is referred internally as ENCODING_ALGORITHM 1. However, more advanced users should recognise that the VF_TILE_STORE can be Browsed in Visual Field just like any other table and extracted as a CSV download. So, if you are technically impelled, you could create a tile store, extract it, and make the CSV available as a web service. You could then define a Table (Import URL) as part of a Page load, Run Menu, or Sequence that will fetch the web accessible CSV Tile Store and reload it into the VF_TILE_STORE table. You will then be able to create a mini application complete with an Offline Map Tile store. Please Note, as mentioned in the Reference Guide, downloading Map tiles for Offline use may be subject to your Tile providers usage policies.

Preparing multi table input for Custom Function invocation - an example

Suppose we have 2 tables; SCHOOLS and HAZARDS. The SCHOOLS table contains a point WKT geometry ("Location 1") and the HAZARDS contains a polygon WKT geometry ("the_geom"). Suppose we want to run the Turf:booleanPointInPolygon test against these 2 tables. To do this, it will be necessary to combine these prior for input to the Custom Function into a single table. The following SQL extract demonstrates how to possibly do this by way of a preparing a Cartesian Product. You could then use this joined table as input to the Custom Function and link the output of the Custom Function by way of referencing the appropriate ROWID's back to the source tables;

create table if not exists SCHOOLS_HAZARD
as select s.rowid as SCHOOLS_ROWID,
  s."Location 1" as the_point,
  h.rowid as HAZARDS_ROWID,
  h."the_geom" as the_polygon
from SCHOOLS s, HAZARDS h;

Chart and Map - Table Visualization Fallback

Table Visualizations in Visual Field are paginated in the number of rows displayed on a page and a limit also set for the maximum field display length. As such, Table Visualizations and SQL Statement (output) is intended to be much more scalable and allow far greater result sets. Browse and Table Visualizations are written in a way that should enable indefinite scalability where SQL Statement (output) is scalable to about 512Mb result set size. However, this is far greater than what is practical as a Chart or Map Visualization. And a Table Visualization should be fully scalable (and render without crashing the browser) even for extremely large tables. This is currently not the case for Chart and Map Visualizations. Currently, Chart (using the Chartjs library internally) and Map (using the Leaflet library internally) Visualizations typically attempt to render (or process) the entire result set. This tends to mean that, currently, such visualizations may not scalable for extremely large datasets and may cause memory issues in the browser (ie, may crash the browser) if left unbounded. To mitigate this, there are some fixed internal fallback thresholds in Visual Field that will render a Table Visualization instead of a Chart or Map after a certain number of rows. Please refer to the Internal Limits section below for more detail. Once a Chart or a Map result set result set exceeds a certain number of rows - a Table Visualization will be shown instead. To avoid such fallbacks, you may need to either reduce or somehow aggregate your data to produce a smaller result set before showing as a Chart or Map Visualization.

Visual Field - Internal Limits

The following are some fixed internal limits within Visual Field;

• 20Mb - Maximum full standards parse (RFC 4180) CSV file size. Larger CSV files will be parsed line-by-line.
• 10,000 - Maximum number of rows in Chart Visualization result set before falling back to Table Visualization.
• 30 - Maximum number of rows in Pie Chart Visualization result set before Pie Chart header legend will be omitted.
• 120,000 - Maximum number of rows in Map Visualization result set before falling back to Table Visualization.
• 3,000 or any coincident Points - Maximum number of rows in Map Visualization Point only result set before Points will clustered.
• 100 unsorted rows - Maximum number of rows displayed in Visual Field mini-lists.

Visual Field - Soft Limits

The following are some observed limits;

• 512Mb - Maximum SQL Statement result set limit. If you generate a result set greater than this size you may encounter browser memory issues (browser crash). For "select" statements, consider using the OFFSET and LIMIT directives to reduce the total result set.
• 512Mb - Maximum CSV Download size. If you generate CSV greater than this size you may encounter browser memory issues (browser crash). You may need to break up your CSV downloads into chunks by using the SQL Statement pane and specifying appropriate OFFSET and LIMIT directives in order to reduce the total result set.

Brief Discussion of Differences and Limitations between Metadata, Export Dump and SQL Dump

Visual Field Metadata Exports primarily define the key aspects of your Visual Field configuration. That is in respect to what you define in various Visual Field screens in the way of CSV file ingestion, defined sequences and visualizations for example. Visual Field Metadata is primarily intended to be useful if you are using Visual Field in an online (web) capacity, or a reporting, or mini data-driven application. Metadata is a JSON representation of your Visual Field configuration and can be imported into Visual Field by use of the METADATA parameter in the Query String when you invoke vf.html as a standalone file or from a web page. Visual Field Metadata can be used in both an online and offline capacity to define a particular Visual Field configuration.

A Visual Field Export Dump is a variable width csv file of encoded DDL and DML instructions. Visual Field Export Dumps are specific to Visual Field and should be able to be imported into Visual Field using another device or HTML5 WebSQL browser running Visual Field. As such Visual Field Dumps are intended to be a robust offline way of transferring your database objects between devices, browsers, or just making a backup.

SQL Dumps just consist of a series of DDL and DML SQL statements. SQL Dumps are intended to be a portable mechanism to facilitate the offline transfer of your Visual Field objects to another database such as sqlite or PostgreSQL. If you are using Visual Field for offline asset management, this SQL Dump option may be useful to send you offline edits to a central database.

However, SQL Dumps are perhaps not as robust as Export Dumps for a number of reasons.

1. SQL Dumps are just SQL statements and are not encoded. This may cause issues if you are using non-ascii (non-English) data or non-ascii object names. Ie, UTF8. Importing UTF8 database objects or data into another database may be problematic when running as SQL in another target database. And hence may cause SQL parsing errors in your target database.
2. Another significant limitation with SQL Dumps, is that, at the time of writing, a single WebSQL SQL statement is limited to 1,000,000 characters. Unfortunately, this is a significant limitation and pretty much means that the exportation of large database objects (such as Audio, Image or Video objects) is not practical in the way of an SQL Export. So again, whilst portable, the SQL Export is perhaps not as robust as the Export Dump facility. Meaning SQL Dumps are probably best suited to exporting tables with field data no greater than 1,000,000 characters.
3. There is currently no "SQL Import" facility provided within Visual Field.

Storage Allocation Error

If you experience this error during the Import phase (there was not enough remaining storage space, or the storage quota was reached and the user declined to allow more space), it may indicate that a request for the initial WebSQL database storage was initially successful however during actual allocation, your browser is enforcing quota restrictions on the amount of usable storage/disk space on your device for TEMPORARY objects. Common quota restrictions for Chrome and Opera on Windows and Android platforms for TEMPORARY objects (ie, WebSQL database) may mean that it is necessary to have up to 10 times the required database storage space available. Such quota restrictions commonly apply the following rules; 1) only 50% of remaining storage/disk space on your device will be considered in the usable "pool", 2) of the usable "pool" a single web page may only be permitted to use a maximum of 20% of the pool. In other words, you may need up to 10 times free space on your device that what your Offline Visual Field WebSQL database will occupy.

Offline Web Application

Whilst Visual Field is intended to be able to function as a stand alone file (via the file:// protocol context), it is also capable of operating as a true offline web application. To do this, it will be necessary to place the optional vf_service_worker.js file into the same directory that you have placed vf.html on your web server. Please see instructions at visualfield.org on how to download and use the vf_service_worker.js file. The optional vf_service_worker.js file needs to reside in same directory that you are hosting vf.html to make it into an offline accessible web application.

About

Copyright (c) 2019-2020 Harris Hudson
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of
conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list
of conditions and the following disclaimer in the documentation and/or other materials
provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Import Media

Comments

Geotagged Photo (Media) Map

From

To

Search

Comments File Name Exif Tags

Go Reset

Export Dump

Max Dump File Size Mb.

Export Single Table/View

Table/View (Case Sensitive)

Import Dump

 

Import Processing File Line:

Import Failed on File Lines:

SQL Dump

Max Dump File Size Mb.

Export Single Table/View

Table/View (Case Sensitive)