Visual Field

Explore, Process and Visualize your data right in your browser.

November 2023 Update

As of Chromium version v119 (November 2023), WebSQL appears to now have been deprecated. In order to keep your copy of Visual Field still functioning, you might consider to add a Chrome "Deprecation Trial Token" API string to your http headers (eg, in your .htaccess file) when serving Visual Field from a web server, or to modify your copy of vf.html to add an appropriate meta tag to include a token for your domain. You will need to do this in order to keep your copy of Visual Field still functioning in Chromium based browsers. Please search; "Google Deprecation Trial Token" and register under "WebSQL" for an API token for yourself. Then setup your API token accordingly. You will need a Google account in order to do this. Chromium version v123 (due March 2024) appears to be the final browser release date that WebSQL will be supported in this way. Going forward - What does this mean for Visual Field? The immediate impact is, as from now, and in to the future, will be that running Visual Field as a stand alone html file (eg, loading in a file:// protocol context) will probably no longer work. Harris Hudson, the author of Visual Field, is currently assessing the proposed alternative solution of using SQLite directly as a wasm/javascript port along with local offline file storage. This will be a significant exercise to analyse, assess the feasibility, refactor and add additional code to the existing 23,500 line code base that makes up Visual Field today. If a SQLite wasm/javascript alternative port can be leveraged, then a new release of Visual Field will be targeted hopefully for some time in 2024. Realistically however, this will be unlikely before March 2024. Having said that, if the alternate SQLite wasm/javascript engine does indeed work in a true offline sense - then future possibilities may be encouraging; including broader browser support and access to more SQLite functions within Visual Field. Stay Tuned.

Run Now or Download

Current Version (3.4.0) Run Now; vf.html, Download; (1013 kB, 164 kB gzipped)

integrity="sha384-Elej0LuLYWgkFmbSDN1tijZIrA9d7l1PAI7WluiCnJJRCXwkjT4GWkFEGWaLQbbr"

Demo

Example 1
SF Schools by CCSF
A Basic Map

Example 2
SF Schools Lead (5Ug/l)
Multi jurisdictional data sets. Map.

Example 3
SF Schools Hazard Zones
Multi data sets. Custom Function. Map.

Example 4
SF Crime Incidents
Mini data driven App. Cache Max-Age. Charts. Map. Drilldown.

Example 5
ACT Cycle Crashes
Heat Map.

Example 6
NYC Vehicle Collisions Person
Very Large Dataset. Table Visualization.

Example 7
NYC Collision Death Rate
Multi series Chart.

Example 8
SF Covid Choropleth
An SQL generated Choropleth Map.

Example 9
NYC Covid Cases
Multi time series chart.

Example 10
SF Street Network
Multi data sets. Custom Function. Shortest Path. Physical Network.

Example 11
Sample Logical Network.

Example 12
Advanced Loading Techniques.
Simple Table Visualization.

Example 15
SF Muni Simple Routes.
Map Visualisation.

For these examples, your Visual Field metadata is not reset and will just be added to cumulatively. Reseting your metadata between each example, may make it easier for you to understand individual example configuration. Go; Dictionary Reset Metadata Reset. Press for a Help Page about each example.

What is Visual Field

Visual Field is essentially an Open Source single html file - vf.html, with some other associated files to support different usage modes. It enables you to consume and process delimited data files and web services (eg; CSV).

It is driven by declarative Metadata via the web interface - No HTML or Javascript programming is required (although SQL may be useful).
You can apply SQL statements, numeric, statistical, text and spatial functions against your datasets.
Tasks can be arranged to run as a predefined sequence based upon page load or other user interactions. You can make ETL driven workflows.
Table, Chart, Map, HeatMap, Media and Network Visualizations can easily be made to present your data.
Visual Field is easy to get up and running. It is intended to function either as a stand alone file (just download it and you are ready to go - no IT skills and no zip file extraction is required). Or it is intended to be able to be served from your webserver - either as a stand alone web page or as an embed (eg, within an iframe). Provided the additional associated files are also included on your web server, in this way, end users can use Visual Field as; an Online Web Application, an Offline Web Application, or an installed Progressive Web Application.
With Visual Field you can process your datasets, or other remote CORS enabled datasets, directly in the browser - no back-end or cloud database is required. It's truly a server-less database.
Visual Field has features to enable the extraction, re-import, and runtime use, of your Metadata, Export Dumps and SQL Dumps. So you can share your data driven workflows and visualizations with your colleagues or put them on your website.
You may find Visual Field useful if you are; a Data Analyst, GIS Analyst, Data Scientist, Report Writer, Web Developer, Teacher, Student or anyone who may wish to create data driven presentations or rapid prototypes.

Change Log

Version 3.4.0 (Current Version) - Released September 2021

Gracefully cater for database objects that have no ROWID field
Numerous minor bugs corrected

Version 3.3.0 - Released August 2021

integrity="sha384-wpGCiOi+e2oOGx1o6BeI5NTwR/EEdOATeWMuiFMAKMo7hGcNX2BsLvCz6EaxAcpA"
Added some configurable Internal Settings to easily modify METADATA Query String parameter processing behaviour
Updated versions of some dependent libraries

Version 3.2.0 - Released July 2021

integrity="sha384-tsku+07BWvp1CuKY+rWnS7Gv3VdXy5NeouzMxkOsaFSYO7RhkiT+s+N+RL4p/VMR"
Updated versions of core dependent libraries
Added Custom Function; Turf pointsWithinPolygon
Added Pan and Zoom to Chart Visualizations

Version 3.1.0 - Released June 2021

integrity="sha384-NiNy5NgO0lS3TnjBkq+0SBpndgSJaQte71J6dwBZPTkJRKVaRbbGB9FiBTSSjNat"
Updated versions of most core dependent libraries
Improved code formatting and readability
Added Custom Functions; Turf boolean.intersect, Turf polygonTangents and Turf polygonSmooth
Added Scatter charts to Chart Visualizations
Added PWA "Install" Add to Home Screen (A2HS) functionality
Rectified numerous minor bugs

Version 3.0.0 - Released October 2020

integrity="sha384-mhbuDuPmNigHOuV7dS7w5bfDh4HTe5xzDx5O5z8+zjIC4Mv+/fmmfxTP0ICZnw3o"
Added Network and HeatMap Visualizations
Added String and Proj4js Custom Functions
Added Save/Restore Network Graph and view Sequence Run Log
Numerous minor bugs corrected and general UI tidy up

Version 2.0.0 - Released May 2020

integrity="sha384-Ld5rtVWf4tthw3R1xDmCIbnaFulG+E1c9DzgZn/YVPqPGx8ozYj/Pw06m0mWzzBo"
Added Multi-Media support - Import your Image, Audio and short Videos in to your database
Photo Map - view a map of your geotagged images
New Custom Function to allow easy resize of your images
New Media Visualization - easily create a context rich slide show of your media
Added ability to update your data directly as displayed in page in table browser, table visualization or map visualization
Export Dump your tables or entire database and import directly into another HTML5 WebSQL browser with Visual Field
SQL Dump your tables or entire database so you can import that into a separate database (eg; PostgreSQL)
Filter Browse results by individual table fields
Updated Chart Visualizations for Time Series
Numerous minor bugs corrected

Version 1.0.0 - Released Dec 2019

Initial Release

Usage

Not all current browsers support HTML5 and WebSQL. And of those browsers that do, not all underlying devices support a WebSQL database when invoked as a standalone html file. The browsers currently supporting a fully WebSQL database is still somewhat fairly dynamic and changing (even as at 2019). See "Get Involved" section below.

Application Usage

Visual Field is intended to be able to be used in any of the following ways;

Online Web Application
Offline Web Application
As a stand alone html file
As an Installed Progressive Web Application

Optional Support Files

Optional Offline Web Configuration - Example.

The vf.html file is intended to be able to be loaded as a stand alone file (from a file:// protocol context) or served via a web server. If you do serve it via a web server, you may wish to download and add the optional vf_service_worker.js file and place it in the same directory as vf.html. This will turn your served version of Visual Field into a true Offline Web Application in that it can be reloaded and used in a browser (from a https:// protocol context) without needing internet access. Something to consider if you make presentations, or you may be an emergency worker out in the field, with unreliable internet access.

To demonstrate the Offline Web capability, follow this scenario;

Load Example1 (using a https:// protocol context) here; Example 1 (SF schools)
From the Map, press the button to select a zoom level and download some Map tiles for offline use.
Disconnect your device from the internet, by going to Airplane mode or similar.
Reload the above link (from Step 1) and your Map and Schools data should be presented intact.

Tutorials

Tutorial 1
SF Schools by CCSF
14:08 min

Tutorial 2
Version 2 - What's new.
21:10 min

Tutorial 3
Building Example 10 shortest path.
17:34 min

FOSS4G 2021 - Visual Field
A look beyond the edge with Visual Field
15:36 min

Repo

https://github.com/harrishudson/VisualField

FOSS4G 2021

Harris, the author of Visual Field, will be giving a short talk about Visual Field in this years FOSS4G 2021 online meetup - with this years focus on Buenos Aires Argentina. Starting with a brief introduction, then a short demonstration, and finishing with what you can do - how to get involved. Whether you are technical or non-technical, if you work with Open Data, data manipulation and visualizations, you may find this talk of interest to you. A link to the video of the talk will be placed here after September 30 2021. Hope to see you there - September 30 2021. To watch the video - see Tutorial "FOSS4G 2021" above.

Get Involved - What Can You Do - Raise Your Voice

WebSQL (browser based SQLite database variant) has had an uncertainty cloud overhead for it's whole lifespan since it's introduction in 2009. There was a W3 impasse, in relation to standards support of WebSQL, in 2010 that persists today. Some browsers support WebSQL, some do not, some browsers have it as an experimental feature, some browsers removed the full text search (FTS) capability from WebSQL over security concerns, some browsers don't support outer joins, some browsers removed support in third party contexts, some are intending to remove other features of WebSQL, and some have future plans to deprecate and remove it entirely. WebSQL is becoming mangled. The Web Ecosystem has continued to evolve since 2010 - things like CORS HTTP headers, the ubiquity of javascript visualization frameworks, and the ubiquity of Open Data CSV datasets - now makes browser based processing of relational style data more important than ever. WebSQL isn't perfect. But to remove WebSQL now from browsers without replacing it with a standards based alternative SQL engine would be a real travesty. Half of the worlds population now have a smartphone (as at 2021) and more than 80% do in 2023. In many use cases now (for processing Open Data) - WebSQL means there is no longer a need for back-end, or cloud, databases for processing datasets in many cases. For those browsers that support it, WebSQL brings these benefits over other browser based storage mechanisms;

It's embedded with Zero installation
It can handle big data (Web Storage cannot for example)
It's incredibly fast to load, index and query
It can interpret end-user input as SQL (ie user input to interact with the storage layer). IndexedDB cannot for example - that's javascript based - and not everyone is a javascript programmer
It can perform exceptionally well for an arbitrary relational join query (multiple tables). IndexedDB cannot for example which requires data extraction to be implemented as code (javascript)
It performs incredibly fast overall. An SQL framework built upon another storage model will NOT. Eg, SQL on top of IndexedDB will not match the performance of WebSQL
WebSQL supports the construction of views (not just tables) Eg; non-persistent stored queries

NOW!

Author

Donate

It has taken a significant effort to develop this application. If you find it useful in your project - consider making a donation. Donations will go directly towards continued development efforts. Whilst there remains incomplete browser support, and incomplete standards support for WebSQL, it's not really possible to provide any sort of product support but its hoped that you still find this application useful none-the-less. Thanks for your support. Donation and Sponsorship opportunities are available by contacting the author directly.

Please, download your copy of Visual Field, and do not link your production web applications to the vf.html page hosted on this website.