Visual Field  

Explore, Process and Visualize your data right in your browser.

June 2024 Update

RIP WebSQL

WebSQL was a HTML standards technology simply ahead of its time.

With the recent release of Chromium v125 in June 2024, WebSQL appears to have now been fully deprecated in Chromium based browsers. A shenanigans of a deprecation trial commenced in Chromium v119 in November 2023 - where the ultimate doomed fate of WebSQL was set. Now, mainstream browsers are unlikely to support WebSQL going forward. WebSQL was always a hot potato and a political mess in terms of HTML standards. Granted - there was merit to the fact that WebSQL should perhaps not have been based on a specific underlying technology. And yes, there were security issues with the underlying implementation of WebSQL. However, such security issues were never truly addressed. Instead, those WebSQL components with security vulnerabilities were simply progressively removed or omitted from the browser. Instead of addressing the core problems, functionality of WebSQL was progressively removed from browsers since its introduction in 2009 to a point the technology was mangled and almost rendered hopeless. You can see a rant by the author, Harris Hudson - see "Get Involved" section below. Now, SQLite as a wasm port has been put forward as the replacement to WebSQL by the browser developer community. And in certain respects, that has a promising future - although anecdotally there may be use cases where it doesn't quite match the performance of WebSQL particularly for complex joins of large datasets. But even if SQLite wasm is a workable alternative - it misses the primary point when it comes to HTML web standards. The need for a HTML standards based relational database built natively in to the browser still exists today (June 2024). Yes, WebSQL might have been becoming unmanageable but now, a SQLite wasm port doesn't solve this need for standards. Where are the standards? Hopefully sometime in the future there will come a time again when the broader community can realize that a standards based relational (probably SQL) engine built into browsers will be a positive move. Not everyone is a javascript programmer and that fact appears to be overlooked by the browser developer community. WebSQL enabled the end user to interact with the storage layers using SQL. The Web Ecosystem continues to evolve and large CORS enabled open relational datasets are now ubiquitous on the web. Yes, WebSQL was not perfect - but the political mess and the mangling by successive removal of its technology components from browsers over the years since its introduction and ultimate deprecation - well, that was simply an abomination! Harris Hudson, the author of Visual Field, is currently looking into porting Visual Field to a SQLite wasm port, but with such a large code base - this is a significant undertaking. If Harris can rework Visual Field as a SQLite wasm port - there would be the potential for a more broader browser base support. Although such a port still won't address the elephant in the room - that is the glaring omission of HTML standards for a browser based relational database. Stay tuned.

Whilst not really recommending the use of older software, if you would like to see the following WebSQL Visual Field examples below in the page function correctly, you may need to use an older browser that supports WebSQL. For Chromium based browsers, you will need to look at a version prior to version v119 - when the deprecation trial shenanigans began for WebSQL. A possible commit token for an arbitrary Chromium v118 version (prior to the WebSQL deprecation) is 1185669. Whilst not associated with this website in any way, if you would like to see the examples below in action, you may be able to download an older Chromium based browser supporting WebSQL from here;

https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html

For example;

Windows x86 download;
https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/1185669/
Linux x64 Build download;
https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Linux_x64/1185669/

Run Now or Download

Current Version (3.4.0) Run Now; vf.html, Download; (1013 kB, 164 kB gzipped)

integrity="sha384-Elej0LuLYWgkFmbSDN1tijZIrA9d7l1PAI7WluiCnJJRCXwkjT4GWkFEGWaLQbbr"

Demo

Example 1
Example 1
SF Schools by CCSF
A Basic Map
Example 3
Example 2
SF Schools Lead (5Ug/l)
Multi jurisdictional data sets. Map.
Example 3
Example 3
SF Schools Hazard Zones
Multi data sets. Custom Function. Map.
Example 4
Example 4
SF Crime Incidents
Mini data driven App. Cache Max-Age. Charts. Map. Drilldown.
Example 5
Example 5
ACT Cycle Crashes
Heat Map.
Example 6
Example 6
NYC Vehicle Collisions Person
Very Large Dataset. Table Visualization.
Example 7
Example 7
NYC Collision Death Rate
Multi series Chart.
Example 8
Example 8
SF Covid Choropleth
An SQL generated Choropleth Map.
Example 9
Example 9
NYC Covid Cases
Multi time series chart.
Example 10
Example 10
SF Street Network
Multi data sets. Custom Function. Shortest Path. Physical Network.
Example 11
Example 11
Sample Logical Network.
Example 12
Example 12
Advanced Loading Techniques.
Simple Table Visualization.
Example 15
Example 15
SF Muni Simple Routes.
Map Visualisation.

For these examples, your Visual Field metadata is not reset and will just be added to cumulatively. Reseting your metadata between each example, may make it easier for you to understand individual example configuration. Go; Dictionary Reset Metadata Reset. Press for a Help Page about each example.

What is Visual Field

Visual Field is essentially an Open Source single html file - vf.html, with some other associated files to support different usage modes. It enables you to consume and process delimited data files and web services (eg; CSV).

Change Log

Version 3.4.0 (Current Version) - Released September 2021
  • Gracefully cater for database objects that have no ROWID field
  • Numerous minor bugs corrected

Version 3.3.0 - Released August 2021
  • integrity="sha384-wpGCiOi+e2oOGx1o6BeI5NTwR/EEdOATeWMuiFMAKMo7hGcNX2BsLvCz6EaxAcpA"
  • Added some configurable Internal Settings to easily modify METADATA Query String parameter processing behaviour
  • Updated versions of some dependent libraries

Version 3.2.0 - Released July 2021
  • integrity="sha384-tsku+07BWvp1CuKY+rWnS7Gv3VdXy5NeouzMxkOsaFSYO7RhkiT+s+N+RL4p/VMR"
  • Updated versions of core dependent libraries
  • Added Custom Function; Turf pointsWithinPolygon
  • Added Pan and Zoom to Chart Visualizations

Version 3.1.0 - Released June 2021
  • integrity="sha384-NiNy5NgO0lS3TnjBkq+0SBpndgSJaQte71J6dwBZPTkJRKVaRbbGB9FiBTSSjNat"
  • Updated versions of most core dependent libraries
  • Improved code formatting and readability
  • Added Custom Functions; Turf boolean.intersect, Turf polygonTangents and Turf polygonSmooth
  • Added Scatter charts to Chart Visualizations
  • Added PWA "Install" Add to Home Screen (A2HS) functionality
  • Rectified numerous minor bugs

Version 3.0.0 - Released October 2020
  • integrity="sha384-mhbuDuPmNigHOuV7dS7w5bfDh4HTe5xzDx5O5z8+zjIC4Mv+/fmmfxTP0ICZnw3o"
  • Added Network and HeatMap Visualizations
  • Added String and Proj4js Custom Functions
  • Added Save/Restore Network Graph and view Sequence Run Log
  • Numerous minor bugs corrected and general UI tidy up
Version 2.0.0 - Released May 2020
  • integrity="sha384-Ld5rtVWf4tthw3R1xDmCIbnaFulG+E1c9DzgZn/YVPqPGx8ozYj/Pw06m0mWzzBo"
  • Added Multi-Media support - Import your Image, Audio and short Videos in to your database
  • Photo Map - view a map of your geotagged images
  • New Custom Function to allow easy resize of your images
  • New Media Visualization - easily create a context rich slide show of your media
  • Added ability to update your data directly as displayed in page in table browser, table visualization or map visualization
  • Export Dump your tables or entire database and import directly into another HTML5 WebSQL browser with Visual Field
  • SQL Dump your tables or entire database so you can import that into a separate database (eg; PostgreSQL)
  • Filter Browse results by individual table fields
  • Updated Chart Visualizations for Time Series
  • Numerous minor bugs corrected
Version 1.0.0 - Released Dec 2019
Initial Release

Usage

Not all current browsers support HTML5 and WebSQL. And of those browsers that do, not all underlying devices support a WebSQL database when invoked as a standalone html file. The browsers currently supporting a fully WebSQL database is still somewhat fairly dynamic and changing (even as at 2019). See "Get Involved" section below.

Application Usage

Visual Field is intended to be able to be used in any of the following ways;

Optional Support Files

Online Web Application

For an online web application, just install the vf.html file in your intended web server directory. This will allow Visual Field on your web server to be used in an online way.

Offline Web Application

For an offline web application, you will need to install both vf.html and the vf_service_worker.js files in your intended web server directory. This will allow Visual Field to be loaded in a browser even if your users do not have an internet connection (after initial load). Both the vf.hml and vf_service_worker.js will need to reside in the same directory.

As a stand alone html file

Visual Field can function as a stand alone html file just by loading the vf.html file in your browser using a file:// protocol context. That is, no web server is necessary to use the full functions of Visual Field. This usage option may be useful if you are unable to install Visual Field on a web server or unable to install it as a Progressive Web Application.

As an Installed Progressive Web Application

Progressive Web Applications are intended to be able to be installed on your device by using HTML component "Add to Home Screen" (A2HS). As of version 3.1.0, Visual Field is typically able to be installed on your device by a simple click on the "Install" option under the "Help" tab. This enables a quick, simple, non-technical install of Visual Field on to your users device. In order to support Visual Field as a Progressive Web Application, you will need to place vf.html, vf_service_worker.js, vf.webmanifest and the favicon.svg all 4 files on to your web server in the same directory. Should all these files be installed on your web server, and your web server can serve these over https, then end users should be able to successfully invoke the "Install" option, under the "Help" tab, to add Visual Field to their device as an installed Progressive Web Application. Ie, a simple, non-technical, install on their device.

Optional Offline Web Configuration - Example.

The vf.html file is intended to be able to be loaded as a stand alone file (from a file:// protocol context) or served via a web server. If you do serve it via a web server, you may wish to download and add the optional vf_service_worker.js file and place it in the same directory as vf.html. This will turn your served version of Visual Field into a true Offline Web Application in that it can be reloaded and used in a browser (from a https:// protocol context) without needing internet access. Something to consider if you make presentations, or you may be an emergency worker out in the field, with unreliable internet access.

To demonstrate the Offline Web capability, follow this scenario;

  1. Load Example1 (using a https:// protocol context) here; Example 1 (SF schools)
  2. From the Map, press the button to select a zoom level and download some Map tiles for offline use.
  3. Disconnect your device from the internet, by going to Airplane mode or similar.
  4. Reload the above link (from Step 1) and your Map and Schools data should be presented intact.

Tutorials

Tutorial 1
Tutorial 1
SF Schools by CCSF
14:08 min
Tutorial 2
Tutorial 2
Version 2 - What's new.
21:10 min
Tutorial 3
Tutorial 3
Building Example 10 shortest path.
17:34 min
FOSS4G 2021
FOSS4G 2021 - Visual Field
A look beyond the edge with Visual Field
15:36 min

Repo

https://github.com/harrishudson/VisualField

FOSS4G 2021

FOSS4G 2021 Harris, the author of Visual Field, will be giving a short talk about Visual Field in this years FOSS4G 2021 online meetup - with this years focus on Buenos Aires Argentina. Starting with a brief introduction, then a short demonstration, and finishing with what you can do - how to get involved. Whether you are technical or non-technical, if you work with Open Data, data manipulation and visualizations, you may find this talk of interest to you. A link to the video of the talk will be placed here after September 30 2021. Hope to see you there - September 30 2021. To watch the video - see Tutorial "FOSS4G 2021" above.

Get Involved - What Can You Do - Raise Your Voice

WebSQL (browser based SQLite database variant) has had an uncertainty cloud overhead for it's whole lifespan since it's introduction in 2009. There was a W3 impasse, in relation to standards support of WebSQL, in 2010 that persists today. Some browsers support WebSQL, some do not, some browsers have it as an experimental feature, some browsers removed the full text search (FTS) capability from WebSQL over security concerns, some browsers don't support outer joins, some browsers removed support in third party contexts, some are intending to remove other features of WebSQL, and some have future plans to deprecate and remove it entirely. WebSQL is becoming mangled. The Web Ecosystem has continued to evolve since 2010 - things like CORS HTTP headers, the ubiquity of javascript visualization frameworks, and the ubiquity of Open Data CSV datasets - now makes browser based processing of relational style data more important than ever. WebSQL isn't perfect. But to remove WebSQL now from browsers without replacing it with a standards based alternative SQL engine would be a real travesty. Half of the worlds population now have a smartphone (as at 2021) and more than 80% do in 2023. In many use cases now (for processing Open Data) - WebSQL means there is no longer a need for back-end, or cloud, databases for processing datasets in many cases. For those browsers that support it, WebSQL brings these benefits over other browser based storage mechanisms;

  1. It's embedded with Zero installation
  2. It can handle big data (Web Storage cannot for example)
  3. It's incredibly fast to load, index and query
  4. It can interpret end-user input as SQL (ie user input to interact with the storage layer). IndexedDB cannot for example - that's javascript based - and not everyone is a javascript programmer
  5. It can perform exceptionally well for an arbitrary relational join query (multiple tables). IndexedDB cannot for example which requires data extraction to be implemented as code (javascript)
  6. It performs incredibly fast overall. An SQL framework built upon another storage model will NOT. Eg, SQL on top of IndexedDB will not match the performance of WebSQL
  7. WebSQL supports the construction of views (not just tables) Eg; non-persistent stored queries

If you can see the benefit of WebSQL, or an alternate standards based SQL engine built right into the browser - to process your relational style datasets (eg; CSV) - then you need to get vocal with the developers of your favourite browser NOW!

Author

© 2019-2024 Harris Hudson.

Donate

It has taken a significant effort to develop this application. If you find it useful in your project - consider making a donation. Donations will go directly towards continued Open Source development efforts and potential rewrite of this app as a SQLite wasm implementation.

https://harrishudson.com/#sponsor

Map tiles © OpenStreetMap

Please, download your copy of Visual Field, and do not link your production web applications to the vf.html page hosted on this website.