Using pre-built user environments (VM images)
WEKEO offers several user environments with preconfigured software for data visualisation, data processing and software development in general. This page explains how to use these environments, and what is included in each one.
Pre-built user environments are available when you create a new VM in WEkEO. They have been designed to meet the needs of three user profiles:
- WEKEO data users, mostly related to the scientific and application domains that typically don’t develop new software but rather analyse data in specific application suites.
- Researchers or software developers that carry out advanced processing of data or write software code that leverages data accessible from WEKEO.
- WEKEO developers or administrators that needs special tools for platform building, integration, testing, administration etc.
Available environments
Currently there are 13 pre-built virtual environments, distributed in the following way:
- A Data Visualization and Analysis (VIS) designed for DIAS data users and available in five Linux flavours: OpenSUSE, CentOS, Ubuntu, Debian, and Scientific Linux.
- A Developer VM (DEV) also available in five Linux flavours, corresponding to five OSes. They inherit all the software from VIS and include additional software development suites. This is because the user who develops data-centric applications will likely need the same tools to visualise and analyse data as the rest of the users. The developer VM also includes the Tensorflow machine-learning engine, which is considered in par with other development tools. The virtual machine in particular contains the master node.
- A Tensorflow Processing Node VM (TF) (coming up soon) which only contains Tensorflow and its libraries. If the user wants to scale the processing in Tensorflow it creates several instances of such VM and, properly configured, will plug to the Developer VM containing the Tensorflow master instance as cluster nodes.
For a detailed description on what is included in each environment, please refer to section "Pre-installed tools" below.
Available operating systems
The Operating Systems offered on the pre-built user environments are the following:
| Operating system | VIS | DEV | TF |
|---|---|---|---|
| Ubuntu 18.04.02 LTS | X | X | X |
| Debian 9.8 | X | X | |
| Opensuse 15 | X | X | |
| Scientific Linux 7.6 | X | X | |
| Centos 7.6 | X | X |
Using pre-built user environments
In order to create a new VM based on one of the pre-built user environments, follow these steps:
As a registered user, go to Dashboard and click on Virtual Machines.
Navigate to Provisioning > Instances and click on +Add to create a new virtual machine instance.
Select the instance type VM Catalog and click on Next.
Configure the instance as per the image below. Version refers to the type of pre-built user environment (the BASE ones are empty, whereas DEV and VIS differ on the tools — see description below). For Plan, dedicated flavors start with wekeo.vmcatalog. The minimum storage size for VM Catalog images is 40 GB.
Pre-installed tools in each environment
| Item | Scope | VIS | DEV | TF | Description | URL | License |
|---|---|---|---|---|---|---|---|
| Science Toolbox Exploitation Platform | Application suites | X | X | This is the community platform that will accompany SNAP and the Sentinel Toolboxes starting now. | link | GPLv3 | |
| Orfeo Toolbox (OTB) | Application suites | X | X | Orfeo ToolBox (OTB) is an open-source project for state-of-the-art remote sensing. Built on the shoulders of the open-source geospatial community, it can process high resolution optical, multispectral and radar images at the terabyte scale. A wide variety of applications are available: from ortho-rectification or pansharpening, all the way to classification, SAR processing, and much more! All of OTB’s algorithms are accessible from Monteverdi, QGIS, Python, the command line or C++. Monteverdi is an easy to use visualization tool with an emphasis on hardware accelerated rendering for high resolution imagery (optical and SAR). | link | Apache 2.0 | |
| Broadview Altimetry Toolbox (BRAT) | Application suites | X | X | The Broadview Radar Altimetry Toolbox is a tool designed to use radar altimetry data. It is available in 32-bit and 64-bit versions for Windows, Mac OS X and Linux. The Broadview Radar Altimetry Tutorial and Toolbox is a joint project between ESA and CNES to develop an open source tool (GPL-3) freely available to all the altimetry community. | link | GPLv3 | |
| Panoply | Application suites | X | X | Panoply netCDF, HDF and GRIB Data Viewer Slice and plot geo-referenced latitude-longitude, latitude-vertical, longitude-vertical, time-latitude or time-vertical arrays from larger multidimensional variables. Slice and plot "generic" 2D arrays from larger multidimensional variables. Slice 1D arrays from larger multidimensional variables and create line plots. Combine two geo-referenced arrays in one plot by differencing, summing or averaging. Plot lon-lat data on a global or regional map using any of over 100 map projections or make a zonal average line plot. Overlay continent outlines or masks on lon-lat map plots. Use any of numerous color tables for the scale colorbar, or apply your own custom ACT, CPT, or RGB color table. Save plots to disk GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files. Export lon-lat map plots in KMZ format. Export animations as MP4 video or as a collection of invididual frame images. Explore remote THREDDS and OpenDAP catalogs and open datasets served from them. Panoply is a cross-platform application that runs on Macintosh, Windows, Linux and other desktop computers. | link | link | |
| ToolsUI | Application suites | X | User Interface to the netCDF-Java library The netCDF-Java library can read data from a wide range of scientific data formats (more details here). To check that your dataset can be read by netCDF-Java, use the "Viewer" tab in ToolsUI. Either cut-and-paste a URL or file path into the "dataset" field or click on the folder icon and select a file. (If you cut-and-paste, you will need to hit "Enter" to open the dataset.) | link | LGPL / (MIT-style) NetCDF C library license | ||
| QGIS Geographic Information System | Application suites | X | X | QGIS (previously known as Quantum GIS) is a free and open-source cross-platform desktop geographic information system (GIS) application that supports viewing, editing, and analysis of geospatial data. | link | GPL | |
| Python 2.7, Python 3 Programming Language | Languages - R + Python | X | link | PSF | |||
| R Programming Language | Languages - R + Python | X | link | various | |||
| Interactive Python (Jupyter) | Languages - R + Python | X | link | BSD 3-Clause License | |||
| Interactive R (Rstudio) | Languages - R + Python | X | link | AGPLv3.0 | |||
| Apache Zeppelin | Notebook | X | link | Apache 2.0 | |||
| Eclipse | IDEs & Editors | X | link | Eclipse Public License (EPL) | |||
| IntelliJ IDEA (Community Edition) | IDEs & Editors | X | link | Apache 2.0 | |||
| Visual Studio Code | IDEs & Editors | X | link | proprietary MIT for source code | |||
| Atom Editor | IDEs & Editors | X | link | MIT | |||
| Brackets Editor | IDEs & Editors | X | link | MIT | |||
| SciPy | Essential Python libraries and tools | X | link | propietary | |||
| Numpy | Essential Python libraries and tools | X | link | propietary | |||
| Pandas | Essential Python libraries and tools | X | link | BSD 3-Clause License | |||
| Ipython | Essential Python libraries and tools | X | link | propietary | |||
| MatPlotLib | Essential Python libraries and tools | X | link | propietary | |||
| Seabron | Essential Python libraries and tools | X | link | BSD 2-Clause License | |||
| Bokeh | Essential Python libraries and tools | X | link | BSD 2-Clause License | |||
| SciKit-learn | Essential Python libraries and tools | X | link | BSD New | |||
| Caret | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| Kernlab | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| randomForest | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| nnet | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| e1071 | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| klaR | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| tree | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| rpart | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| MICE | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| Lasso2 | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| Lars | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| Gbm | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| BayesTree | Essential R libraries and tools | X | link | various (see R packages license archive) | |||
| GDAL | Python remote sensing libraries | X | link | various | |||
| Georasters | Python remote sensing libraries | X | link | propietary | |||
| GeoPandas | Python remote sensing libraries | X | link | BSD 3-Clause "New" or "Revised" License | |||
| CLIMAF | Python remote sensing libraries | X | link | CECILL-2.0 | |||
| Fiona | Python remote sensing libraries | X | link | propietary | |||
| Fmask | Python remote sensing libraries | X | link | MIT | |||
| Shapely | Python remote sensing libraries | X | link | propietary | |||
| Cartopy | Python remote sensing libraries | X | link | LGPL | |||
| Rtree | Python remote sensing libraries | X | link | MIT | |||
| PySal | Python remote sensing libraries | X | link | propietary | |||
| NanSat | Python remote sensing libraries | X | link | GPLv3 | |||
| SatPy | Python remote sensing libraries | X | link | GPLv3 | |||
| Sp | R remote sensing libraries | X | various (see R packages license archive) | ||||
| Rgdal | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Fields | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Maptools | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Rstoolbox | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Sf | R remote sensing libraries | X | various (see R packages license archive) | ||||
| Rasters | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Gdistance | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Geosphere | R remote sensing libraries | X | link | various (see R packages license archive) | |||
| Sentinelsat | Copernicus / Sentinel libraries and utilities | X | X | Sentinelsat makes searching, downloading and retrieving the metadata of Sentinel satellite images from the Copernicus Open Access Hub easy. Sentinelsat is both a Python library and a standalone CLI tool | link | GPLv3 | |
| Sentinel2Productingestor | Copernicus / Sentinel libraries and utilities | X | X | Sentinel2ProductIngestor is used to download earth observation data provided by the http://www.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Sentinel-2 satellites and store it to Amazon S3 | link | MIT | |
| Sat-downloader | Copernicus / Sentinel libraries and utilities | X | X | A python library for download Satellite Imagery. Currently support Landsat-8 and Sentinel-2A. | link | MIT | |
| Sat-api | Copernicus / Sentinel libraries and utilities | X | Sat-api is a STAC compliant web API for searching and serving metadata for geospatial data (including but not limited to satellite imagery). | link | MIT | ||
| sentinelhub-py | Copernicus / Sentinel libraries and utilities | X | X | The package supports Python integration of working with Sentinel Hub services and retreiving data from AWS in SAFE format | link | MIT | |
| eo-learn | Copernicus / Sentinel libraries and utilities | X | X | Machine learning workflows, integration with Jupyter Notebook. | link | MIT | |
| Peps-download | Copernicus / Sentinel libraries and utilities | X | X | a simple piece of code to automatically download the products provided by the French Sentinel collaborative ground segment named PEPS : https://peps.cnes.fr. PEPS is mirroring all the Sentinel data provided by ESA, and is providing a simplified access. | link | N/A | |
| Tensorflow | Deep / machine learning | X | X | TensorFlow™ is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains. | link | Apache 2.0 |