Quickstart#
This notebook shows some very simple example of getting started.
First we need to install some dependancies into your conda or venv environment
!pip install pandas matplotlib fsspec s3fs "zarr<3" pyarrow xarray
Show code cell output
Requirement already satisfied: pandas in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2.0.3)
Requirement already satisfied: matplotlib in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (3.9.0)
Requirement already satisfied: fsspec in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2024.5.0)
Requirement already satisfied: s3fs in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2024.5.0)
Requirement already satisfied: zarr<3 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2.15.0)
Requirement already satisfied: pyarrow in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (12.0.1)
Requirement already satisfied: xarray in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (2023.7.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas) (2024.1)
Requirement already satisfied: numpy>=1.21.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from pandas) (1.25.2)
Requirement already satisfied: contourpy>=1.0.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (4.53.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (24.0)
Requirement already satisfied: pillow>=8 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (10.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from matplotlib) (3.1.2)
Requirement already satisfied: aiobotocore<3.0.0,>=2.5.4 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from s3fs) (2.13.0)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from s3fs) (3.9.5)
Requirement already satisfied: asciitree in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr<3) (0.3.3)
Requirement already satisfied: fasteners in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr<3) (0.19)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from zarr<3) (0.12.1)
Requirement already satisfied: botocore<1.34.107,>=1.34.70 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (1.34.106)
Requirement already satisfied: wrapt<2.0.0,>=1.10.10 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (1.16.0)
Requirement already satisfied: aioitertools<1.0.0,>=0.5.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (0.11.0)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.9.4)
Requirement already satisfied: six>=1.5 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from botocore<1.34.107,>=1.34.70->aiobotocore<3.0.0,>=2.5.4->s3fs) (1.0.1)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from botocore<1.34.107,>=1.34.70->aiobotocore<3.0.0,>=2.5.4->s3fs) (2.2.1)
Requirement already satisfied: idna>=2.0 in /Users/rt2549/miniconda3/envs/mast/lib/python3.11/site-packages (from yarl<2.0,>=1.0->aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (3.7)
import pandas as pd
URL = 'https://mastapp.site'
List All Shots#
shots_df = pd.read_parquet(f'{URL}/parquet/level2/shots')
shots_df
| context | url | preshot_description | postshot_description | campaign | current_range | divertor_config | plasma_shape | commissioner | facility | ... | cpf_vol_ipmax | cpf_vol_max | cpf_vol_truby | cpf_wmhd_ipmax | cpf_wmhd_max | cpf_wmhd_truby | cpf_zeff_ipmax | cpf_zeff_max | cpf_zeff_truby | cpf_zmag_efit | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/11766.zarr | \nRELOAD 10232 PLASMA CONDITIONING SHOT FROM M4\n | \nGOOD PLASMA, RAN FINE. SL JOINT ALARMS RATHE... | M5 | None | Conventional | None | None | MAST | ... | 8.088068 | 8.478948 | NaN | 9286.269388 | 14370.632 | NaN | NaN | NaN | NaN | -0.011828 |
| 1 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/11767.zarr | \nREPEAT\n | \nOK BUT LOST VERTICAL CONTROL - FA2 JUST DIED... | M5 | None | Conventional | None | None | MAST | ... | 7.707199 | 8.559942 | NaN | 7343.788162 | 11461.490 | NaN | NaN | NaN | NaN | -0.007570 |
| 2 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/11768.zarr | \nSELECT ALL 4 FAS AND LOAD EFC SETTINGS FROM ... | \nOK. GOT FA4 BUT NOT FA3\n | M5 | None | Conventional | None | None | MAST | ... | 8.300782 | 8.580827 | NaN | 10585.928086 | 12660.081 | NaN | NaN | NaN | NaN | -0.011107 |
| 3 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/11769.zarr | \nREPEAT, WITH TF RAISED TO 0.58.\n | \nSLIDING JOINT ALARMS A BIT LOWER. PLASMA OK.\n | M5 | None | Conventional | None | None | MAST | ... | 8.112973 | 8.742647 | NaN | 8189.273726 | 12494.456 | NaN | NaN | NaN | NaN | -0.008156 |
| 4 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/11771.zarr | \nRELOADED 11769 AND SET UP PLASMA CURRENT FEE... | \nGOOD PLASMA F/B CONTROL. SLIDING JOINT ALARM... | M5 | None | Conventional | None | None | MAST | ... | 7.978917 | 8.760519 | NaN | 8206.258460 | 12345.816 | NaN | NaN | NaN | NaN | -0.008726 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 11276 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/30467.zarr | \nRepeat with new neutron camera position.\ncH... | \nTwo times lower DD neutron rate than referen... | M9 | 700 kA | Conventional | Connected Double Null | None | MAST | ... | 9.029202 | 9.046394 | NaN | 49469.122469 | 52653.445 | NaN | NaN | NaN | NaN | 0.013202 |
| 11277 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/30468.zarr | \nRepeat with new neutron camera position.\ncH... | \nGood beam.\nGood repeat.\n | M9 | 700 kA | Conventional | Lower Single Null | None | MAST | ... | 9.102411 | 9.107017 | NaN | 48516.962675 | 49382.133 | NaN | NaN | NaN | NaN | 0.012445 |
| 11278 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/30469.zarr | \nRepeat with increased beam power (74 kV)\ncH... | \nGood shot. Modes present.\n | M9 | 700 kA | Conventional | Connected Double Null | None | MAST | ... | 8.988730 | 9.047923 | NaN | 47466.249616 | 49115.805 | NaN | NaN | NaN | NaN | 0.015299 |
| 11279 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/30470.zarr | \nRepeat last using hydrogen in outboard and c... | \nNo HF gas.\n | M9 | 700 kA | Conventional | None | None | MAST | ... | 9.687049 | 10.055509 | NaN | 17290.432865 | 22310.516 | NaN | NaN | NaN | NaN | 0.015164 |
| 11280 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | s3://mast/level2/shots/30471.zarr | \nThe last plasma:\nConvert to i/b Helios 1724... | \nGood shot.\n | M9 | 700 kA | Conventional | Lower Single Null | None | MAST | ... | 8.817559 | 9.283702 | NaN | 38063.582380 | 40906.090 | NaN | NaN | NaN | NaN | 0.014340 |
11281 rows × 290 columns
List All Sources#
sources_df = pd.read_parquet(f"{URL}/parquet/level2/sources")
sources_df
| context | description | quality | type | title | uuid | name | url | endpoint_url | imas | shot_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 5bdcd8ad-146e-5ce1-ad35-ddac06a1aac1 | gas_injection | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | gas_injection | 12063 | |
| 1 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 1323ad31-76bc-5572-97a7-7757e42b536d | magnetics | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | magnetics | 12063 | |
| 2 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 841485f9-b1fa-574f-9ff0-049e43fcd975 | pf_active | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | pf_active | 12063 | |
| 3 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | f856e986-0b9f-5078-bd2b-3bd112eedac0 | pulse_schedule | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | pulse_schedule | 12063 | |
| 4 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 02ec1e70-5223-5142-be82-08d30e546acf | soft_x_rays | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | None | 12063 | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 103161 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 60df2715-71a8-5d86-9c7b-a56074bded07 | soft_x_rays | s3://mast/level2/shots/12058.zarr | https://s3.echo.stfc.ac.uk | None | 12058 | |
| 103162 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 56f896cb-1510-5dcf-ae12-2eaeab2ebb5d | spectrometer_visible | s3://mast/level2/shots/12058.zarr | https://s3.echo.stfc.ac.uk | None | 12058 | |
| 103163 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 2832df7d-5884-503d-9e02-6ee3fb58868c | summary | s3://mast/level2/shots/12058.zarr | https://s3.echo.stfc.ac.uk | summary | 12058 | |
| 103164 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 081fcd86-e34a-500c-89e6-e79aa16ce884 | thomson_scattering | s3://mast/level2/shots/12058.zarr | https://s3.echo.stfc.ac.uk | thomson_scattering | 12058 | |
| 103165 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | Not Checked | dcat:Dataset | Source Dataset | 2421fbad-1964-52cd-895f-ac7fcf726087 | equilibrium | s3://mast/level2/shots/12063.zarr | https://s3.echo.stfc.ac.uk | equilibrium | 12063 |
103166 rows × 11 columns
List all signals Given a Shot Number#
signals_df = pd.read_parquet(f"{URL}/parquet/level2/signals?shot_id=30421")
signals_df
| context | shape | provenance | description | dimensions | quality | type | title | uuid | name | version | rank | url | endpoint_url | source | units | imas | shot_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | e73e1add-b722-59e4-a732-0b60c3216a07 | vloop_static | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 1 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 10ad9b43-953c-5425-ab37-2086971006ee | whmd | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 2 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [4, 146] | None | [n_x_points, time] | Not Checked | dcat:Dataset | Signal Dataset | 140a4646-3bfd-565d-a20c-b6fed0c78a01 | x_point_r | 0 | 2 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 3 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [4, 146] | None | [n_x_points, time] | Not Checked | dcat:Dataset | Signal Dataset | 48ad8afd-8d73-5d7e-b7af-1d6395b3d5f2 | x_point_z | 0 | 2 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 4 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [2906] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 14f84715-826c-5fa4-9cfe-c14a69a63035 | inboard_total | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | gas_injection | count | gas_injection | 30421 | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 68 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 0972a302-29e8-5395-9cfe-7057e83a9c16 | rpsi95_in | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 69 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 58f0142e-1f9c-5fdf-a7ff-b4b89f1eb908 | rpsi95_out | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 70 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 144e84db-d0f2-5941-86f5-23582e4d33e4 | triangularity_lower | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 71 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 14802b5a-f4b1-56a8-9322-1b35626cc03c | triangularity_upper | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 | ||
| 72 | {'dcat': 'http://www.w3.org/ns/dcat#', 'dct': ... | [146] | None | [time] | Not Checked | dcat:Dataset | Signal Dataset | 051fe6c2-7a44-53cf-95ce-17bcab43e7f7 | vloop_dynamic | 0 | 1 | s3://mast/level2/shots/30421.zarr | https://s3.echo.stfc.ac.uk | equilibrium | equilibrium | 30421 |
73 rows × 18 columns
Open and plot shot data.#
Here we are visualising some plasma current data.
import s3fs
import zarr
import xarray as xr
import matplotlib.pyplot as plt
endpoint_url = 'https://s3.echo.stfc.ac.uk'
url = f's3://mast/level2/shots/30421.zarr'
fs = s3fs.S3FileSystem(anon=True, endpoint_url=endpoint_url)
store =zarr.storage.FSStore(fs=fs, url=url)
dataset = xr.open_zarr(store, group='summary')
dataset['ip'].plot()
[<matplotlib.lines.Line2D at 0x36c2e3490>]
