Quickstart#
This notebook shows some very simple example of getting started.
First we need to install some dependancies into your conda or venv environment
!pip install pandas matplotlib zarr fsspec s3fs intake intake_xarray intake_parquet
Show code cell output
Requirement already satisfied: pandas in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2.2.2)
Requirement already satisfied: matplotlib in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (3.9.0)
Requirement already satisfied: zarr in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2.18.2)
Requirement already satisfied: fsspec in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2024.6.0)
Requirement already satisfied: s3fs in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2024.6.0)
Requirement already satisfied: intake in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2.0.5)
Requirement already satisfied: intake_xarray in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (0.7.0)
Requirement already satisfied: intake_parquet in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (0.3.0)
Requirement already satisfied: numpy>=1.23.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (1.26.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2.9.0)
Requirement already satisfied: pytz>=2020.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2024.1)
Requirement already satisfied: contourpy>=1.0.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (4.53.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (24.1)
Requirement already satisfied: pillow>=8 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (10.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from matplotlib) (3.1.2)
Requirement already satisfied: asciitree in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.3.3)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.12.1)
Requirement already satisfied: fasteners in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.19)
Requirement already satisfied: aiobotocore<3.0.0,>=2.5.4 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from s3fs) (2.13.0)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from s3fs) (3.9.5)
Requirement already satisfied: pyyaml in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake) (6.0.1)
Requirement already satisfied: appdirs in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake) (1.4.4)
Requirement already satisfied: xarray>=02022 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_xarray) (2024.5.0)
Requirement already satisfied: dask>=2.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_xarray) (2024.5.2)
Requirement already satisfied: netcdf4 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_xarray) (1.6.5)
Requirement already satisfied: msgpack in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_xarray) (1.0.8)
Requirement already satisfied: requests in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_xarray) (2.32.3)
Requirement already satisfied: fastparquet in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_parquet) (2024.5.0)
Requirement already satisfied: pyarrow in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from intake_parquet) (16.1.0)
Requirement already satisfied: botocore<1.34.107,>=1.34.70 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (1.34.106)
Requirement already satisfied: wrapt<2.0.0,>=1.10.10 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (1.16.0)
Requirement already satisfied: aioitertools<1.0.0,>=0.5.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiobotocore<3.0.0,>=2.5.4->s3fs) (0.11.0)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->s3fs) (1.9.4)
Requirement already satisfied: click>=8.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from dask>=2.2->intake_xarray) (8.1.7)
Requirement already satisfied: cloudpickle>=1.5.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from dask>=2.2->intake_xarray) (3.0.0)
Requirement already satisfied: partd>=1.2.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from dask>=2.2->intake_xarray) (1.4.2)
Requirement already satisfied: toolz>=0.10.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from dask>=2.2->intake_xarray) (0.12.1)
Requirement already satisfied: importlib-metadata>=4.13.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from dask>=2.2->intake_xarray) (7.1.0)
Requirement already satisfied: six>=1.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Requirement already satisfied: cramjam>=2.3 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from fastparquet->intake_parquet) (2.8.3)
Requirement already satisfied: cftime in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from netcdf4->intake_xarray) (1.6.4)
Requirement already satisfied: certifi in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from netcdf4->intake_xarray) (2024.6.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from requests->intake_xarray) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from requests->intake_xarray) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from requests->intake_xarray) (2.2.1)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from botocore<1.34.107,>=1.34.70->aiobotocore<3.0.0,>=2.5.4->s3fs) (1.0.1)
Requirement already satisfied: zipp>=0.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from importlib-metadata>=4.13.0->dask>=2.2->intake_xarray) (3.19.2)
Requirement already satisfied: locket in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from partd>=1.2.0->dask>=2.2->intake_xarray) (1.0.0)
Open the intake Catalog#
import intake
catalog = intake.open_catalog('https://mastapp.site/intake/catalog.yml')
catalog
intake:
args:
path: https://mastapp.site/intake/catalog.yml
description: ''
driver: intake.catalog.local.YAMLFileCatalog
metadata:
version: 1
List All Shots#
shots_df = catalog.index.level1.shots().read()
shots_df
| url | preshot_description | postshot_description | campaign | current_range | divertor_config | plasma_shape | comissioner | facility | shot_id | ... | cpf_vol_ipmax | cpf_vol_max | cpf_vol_truby | cpf_wmhd_ipmax | cpf_wmhd_max | cpf_wmhd_truby | cpf_zeff_ipmax | cpf_zeff_max | cpf_zeff_truby | cpf_zmag_efit | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | s3://mast/level1/shots/11695.zarr | \n0.1T TF SHOT\n | \nOK\n | M5 | None | Conventional | None | None | MAST | 11695 | ... | NaN | NaN | NaN | NaN | NaN | NaN | None | None | None | NaN |
| 1 | s3://mast/level1/shots/11696.zarr | \nSTANDARD 0.3T TF SHOT\n | \nOK\n | M5 | None | Conventional | None | None | MAST | 11696 | ... | NaN | NaN | NaN | NaN | NaN | NaN | None | None | None | NaN |
| 2 | s3://mast/level1/shots/11697.zarr | \nRAISE TO 0.5T\n | \nOK, ALARMS ARE LOWER\n | M5 | None | Conventional | None | None | MAST | 11697 | ... | NaN | NaN | NaN | NaN | NaN | NaN | None | None | None | NaN |
| 3 | s3://mast/level1/shots/11698.zarr | \nRAISE TO .56T\n | \nSTILL ALARMS BUT LOWER AGAIN\n | M5 | None | Conventional | None | None | MAST | 11698 | ... | NaN | NaN | NaN | NaN | NaN | NaN | None | None | None | NaN |
| 4 | s3://mast/level1/shots/11699.zarr | \nRAISE TO .58T\n | \nOK\n | M5 | None | Conventional | None | None | MAST | 11699 | ... | NaN | NaN | NaN | NaN | NaN | NaN | None | None | None | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 15548 | s3://mast/level1/shots/30467.zarr | \nRepeat with new neutron camera position.\ncH... | \nTwo times lower DD neutron rate than referen... | M9 | 700 kA | Conventional | Connected Double Null | None | MAST | 30467 | ... | 9.029202 | 9.046394 | 0.0 | 49469.122469 | 52653.445 | 0.0 | None | None | None | 0.013202 |
| 15549 | s3://mast/level1/shots/30468.zarr | \nRepeat with new neutron camera position.\ncH... | \nGood beam.\nGood repeat.\n | M9 | 700 kA | Conventional | Lower Single Null | None | MAST | 30468 | ... | 9.102411 | 9.107017 | 0.0 | 48516.962675 | 49382.133 | 0.0 | None | None | None | 0.012445 |
| 15550 | s3://mast/level1/shots/30469.zarr | \nRepeat with increased beam power (74 kV)\ncH... | \nGood shot. Modes present.\n | M9 | 700 kA | Conventional | Connected Double Null | None | MAST | 30469 | ... | 8.988730 | 9.047923 | 0.0 | 47466.249616 | 49115.805 | 0.0 | None | None | None | 0.015299 |
| 15551 | s3://mast/level1/shots/30470.zarr | \nRepeat last using hydrogen in outboard and c... | \nNo HF gas.\n | M9 | 700 kA | Conventional | None | None | MAST | 30470 | ... | 9.687049 | 10.055509 | 0.0 | 17290.432865 | 22310.516 | 0.0 | None | None | None | 0.015164 |
| 15552 | s3://mast/level1/shots/30471.zarr | \nThe last plasma:\nConvert to i/b Helios 1724... | \nGood shot.\n | M9 | 700 kA | Conventional | Lower Single Null | None | MAST | 30471 | ... | 8.817559 | 9.283702 | 0.0 | 38063.582380 | 40906.090 | 0.0 | None | None | None | 0.014340 |
15553 rows × 282 columns
List All Sources#
sources_df = catalog.index.level1.sources().read()
sources_df
| description | quality | uuid | shot_id | name | url | |
|---|---|---|---|---|---|---|
| 0 | Plasma Current and PF/TF Coil Currents | Not Checked | 8d2954b9-6059-5b92-a0d9-da514dd93f64 | 11695 | amc | s3://mast/level1/shots/11695.zarr/amc |
| 1 | Basic EFIT | Not Checked | fd0a0dc4-1ed8-546f-8c02-455061374fd4 | 11695 | efm | s3://mast/level1/shots/11695.zarr/efm |
| 2 | Plasma Current and PF/TF Coil Currents | Not Checked | 2ac9a6d3-cc69-5979-9476-52fdf1f9bd25 | 11696 | amc | s3://mast/level1/shots/11696.zarr/amc |
| 3 | Basic EFIT | Not Checked | c553a338-249f-58d5-9ba8-14f5c507c4e4 | 11696 | efm | s3://mast/level1/shots/11696.zarr/efm |
| 4 | Plasma Current and PF/TF Coil Currents | Not Checked | 9b1e5fd1-9dd4-5a30-b7c4-d7fd2ca0f114 | 11697 | amc | s3://mast/level1/shots/11697.zarr/amc |
| ... | ... | ... | ... | ... | ... | ... |
| 99069 | Phantom colour camera | Not Checked | 3ab8c113-1d6e-5b28-b3bc-253520c96d02 | 30471 | rco | s3://mast/level1/shots/30471.zarr/rco |
| 99070 | RBG 2D multi-colour visible bremsstrahlung camera | Not Checked | 85416bc2-faa1-5753-8541-0d27f68434bc | 30471 | rgb | s3://mast/level1/shots/30471.zarr/rgb |
| 99071 | RBG 2D multi-colour visible bremsstrahlung camera | Not Checked | a94efa26-63b2-517f-afa5-08b95c309b96 | 30471 | rgc | s3://mast/level1/shots/30471.zarr/rgc |
| 99072 | Magnetic Field Measurements: OMAHA high freque... | Not Checked | ffffb16c-447b-579e-8798-4e6731539c73 | 30471 | xmo | s3://mast/level1/shots/30471.zarr/xmo |
| 99073 | SRX Camera | Not Checked | 37d9e4b0-29db-5a4a-bdef-609212693b9a | 30471 | xsx | s3://mast/level1/shots/30471.zarr/xsx |
99074 rows × 6 columns
List all signals Given a Shot Number#
signals_df = catalog.index.level1.signals(shot=30420).read()
signals_df
| quality | shape | provenance | description | signal_type | dimensions | uuid | shot_id | name | version | rank | url | source | units | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Not Checked | [] | None | Analysed | [] | c68811ac-f71c-57dd-ad68-54d2df6e0075 | 30420 | efm/nh | 0 | 0 | s3://mast/level1/shots/30420.zarr/efm/nh | efm | None | |
| 1 | Not Checked | [30000] | None | Analysed | [time] | fbfe5693-2fa1-577b-8567-76261174b0c4 | 30420 | amc/p4u_feed_current | 0 | 1 | s3://mast/level1/shots/30420.zarr/amc/p4u_feed... | amc | None | |
| 2 | Not Checked | [] | None | Analysed | [] | afb7d470-1afb-57ab-9f8b-7a1177f28839 | 30420 | abm/calib_shot | 0 | 0 | s3://mast/level1/shots/30420.zarr/abm/calib_shot | abm | None | |
| 3 | Not Checked | [32] | None | Analysed | [dim_0] | 9d4d7d90-99e2-56cb-831c-8b64a7a23244 | 30420 | abm/channel_status | 0 | 1 | s3://mast/level1/shots/30420.zarr/abm/channel_... | abm | None | |
| 4 | Not Checked | [32] | None | Analysed | [dim_0] | c3405cd4-06cc-59ba-b6f7-65e5032e7e9b | 30420 | abm/channel_type | 0 | 1 | s3://mast/level1/shots/30420.zarr/abm/channel_... | abm | None | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1850 | Not Checked | [36, 300000] | None | Raw | [hcam_l_channel, time] | cdbb8a91-8e3a-5411-9cea-670bc546dca0 | 30420 | hcam_l | -1 | 2 | s3://mast/level1/shots/30420.zarr/hcam_l | xsx | None | |
| 1851 | Not Checked | [36, 300000] | None | Raw | [hcam_u_channel, time] | 6cfe584f-276a-5c79-9612-0ee4de158919 | 30420 | hcam_u | -1 | 2 | s3://mast/level1/shots/30420.zarr/hcam_u | xsx | None | |
| 1852 | Not Checked | [300000] | None | Raw | [time] | 30084190-10ca-5cbb-b9fb-48216c98304f | 30420 | xsx/relay_monitor | -1 | 1 | s3://mast/level1/shots/30420.zarr/xsx/relay_mo... | xsx | None | |
| 1853 | Not Checked | [36, 300000] | None | Raw | [tcam_channel, time] | 10170616-9105-5a61-8077-4770560c0295 | 30420 | tcam | -1 | 2 | s3://mast/level1/shots/30420.zarr/tcam | xsx | None | |
| 1854 | Not Checked | [300000] | None | Raw | [time] | 52bb8b25-7c82-5313-8fd3-c42a95f41fba | 30420 | xsx/time1 | -1 | 1 | s3://mast/level1/shots/30420.zarr/xsx/time1 | xsx | None |
1855 rows × 14 columns
Open and plot shot data.#
Here we are visualising some plasma current data.
import matplotlib.pyplot as plt
dataset = catalog.level1.sources(url="s3://mast/level1/shots/30420.zarr/amc").to_dask()
plt.plot(dataset['time'], dataset['plasma_current'])
/Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
'dims': dict(self._ds.dims),
[<matplotlib.lines.Line2D at 0x175cfe0d0>]
