REST API Examples

In this notebook we explore searching for metadata from the REST API. The REST API provides a method to programmatically extract a JSON representation of the meta data from the API.

First we load some python dependancies that we will use as part of this notebook and set the variable API_URL to the location of the REST API.

import pandas as pd
import requests
API_URL = "https://mastapp.site/json"

Below we print the location of the endpoint we will query in this notebook:

print(f"REST API Endpoint: {API_URL}")

REST API Endpoint: https://mastapp.site/json

Querying Shots with the REST API

We’re going to use the python requests library to query metadata from the database. All we need to do to get a result is to query the database with a HTTP GET at the appropriate endpoint. For example, to get information about different experimental shots we can query the /json/shots/ endpoint.

response = requests.get(f'{API_URL}/shots')
result = response.json()
print(f"Query returned status code: {response.status_code}")

Query returned status code: 200

The shots endpoint returns a JSON payload with a list of shots. Let’s look at the first element from the payload:

result['items'][0]

{'title': 'Shot Dataset',
 'shot_id': 11695,
 'uuid': '219e5d59-b9dc-584f-b83b-018c47e8739d',
 'url': 's3://mast/level1/shots/11695.zarr',
 'endpoint_url': 'https://s3.echo.stfc.ac.uk',
 'timestamp': '2004-12-13T11:54:00',
 'preshot_description': '\n0.1T TF SHOT\n',
 'postshot_description': '\nOK\n',
 'campaign': 'M5',
 'reference_shot': None,
 'scenario': None,
 'heating': None,
 'pellets': False,
 'rmp_coil': None,
 'mcs_gdc_pre_shot': None,
 'mcs_select_gas_group_pv1': None,
 'mcs_select_gas_group_pv2': None,
 'mcs_select_gas_group_pv3': None,
 'mcs_select_gas_group_pv4': None,
 'mcs_select_gas_group_pv5': None,
 'mcs_select_gas_group_pv6': None,
 'mcs_select_gas_group_pv7': None,
 'mcs_select_gas_group_pv8': None,
 'mcs_select_gas_group_pv9': None,
 'mcs_select_lvps_tf_power_supply': None,
 'mcs_select_cp3s_start_bank': None,
 'mcs_select_fa1': None,
 'mcs_select_fa2': None,
 'mcs_select_mfps': None,
 'mcs_select_efps': None,
 'mcs_select_sfps': None,
 'mcs_select_scs4': None,
 'mcs_select_fa3': None,
 'mcs_select_fa4': None,
 'mcs_select_p1ps': None,
 'mcs_select_cp3c_counterpulse_bank': None,
 'mcs_select_cp4_start_bank': None,
 'mcs_select_cp5_start_bank': None,
 'mcs_select_p3p': None,
 'mcs_select_p4': None,
 'mcs_select_p5': None,
 'mcs_select_ecrh': None,
 'mcs_select_xmm_os9': None,
 'mcs_select_xpc_os9': None,
 'mcs_select_xcm_os9': None,
 'mcs_select_xmw_os9': None,
 'mcs_select_xma_os9': None,
 'mcs_ss_nbi_gas': None,
 'mcs_sw_nbi_gas_mcs': None,
 'mcs_cp3s_volt': None,
 'mcs_cp3c_volt': None,
 'mcs_cp4s_volt': None,
 'mcs_cp5s_volt': None,
 'mcs_additive_gas_pressure': None,
 'mcs_plenum_gas_pressure': None,
 'mcs_tg1_base_pressure': None,
 'mcs_p3_direction': None,
 'mcs_p4_direction': None,
 'mcs_p5_direction': None,
 'mcs_ss_number': None,
 'mcs_pshot_gdc_time': None,
 'mcs_gdc_start_time': None,
 'mcs_gas_puff_pressure': None,
 'mcs_gas_puff_start_time': None,
 'mcs_gas_puff_duration': None,
 'mcs_p2xo': None,
 'mcs_p2xo_start_forward': None,
 'mcs_p2xo_start_reverse': None,
 'mcs_tf_flat_top_value': None,
 'mcs_tf_flat_top_time': None,
 'mcs_tf_rise_time_value': None,
 'mcs_plasma_direction': None,
 'mcs_p2_config': None,
 'mcs_cp3s_start_time': None,
 'mcs_cp3c_start_time': None,
 'mcs_cp4_start_time': None,
 'mcs_p4_ignitron_start_time': None,
 'mcs_p4_thyristor_start_time': None,
 'mcs_cp5_start_time': None,
 'mcs_p5_ignitron_start_time': None,
 'mcs_p5_thyristor_start_time': None,
 'mcs_mfps_current_limit': None,
 'mcs_mfps_start_time': None,
 'mcs_efps_current_limit': None,
 'mcs_efps_start_time': None,
 'mcs_sfps_current_limit': None,
 'mcs_sfps_start_time': None,
 'mcs_mfps_duration': None,
 'mcs_efps_duration': None,
 'mcs_sfps_duration': None,
 'mcs_p1ps_pos_current_limit': None,
 'mcs_p1ps_negative_current_limit': None,
 'mcs_p1ps_start_time': None,
 'mcs_p1ps_duration': None,
 'mcs_fa1_start_time': None,
 'mcs_fa1_duration': None,
 'mcs_fa2_start_time': None,
 'mcs_fa2_duration': None,
 'mcs_fa3_start_time': None,
 'mcs_fa3_duration': None,
 'mcs_fa4_start_time': None,
 'mcs_fa4_duration': None,
 'mcs_scs4_start_time': None,
 'mcs_scs4_duration': None,
 'shot_abort': None,
 'generic_minor_radius_max_current': None,
 'generic_minor_radius': None,
 'generic_minor_radius_ruby_time': None,
 'generic_poloidal_area_max_current': None,
 'generic_max_poloidal_area': None,
 'generic_poloidal_area_ruby_time': None,
 'generic_beta_poloidal_max_current': None,
 'generic_max_beta_poloidal': None,
 'generic_beta_poloidal_ruby_time': None,
 'generic_beta_time_max_current': None,
 'generic_max_beta_max_current': None,
 'generic_beta_ruby_time': None,
 'generic_toroidal_max_current': None,
 'generic_max_toroidal': None,
 'generic_toroidal_ruby_time': None,
 'radii_c2ratio': None,
 'gas_col_temp_in': None,
 'gas_col_temp_out': None,
 'shot_creation_date': None,
 'generic_dt_energy_max_current': None,
 'generic_dt_total_energy': None,
 'generic_dt_energy_ruby_time': None,
 'nbi_energy_ss_max_power': None,
 'nbi_energy_sw_max_power': None,
 'shot_experiment_date': None,
 'shot_experiment_number': 11695,
 'shot_experiment_time': None,
 'shot_experiment_tags': None,
 'shot_flat_top_duration': None,
 'shot_glow_discharge_duration': None,
 'shot_glow_discharge_time': None,
 'gas_inboard_pressure_pre_pulse': None,
 'plasma_time_avg_current': None,
 'plasma_time_all_avg_current': None,
 'plasma_max_current': None,
 'nbi_injected_energy_max_current': None,
 'nbi_injected_energy_ss_max_current': None,
 'nbi_injected_energy_max': None,
 'nbi_energy_ss_max_current': None,
 'nbi_energy_sw_max_current': None,
 'nbi_total_injected_energy': None,
 'nbi_total_injected_energy_ss': None,
 'nbi_total_injected_energy_sw': None,
 'nbi_injected_energy_ruby_time': None,
 'nbi_injected_energy_ss_ruby_time': None,
 'nbi_injected_energy_sw_ruby_time': None,
 'ohmnic_energy_max_current': None,
 'ohmnic_energy_max_heating': None,
 'ohmnic_energy_total': None,
 'ohmnic_energy_ruby_time': None,
 'generic_plasma_elongation_max_current': None,
 'generic_plasma_elongation_max': None,
 'generic_plasma_elongation_ruby_time': None,
 'equi_li2_max_current': None,
 'equi_max_li2': None,
 'equi_li2_ruby_time': None,
 'equi_li3_max_current': None,
 'equi_max_li3': None,
 'equi_li3_ruby_time': None,
 'gas_tg1_pressure_ps': None,
 'thomson_density_line_max_current': None,
 'thomson_max_density_line': None,
 'thomson_density_line_ruby_time': None,
 'thomson_density_max_current': None,
 'thomson_density_max': None,
 'thomson_density_ruby_time': None,
 'thomson_ne0_rat_line_avg_ne': None,
 'thomson_line_avg_density_co2': None,
 'thomson_greenwald_density_limit': None,
 'thomson_greenwald_rat_line_avg_density': None,
 'rad_o2ratio': None,
 'shot_objective': 'COMMISSION TF AND P1',
 'thomson_pressure_max_current': None,
 'thomson_pressure_max': None,
 'thomson_pressure_ruby_time': None,
 'thomson_pressure_ruby': None,
 'shot_physicist': None,
 'nbi_power_max_current': None,
 'nbi_power_ss_max_current': None,
 'nbi_power_max_power': None,
 'nbi_power_max_ss': None,
 'nbi_power_max_sw': None,
 'nbi_power_ruby_time': None,
 'nbi_power_truby_ss': None,
 'nbi_power_truby_sw': None,
 'ohmnic_heating_max_current': None,
 'ohmnic_max_heating': None,
 'ohmnic_heating_ruby_time': None,
 'shot_postshot_comment': 'OK',
 'generic_rad_power_loss_max_current': None,
 'generic_max_power_loss': None,
 'generic_power_loss_ruby_time': None,
 'rad_flat_top_power_rat_avg_line_density': None,
 'shot_preshot_comment': '0.1T TF SHOT',
 'shot_program': 'COMMISSIONING',
 'shot_pulse_number': None,
 'generic_q95_max_current': None,
 'generic_min_q95': None,
 'generic_q95_ruby_time': None,
 'shot_reference': None,
 'generic_geo_radius_max_current': None,
 'generic_max_geo_radius': None,
 'generic_geo_radius_ruby_time': None,
 'radii_rinner_radius_d_alpha': None,
 'radii_rinner_radius_efit': None,
 'radii_router_radius_d_alpha': None,
 'radii_router_radius_efit': None,
 'radii_s_area_max_current': None,
 'generic_s_area_max': None,
 'generic_s_area_ruby_time': None,
 'cpf_sc': None,
 'shot_scenario': None,
 'shot_summary': 'TF OK BUT WITH SLIDING JOINT ALARMS. P1PS IS RECEIVING DRIVE SIGNAL BUT IGNORING IT. UNRESOLVED...',
 'generic_min_radius_max_plasma': None,
 'generic_energy_time_max_current': None,
 'generic_max_energy_time': None,
 'generic_energy_time_ruby_time': None,
 'generic_max_beta_poloidal_mhd': None,
 'generic_max_beta_mhd': None,
 'generic_max_toroidal_time': None,
 'generic_max_dt_mhd_energy_time': None,
 'thomson_temp_max_current': None,
 'thomson_temp_max': None,
 'thomson_temp_ruby_time': None,
 'thomson_temp_rat_line_avg_temp': None,
 'thomson_temp_ruby': None,
 'thomson_yag_line_avg_temp': None,
 'plasma_end_time': None,
 'gas_puff_end_time': None,
 'nbi_end_time': None,
 'nbi_end_time_ss': None,
 'shot_end_code': None,
 'plasma_flat_top_end_time': None,
 'plasma_flat_top_start_time': None,
 'plasma_max_plasma_current_time': None,
 'generic_max_plasma_elongation_time': None,
 'equi_max_li2_time': None,
 'equi_max_li3_time': None,
 'thomson_max_density_line_time': None,
 'thomson_max_density_time': None,
 'thomson_max_pressure_time': None,
 'nbi_max_power_time': None,
 'nbi_max_ss_power_time': None,
 'nbi_max_sw_power_time': None,
 'ohmnic_max_heating_time': None,
 'generic_max_plasma_power_loss_time': None,
 'generic_min_q95_time': None,
 'generic_max_geo_major_radius_time': None,
 'thomson_ruby_time': None,
 'generic_max_plasma_s_area_time': None,
 'gas_puff_start_time': None,
 'nbi_start_time': None,
 'nbi_ss_start_time': None,
 'nbi_sw_start_time': None,
 'generic_max_total_energy_time': None,
 'thomson_max_temp_time': None,
 'generic_max_plasma_vol_time': None,
 'generic_max_mhd_energy_time': None,
 'generic_max_plasma_zeff_time': None,
 'shot_useful': None,
 'generic_plasma_vol_max_current': None,
 'generic_max_plasma_vol': None,
 'generic_plasma_vol_ruby_time': None,
 'generic_energy_current_max': None,
 'generic_energy_max': None,
 'generic_energy_ruby_time': None,
 'generic_plasma_zeff_max_current': None,
 'generic_plasma_zeff_max': None,
 'generic_plasma_zeff_ruby_time': None,
 'rad_magnetic_height_efit': None,
 'current_range': None,
 'divertor_config': 'Conventional',
 'plasma_shape': None,
 'commissioner': None,
 'facility': 'MAST',
 'radii_magnetic:radius_efit': None}

Each item in the list is a json object. This contains the meta-data items that corresponded to our query. In this case, each item contains information about a different MAST shot. Each item has lots of information about different shots, for example the shot ID, the campaign the shot was part of, the pre- and post-shot description by investigators.

For more information on the what’s returned by the API you can look at the endpoint documentation:

https://mast-app.site/redoc

Of course, we can read all this JSON data directly into common python data analysis packages, for example, we can create a pandas dataframe directly from the endpoint data

df = pd.read_json(f'{API_URL}/shots')['items']
df = pd.DataFrame(df.to_list())
df.head()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 df = pd.read_json(f'{API_URL}/shots')['items']
      2 df = pd.DataFrame(df.to_list())
      3 df.head()

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/io/json/_json.py:784, in read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates, keep_default_dates, precise_float, date_unit, encoding, encoding_errors, lines, chunksize, compression, nrows, storage_options, dtype_backend, engine)
    782     return json_reader
    783 else:
--> 784     return json_reader.read()

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/io/json/_json.py:975, in JsonReader.read(self)
    973         obj = self._get_object_parser(self._combine_lines(data_lines))
    974 else:
--> 975     obj = self._get_object_parser(self.data)
    976 if self.dtype_backend is not lib.no_default:
    977     return obj.convert_dtypes(
    978         infer_objects=False, dtype_backend=self.dtype_backend
    979     )

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/io/json/_json.py:1001, in JsonReader._get_object_parser(self, json)
    999 obj = None
   1000 if typ == "frame":
-> 1001     obj = FrameParser(json, **kwargs).parse()
   1003 if typ == "series" or obj is None:
   1004     if not isinstance(dtype, bool):

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/io/json/_json.py:1134, in Parser.parse(self)
   1133 def parse(self):
-> 1134     self._parse()
   1136     if self.obj is None:
   1137         return None

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/io/json/_json.py:1319, in FrameParser._parse(self)
   1316 orient = self.orient
   1318 if orient == "columns":
-> 1319     self.obj = DataFrame(
   1320         loads(json, precise_float=self.precise_float), dtype=None
   1321     )
   1322 elif orient == "split":
   1323     decoded = {
   1324         str(k): v
   1325         for k, v in loads(json, precise_float=self.precise_float).items()
   1326     }

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/core/frame.py:709, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    703     mgr = self._init_mgr(
    704         data, axes={"index": index, "columns": columns}, dtype=dtype, copy=copy
    705     )
    707 elif isinstance(data, dict):
    708     # GH#38939 de facto copy defaults to False only in non-dict cases
--> 709     mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
    710 elif isinstance(data, ma.MaskedArray):
    711     from numpy.ma import mrecords

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/core/internals/construction.py:481, in dict_to_mgr(data, index, columns, dtype, typ, copy)
    477     else:
    478         # dtype check to exclude e.g. range objects, scalars
    479         arrays = [x.copy() if hasattr(x, "dtype") else x for x in arrays]
--> 481 return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/core/internals/construction.py:115, in arrays_to_mgr(arrays, columns, index, dtype, verify_integrity, typ, consolidate)
    112 if verify_integrity:
    113     # figure out the index, if necessary
    114     if index is None:
--> 115         index = _extract_index(arrays)
    116     else:
    117         index = ensure_index(index)

File ~/miniconda3/envs/mast/lib/python3.11/site-packages/pandas/core/internals/construction.py:658, in _extract_index(data)
    655     raise ValueError("All arrays must be of the same length")
    657 if have_dicts:
--> 658     raise ValueError(
    659         "Mixing dicts with non-Series may lead to ambiguous ordering."
    660     )
    662 if have_series:
    663     if lengths[0] != len(index):

ValueError: Mixing dicts with non-Series may lead to ambiguous ordering.

Searching & Filtering Data

All REST API endpoints can take query parameters to filter the data returned. For example, we can return all shots for the M9 campaign by using the approrpiate query string.

For example, we can query for everything from the M9 campaign by adding ?campaign=M9 to our query string.

df = pd.read_json(f"{API_URL}/shots?campaign=M9")['items']
df = pd.DataFrame(df.to_list())
df.head()

	shot_id	uuid	url	timestamp	preshot_description	postshot_description	campaign	reference_shot	scenario	heating	...	cpf_vol_ipmax	cpf_vol_max	cpf_vol_truby	cpf_wmhd_ipmax	cpf_wmhd_max	cpf_wmhd_truby	cpf_zeff_ipmax	cpf_zeff_max	cpf_zeff_truby	cpf_zmag_efit
0	15085	0bb8ea1b-954b-5cce-8b2d-df40a0703b2f	s3://mast/level1/shots/15085.zarr	2006-04-25T13:48:00	\nReplace ccbv connector. repeat.\n	\nRadial control unchanged. Zip/FA jump less ...	M6	15084.0	None	None	...	None	None	None	None	None	None	None	None	None	None
1	15086	2ac3c2b7-0444-5f3b-86a9-38ef6bb3c160	s3://mast/level1/shots/15086.zarr	2006-04-25T14:15:00	\nAdd another 5cm to Drref during the ramp. Re...	\nEarly termination, at 180ms.\n	M6	15085.0	None	None	...	None	None	None	None	None	None	None	None	None	None
2	15087	9867930b-af84-5866-a62c-9fe77d9a3def	s3://mast/level1/shots/15087.zarr	2006-04-25T14:34:00	\nRepeat, with small change to IDIV.\n	\nEarly termination at 200ms.\n	M6	15086.0	None	None	...	None	None	None	None	None	None	None	None	None	None
3	15088	f3f7ffe4-bfd1-5b8d-8856-9d7e4377724b	s3://mast/level1/shots/15088.zarr	2006-04-25T14:52:00	\nRestore shot 15085.\n	\nOK.\n	M6	15085.0	None	None	...	None	None	None	None	None	None	None	None	None	None
4	15089	1adae615-46c4-5262-a49d-b25f707b0857	s3://mast/level1/shots/15089.zarr	2006-04-25T15:17:00	\nRepeat, with IDIV, BV tweaked to match last ...	\nTerminated early.\n	M6	15088.0	None	None	...	None	None	None	None	None	None	None	None	None	None

5 rows × 282 columns

Pagination

The REST API responses are paginated, meaning that only a subset of the full items are returned with each query. Pagination is used to limit the total number of requests made by each user to prevent any single user overloading the server with huge data requests.

Pagination information is included in the response and corresponds to RFC 8288. The response contains cursors at the bottom of the response:

• current_page - this is the cursor relating to the current page of results.

• next_page - this is the cursor for the next page of results.

• previous_page - this is the cursor for the previous page of results, will result in null if on the first page.

You can control the pagination using the following options as query arguments:

• cursor=xx to set the cursor for the displayed set of results

• size=xx to set the number of results returned by each page.

response = requests.get(f'{API_URL}/shots?size=2&cursor=Pmk6MzAxMTE%3D')
result = response.json()
headers = response.headers

print("Current page cursor", result['current_page'])
print("Next page cursor", result['next_page'])
print("Previous page cursor", result['previous_page'])

df = pd.DataFrame(result['items'])
df

Current page cursor Pmk6MzAxMTE%3D
Next page cursor Pmk6MzAxMTM%3D
Previous page cursor PGk6MzAxMTI%3D

	shot_id	uuid	url	timestamp	preshot_description	postshot_description	campaign	reference_shot	scenario	heating	...	cpf_vol_ipmax	cpf_vol_max	cpf_vol_truby	cpf_wmhd_ipmax	cpf_wmhd_max	cpf_wmhd_truby	cpf_zeff_ipmax	cpf_zeff_max	cpf_zeff_truby	cpf_zmag_efit
0	30112	eefe2467-69b0-5fe7-9863-e21289daa135	s3://mast/level1/shots/30112.zarr	2013-09-09T11:42:00	\nReload 30110 - reduce IP flat top to 750 KA ...	\nPIC happy with data.\n	M9	30110	4	SS Beam	...	7.845118	8.841636	0.0	65898.305250	82056.664	0.0	None	None	None	0.047497
1	30113	742c78fa-3811-5404-93b3-5ab419d330bd	s3://mast/level1/shots/30113.zarr	2013-09-09T12:00:00	\nRestore 30111\n	\nShot OK for programme. Two good beams. Big l...	M9	30111	3	2 Beams,SS Beam,SW Beam	...	8.391672	8.426641	0.0	43127.703046	60260.926	0.0	None	None	None	0.042220

2 rows × 282 columns