REST API Examples

REST API Examples#

In this notebook we explore searching for metadata from the REST API. The REST API provides a method to programmatically extract a JSON representation of the meta data from the API.

Querying Shots with the REST API#

We’re going to use the python requests library to query metadata from the database. All we need to do to get a result is to query the database with a HTTP GET at the appropriate endpoint. For example, to get information about different experimental shots we can query the /json/shots/ endpoint.

response = requests.get(f'{API_URL}/shots')
result = response.json()
print(f"Query returned status code: {response.status_code}")

Query returned status code: 200

The shots endpoint returns a JSON payload with a list of shots. Let’s look at the first element from the payload:

print(result['items'][0])

{'title': 'Shot Dataset', 'shot_id': 11695, 'uuid': '219e5d59-b9dc-584f-b83b-018c47e8739d', 'url': 's3://mast/level1/shots/11695.zarr', 'endpoint_url': 'https://s3.echo.stfc.ac.uk', 'timestamp': '2004-12-13T11:54:00', 'preshot_description': '\n0.1T TF SHOT\n', 'postshot_description': '\nOK\n', 'campaign': 'M5', 'reference_shot': None, 'scenario': None, 'heating': None, 'pellets': False, 'rmp_coil': None, 'mcs_gdc_pre_shot': None, 'mcs_select_gas_group_pv1': None, 'mcs_select_gas_group_pv2': None, 'mcs_select_gas_group_pv3': None, 'mcs_select_gas_group_pv4': None, 'mcs_select_gas_group_pv5': None, 'mcs_select_gas_group_pv6': None, 'mcs_select_gas_group_pv7': None, 'mcs_select_gas_group_pv8': None, 'mcs_select_gas_group_pv9': None, 'mcs_select_lvps_tf_power_supply': None, 'mcs_select_cp3s_start_bank': None, 'mcs_select_fa1': None, 'mcs_select_fa2': None, 'mcs_select_mfps': None, 'mcs_select_efps': None, 'mcs_select_sfps': None, 'mcs_select_scs4': None, 'mcs_select_fa3': None, 'mcs_select_fa4': None, 'mcs_select_p1ps': None, 'mcs_select_cp3c_counterpulse_bank': None, 'mcs_select_cp4_start_bank': None, 'mcs_select_cp5_start_bank': None, 'mcs_select_p3p': None, 'mcs_select_p4': None, 'mcs_select_p5': None, 'mcs_select_ecrh': None, 'mcs_select_xmm_os9': None, 'mcs_select_xpc_os9': None, 'mcs_select_xcm_os9': None, 'mcs_select_xmw_os9': None, 'mcs_select_xma_os9': None, 'mcs_ss_nbi_gas': None, 'mcs_sw_nbi_gas_mcs': None, 'mcs_cp3s_volt': None, 'mcs_cp3c_volt': None, 'mcs_cp4s_volt': None, 'mcs_cp5s_volt': None, 'mcs_additive_gas_pressure': None, 'mcs_plenum_gas_pressure': None, 'mcs_tg1_base_pressure': None, 'mcs_p3_direction': None, 'mcs_p4_direction': None, 'mcs_p5_direction': None, 'mcs_ss_number': None, 'mcs_pshot_gdc_time': None, 'mcs_gdc_start_time': None, 'mcs_gas_puff_pressure': None, 'mcs_gas_puff_start_time': None, 'mcs_gas_puff_duration': None, 'mcs_p2xo': None, 'mcs_p2xo_start_forward': None, 'mcs_p2xo_start_reverse': None, 'mcs_tf_flat_top_value': None, 'mcs_tf_flat_top_time': None, 'mcs_tf_rise_time_value': None, 'mcs_plasma_direction': None, 'mcs_p2_config': None, 'mcs_cp3s_start_time': None, 'mcs_cp3c_start_time': None, 'mcs_cp4_start_time': None, 'mcs_p4_ignitron_start_time': None, 'mcs_p4_thyristor_start_time': None, 'mcs_cp5_start_time': None, 'mcs_p5_ignitron_start_time': None, 'mcs_p5_thyristor_start_time': None, 'mcs_mfps_current_limit': None, 'mcs_mfps_start_time': None, 'mcs_efps_current_limit': None, 'mcs_efps_start_time': None, 'mcs_sfps_current_limit': None, 'mcs_sfps_start_time': None, 'mcs_mfps_duration': None, 'mcs_efps_duration': None, 'mcs_sfps_duration': None, 'mcs_p1ps_pos_current_limit': None, 'mcs_p1ps_negative_current_limit': None, 'mcs_p1ps_start_time': None, 'mcs_p1ps_duration': None, 'mcs_fa1_start_time': None, 'mcs_fa1_duration': None, 'mcs_fa2_start_time': None, 'mcs_fa2_duration': None, 'mcs_fa3_start_time': None, 'mcs_fa3_duration': None, 'mcs_fa4_start_time': None, 'mcs_fa4_duration': None, 'mcs_scs4_start_time': None, 'mcs_scs4_duration': None, 'shot_abort': None, 'generic_minor_radius_max_current': None, 'generic_minor_radius': None, 'generic_minor_radius_ruby_time': None, 'generic_poloidal_area_max_current': None, 'generic_max_poloidal_area': None, 'generic_poloidal_area_ruby_time': None, 'generic_beta_poloidal_max_current': None, 'generic_max_beta_poloidal': None, 'generic_beta_poloidal_ruby_time': None, 'generic_beta_time_max_current': None, 'generic_max_beta_max_current': None, 'generic_beta_ruby_time': None, 'generic_toroidal_max_current': None, 'generic_max_toroidal': None, 'generic_toroidal_ruby_time': None, 'radii_c2ratio': None, 'gas_col_temp_in': None, 'gas_col_temp_out': None, 'shot_creation_date': None, 'generic_dt_energy_max_current': None, 'generic_dt_total_energy': None, 'generic_dt_energy_ruby_time': None, 'nbi_energy_ss_max_power': None, 'nbi_energy_sw_max_power': None, 'shot_experiment_date': None, 'shot_experiment_number': 11695, 'shot_experiment_time': None, 'shot_experiment_tags': None, 'shot_flat_top_duration': None, 'shot_glow_discharge_duration': None, 'shot_glow_discharge_time': None, 'gas_inboard_pressure_pre_pulse': None, 'plasma_time_avg_current': None, 'plasma_time_all_avg_current': None, 'plasma_max_current': None, 'nbi_injected_energy_max_current': None, 'nbi_injected_energy_ss_max_current': None, 'nbi_injected_energy_max': None, 'nbi_energy_ss_max_current': None, 'nbi_energy_sw_max_current': None, 'nbi_total_injected_energy': None, 'nbi_total_injected_energy_ss': None, 'nbi_total_injected_energy_sw': None, 'nbi_injected_energy_ruby_time': None, 'nbi_injected_energy_ss_ruby_time': None, 'nbi_injected_energy_sw_ruby_time': None, 'ohmnic_energy_max_current': None, 'ohmnic_energy_max_heating': None, 'ohmnic_energy_total': None, 'ohmnic_energy_ruby_time': None, 'generic_plasma_elongation_max_current': None, 'generic_plasma_elongation_max': None, 'generic_plasma_elongation_ruby_time': None, 'equi_li2_max_current': None, 'equi_max_li2': None, 'equi_li2_ruby_time': None, 'equi_li3_max_current': None, 'equi_max_li3': None, 'equi_li3_ruby_time': None, 'gas_tg1_pressure_ps': None, 'thomson_density_line_max_current': None, 'thomson_max_density_line': None, 'thomson_density_line_ruby_time': None, 'thomson_density_max_current': None, 'thomson_density_max': None, 'thomson_density_ruby_time': None, 'thomson_ne0_rat_line_avg_ne': None, 'thomson_line_avg_density_co2': None, 'thomson_greenwald_density_limit': None, 'thomson_greenwald_rat_line_avg_density': None, 'rad_o2ratio': None, 'shot_objective': 'COMMISSION TF AND P1', 'thomson_pressure_max_current': None, 'thomson_pressure_max': None, 'thomson_pressure_ruby_time': None, 'thomson_pressure_ruby': None, 'shot_physicist': None, 'nbi_power_max_current': None, 'nbi_power_ss_max_current': None, 'nbi_power_max_power': None, 'nbi_power_max_ss': None, 'nbi_power_max_sw': None, 'nbi_power_ruby_time': None, 'nbi_power_truby_ss': None, 'nbi_power_truby_sw': None, 'ohmnic_heating_max_current': None, 'ohmnic_max_heating': None, 'ohmnic_heating_ruby_time': None, 'shot_postshot_comment': 'OK', 'generic_rad_power_loss_max_current': None, 'generic_max_power_loss': None, 'generic_power_loss_ruby_time': None, 'rad_flat_top_power_rat_avg_line_density': None, 'shot_preshot_comment': '0.1T TF SHOT', 'shot_program': 'COMMISSIONING', 'shot_pulse_number': None, 'generic_q95_max_current': None, 'generic_min_q95': None, 'generic_q95_ruby_time': None, 'shot_reference': None, 'generic_geo_radius_max_current': None, 'generic_max_geo_radius': None, 'generic_geo_radius_ruby_time': None, 'radii_rinner_radius_d_alpha': None, 'radii_rinner_radius_efit': None, 'radii_router_radius_d_alpha': None, 'radii_router_radius_efit': None, 'radii_s_area_max_current': None, 'generic_s_area_max': None, 'generic_s_area_ruby_time': None, 'cpf_sc': None, 'shot_scenario': None, 'shot_summary': 'TF OK BUT WITH SLIDING JOINT ALARMS. P1PS IS RECEIVING DRIVE SIGNAL BUT IGNORING IT. UNRESOLVED...', 'generic_min_radius_max_plasma': None, 'generic_energy_time_max_current': None, 'generic_max_energy_time': None, 'generic_energy_time_ruby_time': None, 'generic_max_beta_poloidal_mhd': None, 'generic_max_beta_mhd': None, 'generic_max_toroidal_time': None, 'generic_max_dt_mhd_energy_time': None, 'thomson_temp_max_current': None, 'thomson_temp_max': None, 'thomson_temp_ruby_time': None, 'thomson_temp_rat_line_avg_temp': None, 'thomson_temp_ruby': None, 'thomson_yag_line_avg_temp': None, 'plasma_end_time': None, 'gas_puff_end_time': None, 'nbi_end_time': None, 'nbi_end_time_ss': None, 'shot_end_code': None, 'plasma_flat_top_end_time': None, 'plasma_flat_top_start_time': None, 'plasma_max_plasma_current_time': None, 'generic_max_plasma_elongation_time': None, 'equi_max_li2_time': None, 'equi_max_li3_time': None, 'thomson_max_density_line_time': None, 'thomson_max_density_time': None, 'thomson_max_pressure_time': None, 'nbi_max_power_time': None, 'nbi_max_ss_power_time': None, 'nbi_max_sw_power_time': None, 'ohmnic_max_heating_time': None, 'generic_max_plasma_power_loss_time': None, 'generic_min_q95_time': None, 'generic_max_geo_major_radius_time': None, 'thomson_ruby_time': None, 'generic_max_plasma_s_area_time': None, 'gas_puff_start_time': None, 'nbi_start_time': None, 'nbi_ss_start_time': None, 'nbi_sw_start_time': None, 'generic_max_total_energy_time': None, 'thomson_max_temp_time': None, 'generic_max_plasma_vol_time': None, 'generic_max_mhd_energy_time': None, 'generic_max_plasma_zeff_time': None, 'shot_useful': None, 'generic_plasma_vol_max_current': None, 'generic_max_plasma_vol': None, 'generic_plasma_vol_ruby_time': None, 'generic_energy_current_max': None, 'generic_energy_max': None, 'generic_energy_ruby_time': None, 'generic_plasma_zeff_max_current': None, 'generic_plasma_zeff_max': None, 'generic_plasma_zeff_ruby_time': None, 'rad_magnetic_height_efit': None, 'current_range': None, 'divertor_config': 'Conventional', 'plasma_shape': None, 'commissioner': None, 'facility': 'MAST', 'radii_magnetic:radius_efit': None}

Each item in the list is a json object. This contains the meta-data items that corresponded to our query. In this case, each item contains information about a different MAST shot. Each item has lots of information about different shots, for example the shot ID, the campaign the shot was part of, the pre- and post-shot description by investigators.

For more information on the what’s returned by the API you can look at the endpoint documentation:

https://mastapp.site/redoc

Of course, we can read all this JSON data directly into common python data analysis packages, for example, we can create a pandas dataframe directly from the endpoint data

df = pd.read_json(f'{API_URL}/shots', lines=True)['items'][0]
df = pd.DataFrame(df)
df.head()

	title	shot_id	uuid	url	endpoint_url	timestamp	preshot_description	postshot_description	campaign	reference_shot	...	generic_plasma_zeff_max_current	generic_plasma_zeff_max	generic_plasma_zeff_ruby_time	rad_magnetic_height_efit	current_range	divertor_config	plasma_shape	commissioner	facility	radii_magnetic:radius_efit
0	Shot Dataset	11695	219e5d59-b9dc-584f-b83b-018c47e8739d	s3://mast/level1/shots/11695.zarr	https://s3.echo.stfc.ac.uk	2004-12-13T11:54:00	\n0.1T TF SHOT\n	\nOK\n	M5	None	...	None	None	None	None	None	Conventional	None	None	MAST	None
1	Shot Dataset	11696	9ba69c96-34ac-5cd4-a1b8-8e9d7a44f871	s3://mast/level1/shots/11696.zarr	https://s3.echo.stfc.ac.uk	2004-12-13T12:07:00	\nSTANDARD 0.3T TF SHOT\n	\nOK\n	M5	None	...	None	None	None	None	None	Conventional	None	None	MAST	None
2	Shot Dataset	11697	6ecd5c59-6d16-5a0f-a715-be31a88ca34e	s3://mast/level1/shots/11697.zarr	https://s3.echo.stfc.ac.uk	2004-12-13T12:19:00	\nRAISE TO 0.5T\n	\nOK, ALARMS ARE LOWER\n	M5	None	...	None	None	None	None	None	Conventional	None	None	MAST	None
3	Shot Dataset	11698	94366fda-b75e-5046-bb44-401844c563d8	s3://mast/level1/shots/11698.zarr	https://s3.echo.stfc.ac.uk	2004-12-13T12:31:00	\nRAISE TO .56T\n	\nSTILL ALARMS BUT LOWER AGAIN\n	M5	None	...	None	None	None	None	None	Conventional	None	None	MAST	None
4	Shot Dataset	11699	0e0f522d-a17f-5575-9bd0-d931231e8a71	s3://mast/level1/shots/11699.zarr	https://s3.echo.stfc.ac.uk	2004-12-13T12:45:00	\nRAISE TO .58T\n	\nOK\n	M5	None	...	None	None	None	None	None	Conventional	None	None	MAST	None

5 rows × 281 columns

Searching & Filtering Data#

All REST API endpoints can take query parameters to filter the data returned. For example, we can return all shots for the M9 campaign by using the approrpiate query string.

We can query for everything from the M9 campaign by adding ?filters=campaign$eq:M9 to our query string.

df = pd.read_json(f'{API_URL}/shots?filters=campaign$eq:M9', lines=True)['items'][0]
df = pd.DataFrame(df)
df.head()

	title	shot_id	uuid	url	endpoint_url	timestamp	preshot_description	postshot_description	campaign	reference_shot	...	generic_plasma_zeff_max_current	generic_plasma_zeff_max	generic_plasma_zeff_ruby_time	rad_magnetic_height_efit	current_range	divertor_config	plasma_shape	commissioner	facility	radii_magnetic:radius_efit
0	Shot Dataset	28390	4310e5b2-f025-54c3-8cf2-cd84e2111f8f	s3://mast/level1/shots/28390.zarr	https://s3.echo.stfc.ac.uk	2012-03-06T14:47:00	\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For r...	\nOk.\n	M9	NaN	...	None	None	None	None	None	Conventional	Connected Double Null	None	MAST	None
1	Shot Dataset	28391	b6006fb2-cba1-5351-be52-11173f39db91	s3://mast/level1/shots/28391.zarr	https://s3.echo.stfc.ac.uk	2012-03-06T14:52:00	\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For r...	\nOk.\n	M9	NaN	...	None	None	None	None	None	Conventional	Connected Double Null	None	MAST	None
2	Shot Dataset	28392	63aefbf6-78c4-5684-b415-77820a06a1fe	s3://mast/level1/shots/28392.zarr	https://s3.echo.stfc.ac.uk	2012-03-06T15:03:00	\nHL11, 300 ms, 2 V. He plenum 1047.\n	\nOk.\n	M9	NaN	...	None	None	None	None	None	Conventional	Connected Double Null	None	MAST	None
3	Shot Dataset	28393	6922f5c8-8b8f-5771-8b15-5a48c4b0ef19	s3://mast/level1/shots/28393.zarr	https://s3.echo.stfc.ac.uk	2012-03-06T15:09:00	\nHL11, 300 ms, 3 V. He plenum 1047.\n	\nOk.\n	M9	NaN	...	None	None	None	None	None	Conventional	Connected Double Null	None	MAST	None
4	Shot Dataset	28394	17b22eb0-ea8b-5a4a-8061-c62681149034	s3://mast/level1/shots/28394.zarr	https://s3.echo.stfc.ac.uk	2012-03-06T15:13:00	\nHL11, 300 ms, 5 V. He plenum 1047.\n	\nOk.\n	M9	NaN	...	None	None	None	None	None	Conventional	Connected Double Null	None	MAST	None

5 rows × 281 columns

Pagination#

The REST API responses are paginated, meaning that only a subset of the full items are returned with each query. Pagination is used to limit the total number of requests made by each user to prevent any single user overloading the server with huge data requests.

Pagination information is included in the response and corresponds to RFC 8288. The response contains cursors at the bottom of the response:

current_page - this is the cursor relating to the current page of results.
next_page - this is the cursor for the next page of results.
previous_page - this is the cursor for the previous page of results, will result in null if on the first page.

You can control the pagination using the following options as query arguments:

cursor=xx to set the cursor for the displayed set of results
size=xx to set the number of results returned by each page.

response = requests.get(f'{API_URL}/shots?size=2&cursor=Pmk6MzAxMTE%3D')
result = response.json()
headers = response.headers

print("Current page cursor", result['current_page'])
print("Next page cursor", result['next_page'])
print("Previous page cursor", result['previous_page'])

df = pd.DataFrame(result['items'])
df

Current page cursor Pmk6MzAxMTE%3D
Next page cursor Pmk6MzAxMTM%3D
Previous page cursor PGk6MzAxMTI%3D

	title	shot_id	uuid	url	endpoint_url	timestamp	preshot_description	postshot_description	campaign	reference_shot	...	generic_plasma_zeff_max_current	generic_plasma_zeff_max	generic_plasma_zeff_ruby_time	rad_magnetic_height_efit	current_range	divertor_config	plasma_shape	commissioner	facility	radii_magnetic:radius_efit
0	Shot Dataset	30112	eefe2467-69b0-5fe7-9863-e21289daa135	s3://mast/level1/shots/30112.zarr	https://s3.echo.stfc.ac.uk	2013-09-09T11:42:00	\nReload 30110 - reduce IP flat top to 750 KA ...	\nPIC happy with data.\n	M9	30110	...	None	None	None	0.047497	700 kA	Conventional	Connected Double Null	None	MAST	0.922218
1	Shot Dataset	30113	742c78fa-3811-5404-93b3-5ab419d330bd	s3://mast/level1/shots/30113.zarr	https://s3.echo.stfc.ac.uk	2013-09-09T12:00:00	\nRestore 30111\n	\nShot OK for programme. Two good beams. Big l...	M9	30111	...	None	None	None	0.042220	700 kA	Conventional	Connected Double Null	None	MAST	0.907091

2 rows × 281 columns

REST API Examples

Contents

REST API Examples#

Querying Shots with the REST API#

Searching & Filtering Data#

Pagination#