{ "cells": [ { "cell_type": "code", "execution_count": 8, "id": "19068483", "metadata": { "tags": [ "hide-output" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pandas in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2.2.2)\n", "Requirement already satisfied: gql in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (3.5.0)\n", "Requirement already satisfied: numpy>=1.23.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (1.26.4)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2.9.0)\n", "Requirement already satisfied: pytz>=2020.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2024.1)\n", "Requirement already satisfied: tzdata>=2022.7 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas) (2024.1)\n", "Requirement already satisfied: graphql-core<3.3,>=3.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from gql) (3.2.3)\n", "Requirement already satisfied: yarl<2.0,>=1.6 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from gql) (1.9.4)\n", "Requirement already satisfied: backoff<3.0,>=1.11.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from gql) (2.2.1)\n", "Requirement already satisfied: anyio<5,>=3.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from gql) (4.4.0)\n", "Requirement already satisfied: idna>=2.8 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from anyio<5,>=3.0->gql) (3.7)\n", "Requirement already satisfied: sniffio>=1.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from anyio<5,>=3.0->gql) (1.3.1)\n", "Requirement already satisfied: six>=1.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n", "Requirement already satisfied: multidict>=4.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from yarl<2.0,>=1.6->gql) (6.0.5)\n" ] } ], "source": [ "! pip install pandas gql" ] }, { "cell_type": "markdown", "id": "76fa365c-b6d9-479f-99d8-d0d53ec30e8e", "metadata": {}, "source": [ "# GraphQL Examples\n", "\n", "In this notebook we explore searching for metadata from the GraphQL API. The GraphQL API provides a method to programmatically extract a JSON representation of the meta data from the API. It has the advantage over REST that it can prevent under and over fetching data.\n", "\n", "First we load some python dependancies that we will use as part of this notebook and set the variable `API_URL` to the location of the GraphQL API." ] }, { "cell_type": "code", "execution_count": 9, "id": "ebba03b7-34c4-4b49-94d8-112c7176ac7f", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "from gql import gql, Client\n", "from gql.transport.requests import RequestsHTTPTransport\n", "\n", "# This is the location of the GraphQL API\n", "API_URL = \"https://mastapp.site/graphql\"" ] }, { "cell_type": "markdown", "id": "22ff9d86", "metadata": {}, "source": [ "Setup a `gql` client. This is used to query the graphql API" ] }, { "cell_type": "code", "execution_count": 10, "id": "bc504ba6", "metadata": {}, "outputs": [], "source": [ "# Select your transport with a defined url endpoint\n", "transport = RequestsHTTPTransport(url=API_URL)\n", "# Create a GraphQL client using the defined transport\n", "client = Client(transport=transport, fetch_schema_from_transport=True)" ] }, { "cell_type": "markdown", "id": "8a9a779a-3cc2-4557-8cb1-a9849f01a434", "metadata": {}, "source": [ "## Querying Shots with GraphQL\n", "\n", "With GraphQL you can query exactly what you want, rather than having to recieve the whole table from the database This is useful in cases where the whole table has many columns, but you are interested in just a subset of them.\n", "\n", "The GraphQL endpoint is located at `/graphql`. You can find the documentation and an interactive query explorer at the URL below:" ] }, { "cell_type": "code", "execution_count": 11, "id": "dfb0f1c7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GraphQL API Endpoint https://mastapp.site/graphql\n" ] } ], "source": [ "print(f\"GraphQL API Endpoint {API_URL}\")" ] }, { "cell_type": "markdown", "id": "fa9ab9b8", "metadata": {}, "source": [ "Unlike the REST API which uses HTTP `GET` requests to return data, with GraphQL we use HTTP `POST` to post our query to the API.\n", "\n", "Here is a simple example of getting some shot data from the GraphQL API. We need to explicity state what information we want to return from the API. Here we are asking for:\n", " - the shot ID\n", " - the timestamp that the shot was taken\n", " - the preshot description\n", " - the divertor configuration" ] }, { "cell_type": "code", "execution_count": 12, "id": "4970f970-2e6a-414a-8722-427f57c62964", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
shot_idtimestamppreshot_descriptiondivertor_config
0150852006-04-25T13:48:00\\nReplace ccbv connector. repeat.\\nconventional
1150862006-04-25T14:15:00\\nAdd another 5cm to Drref during the ramp. Re...conventional
2150872006-04-25T14:34:00\\nRepeat, with small change to IDIV.\\nconventional
3150882006-04-25T14:52:00\\nRestore shot 15085.\\nconventional
4150892006-04-25T15:17:00\\nRepeat, with IDIV, BV tweaked to match last ...conventional
\n", "
" ], "text/plain": [ " shot_id timestamp \\\n", "0 15085 2006-04-25T13:48:00 \n", "1 15086 2006-04-25T14:15:00 \n", "2 15087 2006-04-25T14:34:00 \n", "3 15088 2006-04-25T14:52:00 \n", "4 15089 2006-04-25T15:17:00 \n", "\n", " preshot_description divertor_config \n", "0 \\nReplace ccbv connector. repeat.\\n conventional \n", "1 \\nAdd another 5cm to Drref during the ramp. Re... conventional \n", "2 \\nRepeat, with small change to IDIV.\\n conventional \n", "3 \\nRestore shot 15085.\\n conventional \n", "4 \\nRepeat, with IDIV, BV tweaked to match last ... conventional " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Write our GraphQL query.\n", "query = gql(\"\"\"\n", "query {\n", " all_shots {\n", " shots {\n", " shot_id\n", " timestamp\n", " preshot_description\n", " divertor_config\n", " }\n", " }\n", "}\n", "\"\"\")\n", "\n", "# # Query the API and get a JSON response\n", "result = client.execute(query)\n", "shots = result['all_shots']['shots']\n", "df = pd.DataFrame(shots)\n", "df.head()" ] }, { "cell_type": "markdown", "id": "6c8a11b9", "metadata": {}, "source": [ "### Searching & Filtering Data\n", "We can also supply query parameters to GraphQL, such as limiting the number of returned values or filtering by value. Here we are limiting the first 3 values and we are selcting only shots from the M9 campaign." ] }, { "cell_type": "code", "execution_count": 13, "id": "399c75c1-16ab-4a34-b968-5a9f3c6f99e0", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
shot_idtimestamppreshot_descriptiondivertor_config
0283902012-03-06T14:47:00\\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For r...conventional
1283912012-03-06T14:52:00\\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For r...conventional
2283922012-03-06T15:03:00\\nHL11, 300 ms, 2 V. He plenum 1047.\\nconventional
\n", "
" ], "text/plain": [ " shot_id timestamp \\\n", "0 28390 2012-03-06T14:47:00 \n", "1 28391 2012-03-06T14:52:00 \n", "2 28392 2012-03-06T15:03:00 \n", "\n", " preshot_description divertor_config \n", "0 \\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For r... conventional \n", "1 \\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For r... conventional \n", "2 \\nHL11, 300 ms, 2 V. He plenum 1047.\\n conventional " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Write our GraphQL query.\n", "query = gql(\"\"\"\n", "query ($campaign: String!) {\n", " all_shots (limit: 3, where: {campaign: {eq: $campaign}}) {\n", " shots {\n", " shot_id\n", " timestamp\n", " preshot_description\n", " divertor_config\n", " }\n", " }\n", "}\n", "\"\"\")\n", "# Query the API and get a JSON response\n", "result = client.execute(query, variable_values={\"campaign\": \"M9\"})\n", "df = pd.DataFrame(result['all_shots']['shots'])\n", "df.head()" ] }, { "cell_type": "markdown", "id": "5aa79e2b", "metadata": {}, "source": [ "### Nested Queries\n", "\n", "One feature which makes GraphQL much more powerful than REST is that you may perform nested queries to gather different subsets of the data. For example, here we are going to query for all datasets with \"AMC\" in the name, and query for information about shots associated with this dataset." ] }, { "cell_type": "code", "execution_count": 14, "id": "88bdab08", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'all_signals': {'signals': []}}" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Write our GraphQL query.\n", "query = gql(\"\"\"\n", "query ($signal: String!) {\n", " all_signals (limit: 3, where: {name: {contains: $signal}}) {\n", " signals {\n", " name\n", " url\n", " shot {\n", " shot_id\n", " timestamp\n", " divertor_config\n", " }\n", " }\n", " }\n", "}\n", "\"\"\")\n", "# Query the API and get a JSON response\n", "client.execute(query, variable_values={\"signal\": \"AMC\"})" ] }, { "cell_type": "markdown", "id": "6e2ac419-db72-4a95-88a0-d5226b399376", "metadata": {}, "source": [ "### Pagination in GraphQL\n", "\n", "GraphQL queries are paginated. You may access other entries by including the page metadata and its associated elements. Here's an example of getting paginated entries:" ] }, { "cell_type": "code", "execution_count": 15, "id": "f4566e66-215c-4d41-aa3a-44fb9d6a0447", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'all_shots': {'shots': [{'shot_id': 28390, 'timestamp': '2012-03-06T14:47:00', 'preshot_description': '\\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For reference.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28391, 'timestamp': '2012-03-06T14:52:00', 'preshot_description': '\\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For reference.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28392, 'timestamp': '2012-03-06T15:03:00', 'preshot_description': '\\nHL11, 300 ms, 2 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'Mw==', 'total_items': 1675, 'total_pages': 559}}}\n", "{'all_shots': {'shots': [{'shot_id': 28393, 'timestamp': '2012-03-06T15:09:00', 'preshot_description': '\\nHL11, 300 ms, 3 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28394, 'timestamp': '2012-03-06T15:13:00', 'preshot_description': '\\nHL11, 300 ms, 5 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28395, 'timestamp': '2012-03-06T15:21:00', 'preshot_description': '\\nHL11, 300 ms, 6 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'Ng==', 'total_items': 1675, 'total_pages': 559}}}\n", "{'all_shots': {'shots': [{'shot_id': 28396, 'timestamp': '2012-03-06T15:31:00', 'preshot_description': '\\nHL11, 300 ms, 3 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28397, 'timestamp': '2012-03-06T15:38:00', 'preshot_description': '\\nHL11, 300 ms, 5 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28398, 'timestamp': '2012-03-06T15:43:00', 'preshot_description': '\\nHL11, 300 ms, 4 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'OQ==', 'total_items': 1675, 'total_pages': 559}}}\n" ] } ], "source": [ "def do_query(cursor: str = None):\n", " query = gql(\"\"\"\n", " query ($cursor: String) {\n", " all_shots (limit: 3, where: {campaign: {contains: \"M9\"}}, cursor: $cursor) {\n", " shots {\n", " shot_id\n", " timestamp\n", " preshot_description\n", " divertor_config\n", " }\n", " page_meta {\n", " next_cursor\n", " total_items\n", " total_pages\n", " }\n", " }\n", " }\n", " \"\"\")\n", " return client.execute(query, {'cursor': cursor})\n", "\n", "\n", "def iterate_responses():\n", " cursor = None\n", " while True:\n", " response = do_query(cursor)\n", " yield response\n", " cursor = response['all_shots']['page_meta']['next_cursor']\n", " if cursor is None:\n", " return\n", "\n", "responses = iterate_responses()\n", "print(next(responses))\n", "print(next(responses))\n", "print(next(responses))" ] } ], "metadata": { "kernelspec": { "display_name": "mast", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }