!pip install aiohttp requests s5cmd "xarray[io]"

Show code cell output

Hide code cell output

Requirement already satisfied: aiohttp in /home/deploy/.local/lib/python3.8/site-packages (3.10.11)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.22.0)
Requirement already satisfied: s5cmd in /home/deploy/.local/lib/python3.8/site-packages (0.2.0)
Requirement already satisfied: xarray[io] in /home/deploy/.local/lib/python3.8/site-packages (2023.1.0)

Requirement already satisfied: async-timeout<6.0,>=4.0; python_version < "3.11" in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (5.0.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/lib/python3/dist-packages (from aiohttp) (19.3.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (6.1.0)
Requirement already satisfied: yarl<2.0,>=1.12.0 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.15.2)
Requirement already satisfied: frozenlist>=1.1.1 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.5.0)
Requirement already satisfied: pandas>=1.3 in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2.0.3)

Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.8/dist-packages (from xarray[io]) (24.1)
Requirement already satisfied: numpy>=1.20 in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.24.4)
Requirement already satisfied: pydap; python_version < "3.10" and extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (3.4.1)
Requirement already satisfied: netCDF4; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.7.2)
Requirement already satisfied: rasterio; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.3.11)

Requirement already satisfied: zarr; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2.16.1)
Requirement already satisfied: scipy; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.10.1)
Requirement already satisfied: fsspec; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2024.10.0)

Requirement already satisfied: pooch; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.8.2)
Requirement already satisfied: h5netcdf; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.1.0)
Requirement already satisfied: cftime; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.6.4.post1)
Requirement already satisfied: cfgrib; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (0.9.15.0)
Requirement already satisfied: typing-extensions>=4.1.0; python_version < "3.11" in /home/deploy/.local/lib/python3.8/site-packages (from multidict<7.0,>=4.5->aiohttp) (4.13.2)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.12.0->aiohttp) (2.8)
Requirement already satisfied: propcache>=0.2.0 in /home/deploy/.local/lib/python3.8/site-packages (from yarl<2.0,>=1.12.0->aiohttp) (0.2.0)
Requirement already satisfied: pytz>=2020.1 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2025.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2.9.0.post0)

Requirement already satisfied: tzdata>=2022.1 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2025.2)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (1.14.0)
Requirement already satisfied: Webob in /home/deploy/.local/lib/python3.8/site-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (1.8.9)
Requirement already satisfied: beautifulsoup4 in /home/deploy/.local/lib/python3.8/site-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (4.13.4)
Requirement already satisfied: Jinja2 in /usr/local/lib/python3.8/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (3.1.4)
Requirement already satisfied: docopt in /usr/lib/python3/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (0.6.2)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from netCDF4; extra == "io"->xarray[io]) (2019.11.28)
Requirement already satisfied: click>=4.0 in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (7.0)
Requirement already satisfied: click-plugins in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (1.1.1.2)
Requirement already satisfied: affine in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (2.4.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (45.2.0)
Requirement already satisfied: cligj>=0.5 in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (0.7.2)
Requirement already satisfied: snuggs>=1.4.1 in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (1.4.7)
Requirement already satisfied: importlib-metadata; python_version < "3.10" in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (1.5.0)
Requirement already satisfied: numcodecs>=0.10.0 in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.12.1)
Requirement already satisfied: asciitree in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.3.3)
Requirement already satisfied: fasteners in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.19)
Requirement already satisfied: platformdirs>=2.5.0 in /home/deploy/.local/lib/python3.8/site-packages (from pooch; extra == "io"->xarray[io]) (4.3.6)

Requirement already satisfied: h5py in /home/deploy/.local/lib/python3.8/site-packages (from h5netcdf; extra == "io"->xarray[io]) (3.11.0)
Requirement already satisfied: eccodes>=0.9.8 in /home/deploy/.local/lib/python3.8/site-packages (from cfgrib; extra == "io"->xarray[io]) (2.42.0)
Requirement already satisfied: soupsieve>1.2 in /home/deploy/.local/lib/python3.8/site-packages (from beautifulsoup4->pydap; python_version < "3.10" and extra == "io"->xarray[io]) (2.7)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from Jinja2->pydap; python_version < "3.10" and extra == "io"->xarray[io]) (2.1.5)
Requirement already satisfied: pyparsing>=2.1.6 in /usr/lib/python3/dist-packages (from snuggs>=1.4.1->rasterio; extra == "io"->xarray[io]) (2.4.6)
Requirement already satisfied: cffi in /home/deploy/.local/lib/python3.8/site-packages (from eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (1.17.1)
Requirement already satisfied: findlibs in /home/deploy/.local/lib/python3.8/site-packages (from eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (0.1.1)
Requirement already satisfied: pycparser in /home/deploy/.local/lib/python3.8/site-packages (from cffi->eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (2.22)

import xarray as xr

Bulk Data Download

This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!

We can download data in bulk using any command line that supports the S3 protocol. We recommend using the s5cmd tool, which can be simply installed by running:

pip install s5cmd

Now we can download data using the cp command.

In this example, we are going to transfer the thompson scattering data for shot 30420 locally.

We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.

%%capture --no-display
%%bash
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level2/shots/30420.zarr/thomson_scattering/* ./30420.zarr/thomson_scattering;

Finally, we can open the file locally:

xr.open_zarr('30420.zarr/thomson_scattering', consolidated=False)

/srv/fair-mast/.docs-venv/lib/python3.12/site-packages/zarr/core/group.py:551: UserWarning: Both zarr.json (Zarr format 3) and .zgroup (Zarr format 2) metadata objects exist at file:///srv/fair-mast/docs/30420.zarr/thomson_scattering. Zarr format 3 will be used.
  warnings.warn(msg, stacklevel=1)
/srv/fair-mast/.docs-venv/lib/python3.12/site-packages/zarr/core/group.py:3376: UserWarning: Object at .zattrs is not recognized as a component of a Zarr hierarchy.
  warnings.warn(
/srv/fair-mast/.docs-venv/lib/python3.12/site-packages/zarr/core/group.py:3376: UserWarning: Object at .zgroup is not recognized as a component of a Zarr hierarchy.
  warnings.warn(

<xarray.Dataset> Size: 257kB
Dimensions:       (time: 88, major_radius: 120)
Coordinates:
  * major_radius  (major_radius) float64 960B 0.3 0.31 0.32 ... 1.47 1.48 1.49
  * time          (time) float64 704B -0.0568 -0.0518 -0.0468 ... 0.3732 0.3782
Data variables:
    n_e_core      (time) float64 704B ...
    p_e           (major_radius, time) float64 84kB ...
    n_e           (major_radius, time) float64 84kB ...
    t_e           (major_radius, time) float64 84kB ...
    t_e_core      (time) float64 704B ...
Attributes:
    name:          thomson_scattering
    description:   Thomson scattering diagnostic
    imas:          thomson_scattering
    license_name:  Creative Commons 4.0 BY-SA
    license_url:   https://creativecommons.org/licenses/by-sa/4.0/

xarray.Dataset

• Dimensions:
  • time: 88
  • major_radius: 120
• Coordinates: (2)
  • major_radius
    (major_radius)
    float64
    0.3 0.31 0.32 ... 1.47 1.48 1.49

    description :

    label :

    major radius

    imas :

    thomson_scattering.channel[:].position.r

    units :

    m

    array([0.3 , 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4 , 0.41,
           0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53,
           0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63, 0.64, 0.65,
           0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77,
           0.78, 0.79, 0.8 , 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89,
           0.9 , 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.  , 1.01,
           1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.1 , 1.11, 1.12, 1.13,
           1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.2 , 1.21, 1.22, 1.23, 1.24, 1.25,
           1.26, 1.27, 1.28, 1.29, 1.3 , 1.31, 1.32, 1.33, 1.34, 1.35, 1.36, 1.37,
           1.38, 1.39, 1.4 , 1.41, 1.42, 1.43, 1.44, 1.45, 1.46, 1.47, 1.48, 1.49])

  • time
    (time)
    float64
    -0.0568 -0.0518 ... 0.3732 0.3782

    units :

    s

    array([-0.0568, -0.0518, -0.0468, -0.0418, -0.0368, -0.0318, -0.0268, -0.0218,
           -0.0168, -0.0118, -0.0068, -0.0018,  0.0032,  0.0082,  0.0132,  0.0182,
            0.0232,  0.0282,  0.0332,  0.0382,  0.0432,  0.0482,  0.0532,  0.0582,
            0.0632,  0.0682,  0.0732,  0.0782,  0.0832,  0.0882,  0.0932,  0.0982,
            0.1032,  0.1082,  0.1132,  0.1182,  0.1232,  0.1282,  0.1332,  0.1382,
            0.1432,  0.1482,  0.1532,  0.1582,  0.1632,  0.1682,  0.1732,  0.1782,
            0.1832,  0.1882,  0.1932,  0.1982,  0.2032,  0.2082,  0.2132,  0.2182,
            0.2232,  0.2282,  0.2332,  0.2382,  0.2432,  0.2482,  0.2532,  0.2582,
            0.2632,  0.2682,  0.2732,  0.2782,  0.2832,  0.2882,  0.2932,  0.2982,
            0.3032,  0.3082,  0.3132,  0.3182,  0.3232,  0.3282,  0.3332,  0.3382,
            0.3432,  0.3482,  0.3532,  0.3582,  0.3632,  0.3682,  0.3732,  0.3782])

• Data variables: (5)
  • n_e_core
    (time)
    float64
    ...

    data_block_size :

    115304

    description :

    label :

    core density

    name :

    n_e_core

    uda_name :

    AYC_NE_CORE

    units :

    1 / m ** 3

    [88 values with dtype=float64]

  • p_e
    (major_radius, time)
    float64
    ...

    data_block_size :

    193352

    description :

    label :

    electron pressure

    name :

    p_e

    uda_name :

    AYC_PE

    units :

    u * a

    [10560 values with dtype=float64]

  • n_e
    (major_radius, time)
    float64
    ...

    data_block_size :

    193352

    description :

    Electron density

    label :

    electron density

    name :

    n_e

    uda_name :

    AYC_NE

    imas :

    thomson_scattering.channel[:].n_e

    units :

    1 / m ** 3

    [10560 values with dtype=float64]

  • t_e
    (major_radius, time)
    float64
    ...

    data_block_size :

    193352

    description :

    label :

    electron temperature

    name :

    t_e

    uda_name :

    AYC_TE

    imas :

    thomson_scattering.channel[:].t_e

    units :

    eV

    [10560 values with dtype=float64]

  • t_e_core
    (time)
    float64
    ...

    data_block_size :

    115304

    description :

    label :

    core temperature

    name :

    t_e_core

    uda_name :

    AYC_TE_CORE

    units :

    eV

    [88 values with dtype=float64]

• Indexes: (2)
  • major_radius
    PandasIndex

    PandasIndex(Index([                0.3,                0.31,                0.32,
                          0.33,                0.34, 0.35000000000000003,
           0.36000000000000004, 0.37000000000000005, 0.38000000000000006,
           0.39000000000000007,
           ...
             1.400000000000001,   1.410000000000001,   1.420000000000001,
             1.430000000000001,   1.440000000000001,   1.450000000000001,
             1.460000000000001,   1.470000000000001,   1.480000000000001,
             1.490000000000001],
          dtype='float64', name='major_radius', length=120))

  • time
    PandasIndex

    PandasIndex(Index([ -0.056800246238708496,    -0.0518002462387085,    -0.0468002462387085,
            -0.041800246238708504,  -0.036800246238708506,   -0.03180024623870851,
             -0.02680024623870851,  -0.021800246238708514,  -0.016800246238708516,
            -0.011800246238708519,  -0.006800246238708521, -0.0018002462387085236,
             0.003199753761291474,   0.008199753761291478,   0.013199753761291469,
              0.01819975376129146,   0.023199753761291464,    0.02819975376129147,
              0.03319975376129146,    0.03819975376129145,   0.043199753761291454,
              0.04819975376129146,    0.05319975376129145,    0.05819975376129144,
              0.06319975376129144,    0.06819975376129145,    0.07319975376129145,
              0.07819975376129143,    0.08319975376129143,    0.08819975376129144,
              0.09319975376129142,    0.09819975376129142,    0.10319975376129142,
              0.10819975376129143,    0.11319975376129143,    0.11819975376129141,
              0.12319975376129141,    0.12819975376129142,     0.1331997537612914,
               0.1381997537612914,     0.1431997537612914,     0.1481997537612914,
               0.1531997537612914,     0.1581997537612914,     0.1631997537612914,
               0.1681997537612914,    0.17319975376129138,    0.17819975376129138,
              0.18319975376129138,     0.1881997537612914,     0.1931997537612914,
               0.1981997537612914,     0.2031997537612914,    0.20819975376129135,
              0.21319975376129136,    0.21819975376129136,    0.22319975376129136,
              0.22819975376129137,    0.23319975376129137,    0.23819975376129138,
              0.24319975376129133,    0.24819975376129133,    0.25319975376129134,
              0.25819975376129134,    0.26319975376129134,    0.26819975376129135,
              0.27319975376129135,    0.27819975376129136,    0.28319975376129136,
               0.2881997537612913,     0.2931997537612913,     0.2981997537612913,
               0.3031997537612913,    0.30819975376129133,    0.31319975376129133,
              0.31819975376129134,     0.3231997537612913,     0.3281997537612913,
               0.3331997537612913,     0.3381997537612913,     0.3431997537612913,
               0.3481997537612913,     0.3531997537612913,     0.3581997537612913,
               0.3631997537612913,    0.36819975376129127,     0.3731997537612913,
               0.3781997537612913],
          dtype='float64', name='time'))

• Attributes: (5)

  name :

  thomson_scattering

  description :

  Thomson scattering diagnostic

  imas :

  thomson_scattering

  license_name :

  Creative Commons 4.0 BY-SA

  license_url :

  https://creativecommons.org/licenses/by-sa/4.0/