!pip install aiohttp requests s5cmd "xarray[io]"

Bulk Data Download

This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!

We can download data in bulk using any command line that supports the S3 protocol. We recommend using the s5cmd tool, which can be simply installed by running:

pip install s5cmd

Now we can download data using the cp command.

In this example, we are going to transfer the thompson scattering data for shot 30420 locally.

We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.

%%capture --no-display
%%bash
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level2/shots/30420.zarr/thomson_scattering/* ./30420.zarr/thomson_scattering;

Finally, we can open the file locally:

import xarray as xr
xr.open_zarr('30420.zarr/thomson_scattering', consolidated=False)

<xarray.Dataset> Size: 257kB
Dimensions:       (time: 88, major_radius: 120)
Coordinates:
  * time          (time) float64 704B -0.0568 -0.0518 -0.0468 ... 0.3732 0.3782
  * major_radius  (major_radius) float64 960B 0.3 0.31 0.32 ... 1.47 1.48 1.49
Data variables:
    n_e           (time, major_radius) float64 84kB ...
    p_e           (time, major_radius) float64 84kB ...
    t_e_core      (time) float64 704B ...
    n_e_core      (time) float64 704B ...
    t_e           (time, major_radius) float64 84kB ...
Attributes:
    description:  
    imas:         thomson_scattering
    label:        core temperature
    name:         thomson_scattering
    uda_name:     AYC_TE_CORE
    units:        eV

xarray.Dataset

• Dimensions:
  • time: 88
  • major_radius: 120
• Coordinates: (2)
  • time
    (time)
    float64
    -0.0568 -0.0518 ... 0.3732 0.3782

    units :

    s

    array([-0.0568, -0.0518, -0.0468, -0.0418, -0.0368, -0.0318, -0.0268, -0.0218,
           -0.0168, -0.0118, -0.0068, -0.0018,  0.0032,  0.0082,  0.0132,  0.0182,
            0.0232,  0.0282,  0.0332,  0.0382,  0.0432,  0.0482,  0.0532,  0.0582,
            0.0632,  0.0682,  0.0732,  0.0782,  0.0832,  0.0882,  0.0932,  0.0982,
            0.1032,  0.1082,  0.1132,  0.1182,  0.1232,  0.1282,  0.1332,  0.1382,
            0.1432,  0.1482,  0.1532,  0.1582,  0.1632,  0.1682,  0.1732,  0.1782,
            0.1832,  0.1882,  0.1932,  0.1982,  0.2032,  0.2082,  0.2132,  0.2182,
            0.2232,  0.2282,  0.2332,  0.2382,  0.2432,  0.2482,  0.2532,  0.2582,
            0.2632,  0.2682,  0.2732,  0.2782,  0.2832,  0.2882,  0.2932,  0.2982,
            0.3032,  0.3082,  0.3132,  0.3182,  0.3232,  0.3282,  0.3332,  0.3382,
            0.3432,  0.3482,  0.3532,  0.3582,  0.3632,  0.3682,  0.3732,  0.3782])

  • major_radius
    (major_radius)
    float64
    0.3 0.31 0.32 ... 1.47 1.48 1.49

    description :

    imas :

    thomson_scattering.channel[:].position.r

    label :

    major radius

    units :

    m

    array([0.3 , 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4 , 0.41,
           0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53,
           0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63, 0.64, 0.65,
           0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77,
           0.78, 0.79, 0.8 , 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89,
           0.9 , 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.  , 1.01,
           1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.1 , 1.11, 1.12, 1.13,
           1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.2 , 1.21, 1.22, 1.23, 1.24, 1.25,
           1.26, 1.27, 1.28, 1.29, 1.3 , 1.31, 1.32, 1.33, 1.34, 1.35, 1.36, 1.37,
           1.38, 1.39, 1.4 , 1.41, 1.42, 1.43, 1.44, 1.45, 1.46, 1.47, 1.48, 1.49])

• Data variables: (5)
  • n_e
    (time, major_radius)
    float64
    ...

    description :

    imas :

    thomson_scattering.channel[:].n_e

    label :

    electron density

    name :

    n_e

    uda_name :

    AYC_NE

    units :

    1 / m ** 3

    [10560 values with dtype=float64]

  • p_e
    (time, major_radius)
    float64
    ...

    description :

    label :

    electron pressure

    name :

    p_e

    uda_name :

    AYC_PE

    units :

    u * a

    [10560 values with dtype=float64]

  • t_e_core
    (time)
    float64
    ...

    description :

    label :

    core temperature

    name :

    t_e_core

    uda_name :

    AYC_TE_CORE

    units :

    eV

    [88 values with dtype=float64]

  • n_e_core
    (time)
    float64
    ...

    description :

    label :

    core density

    name :

    n_e_core

    uda_name :

    AYC_NE_CORE

    units :

    1 / m ** 3

    [88 values with dtype=float64]

  • t_e
    (time, major_radius)
    float64
    ...

    description :

    imas :

    thomson_scattering.channel[:].t_e

    label :

    electron temperature

    name :

    t_e

    uda_name :

    AYC_TE

    units :

    eV

    [10560 values with dtype=float64]

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(Index([ -0.056800246238708496,    -0.0518002462387085,    -0.0468002462387085,
            -0.041800246238708504,  -0.036800246238708506,   -0.03180024623870851,
             -0.02680024623870851,  -0.021800246238708514,  -0.016800246238708516,
            -0.011800246238708519,  -0.006800246238708521, -0.0018002462387085236,
             0.003199753761291474,   0.008199753761291478,   0.013199753761291469,
              0.01819975376129146,   0.023199753761291464,    0.02819975376129147,
              0.03319975376129146,    0.03819975376129145,   0.043199753761291454,
              0.04819975376129146,    0.05319975376129145,    0.05819975376129144,
              0.06319975376129144,    0.06819975376129145,    0.07319975376129145,
              0.07819975376129143,    0.08319975376129143,    0.08819975376129144,
              0.09319975376129142,    0.09819975376129142,    0.10319975376129142,
              0.10819975376129143,    0.11319975376129143,    0.11819975376129141,
              0.12319975376129141,    0.12819975376129142,     0.1331997537612914,
               0.1381997537612914,     0.1431997537612914,     0.1481997537612914,
               0.1531997537612914,     0.1581997537612914,     0.1631997537612914,
               0.1681997537612914,    0.17319975376129138,    0.17819975376129138,
              0.18319975376129138,     0.1881997537612914,     0.1931997537612914,
               0.1981997537612914,     0.2031997537612914,    0.20819975376129135,
              0.21319975376129136,    0.21819975376129136,    0.22319975376129136,
              0.22819975376129137,    0.23319975376129137,    0.23819975376129138,
              0.24319975376129133,    0.24819975376129133,    0.25319975376129134,
              0.25819975376129134,    0.26319975376129134,    0.26819975376129135,
              0.27319975376129135,    0.27819975376129136,    0.28319975376129136,
               0.2881997537612913,     0.2931997537612913,     0.2981997537612913,
               0.3031997537612913,    0.30819975376129133,    0.31319975376129133,
              0.31819975376129134,     0.3231997537612913,     0.3281997537612913,
               0.3331997537612913,     0.3381997537612913,     0.3431997537612913,
               0.3481997537612913,     0.3531997537612913,     0.3581997537612913,
               0.3631997537612913,    0.36819975376129127,     0.3731997537612913,
               0.3781997537612913],
          dtype='float64', name='time'))

  • major_radius
    PandasIndex

    PandasIndex(Index([                0.3,                0.31,                0.32,
                          0.33,                0.34, 0.35000000000000003,
           0.36000000000000004, 0.37000000000000005, 0.38000000000000006,
           0.39000000000000007,
           ...
             1.400000000000001,   1.410000000000001,   1.420000000000001,
             1.430000000000001,   1.440000000000001,   1.450000000000001,
             1.460000000000001,   1.470000000000001,   1.480000000000001,
             1.490000000000001],
          dtype='float64', name='major_radius', length=120))

• Attributes: (6)

  description :

  imas :

  thomson_scattering

  label :

  core temperature

  name :

  thomson_scattering

  uda_name :

  AYC_TE_CORE

  units :

  eV