Secrets

Secrets allow you to inject secret values into your runs, such as API keys, passwords, access keys, or any other sensitive information. These secrets can be made accessible inside your run either through a mounted text file or, more typically, as an environment variable. Secrets are available across all the user’s clusters.

Secrets can be added as environment variables:

mcli create secret env NAME=my-super-secret-name

✔  Created environment secret: name
✔  Synced to all clusters

The above secret is now available inside your runs as the environment variable NAME with the value my-super-secret-name.

Secrets can also be injected as mounted text files:

mcli create secret mounted

Modifying Secrets

Secrets by design are not modifiable. To edit an existing secret, delete the secret with mcli delete secret <secret-name> and then recreate the secret.

Common Secrets

For convenience, MCLI also supports a few common secrets that can be created with mcli create secret <>:

Type	Argument	Purpose	Link
Github	git-ssh	Access private github repositories	SSH Secrets
S3	s3	Access AWS S3 buckets	S3 Secrets
OCI	oci	Access OCI storage buckets	OCI
SFTP	sftp-ssh	Stream data and upload results	SSH Secrets
Container registry	docker	Access private container repository (e.g. dockerhub)	Docker Secrets

Simple Example

In this example, we create a secret, and then show how to access that within a run. First, create an environment variable secret with:

mcli create secret env SECRET_STUFF='super-secret-name'

Now, let’s run a simple “Hello World” run, except we access the injected secret in the command with echo Hello $SECRET_STUFF:

name: hello-secret-user
gpu_type: a100_80gb
gpu_num: 1
cluster: r1z1
image: python
command: |
  echo Hello $SECRET_STUFF!

Creating a run with the above YAML should yield:

> mcli run -f hello-secret-user.yaml

i  Run hello-secret-user submitted. Waiting for it to start...
i  You can press Ctrl+C to quit and follow your run manually.
✔  Run hello-secret-user started
i  Following run logs. Press Ctrl+C to quit.

Hello super-secret-name!

Clean up by deleting this secret:

mcli delete secret secret-stuff

W&B Example

Weights & Biases, a popular experiment tracking tool, read its API Key from an environment variable named WANDB_API_KEY.

To enable this within MosaicML platform, create an environment variable secret as:

mcli create secret env --name wandb WANDB_API_KEY=<your-wandb-api-key>

✔  Created secret: wandb

Then, test the API key was added correctly with a small test wandb login call:

name: wandb-login
gpu_num: 0
image: python
integrations:
  - integration_type: pip_packages
    packages:
      - wandb
command: |
  wandb login