job

Job operations

Usage

apolo job [OPTIONS] COMMAND [ARGS]...

Job operations.

Commands:

Usage	Description
attach	Attach terminal to a job
browse	Opens a job's URL in a web browser
bump-life-span	Increase job life span
exec	Execute command in a running job
generate-run-command	Generate command that will rerun given job
kill	Kill job(s)
logs	Print the logs for a job
ls	List all jobs
port-forward	Forward port(s) of a job
run	Run a job
save	Save job's state to an image
status	Display status of a job
top	Display GPU/CPU/Memory usage

attach

Attach terminal to a job

Usage

apolo job attach [OPTIONS] JOB

Attach terminal to a job

Attach local standard input, output, and error streams to a running job.

Options

Name	Description
--help	Show this message and exit.
--port-forward LOCAL_PORT:REMOTE_RORT	Forward port(s) of a running job to local port(s) (use multiple times for forwarding several ports)

browse

Opens a job's URL in a web browser

Usage

apolo job browse [OPTIONS] JOB

Opens a job's URL in a web browser.

Options

Name	Description
--help	Show this message and exit.

bump-life-span

Increase job life span

Usage

apolo job bump-life-span [OPTIONS] JOB TIMEDELTA

Increase job life span

Options

Name	Description
--help	Show this message and exit.

exec

Execute command in a running job

Usage

apolo job exec [OPTIONS] JOB -- CMD...

Execute command in a running job.

Examples

# Provides a shell to the container:
$ apolo exec my-job -- /bin/bash

# Executes a single command in the container and returns the control:
$ apolo exec --no-tty my-job -- ls -l

Options

Name	Description
--help	Show this message and exit.
-t, --tty / -T, --no-tty	Allocate a TTY, can be useful for interactive jobs. By default is on if the command is executed from a terminal, non-tty mode is used if executed from a script.

generate-run-command

Generate command that will rerun given job

Usage

apolo job generate-run-command [OPTIONS] JOB

Generate command that will rerun given job.

Examples

# You can use the following to directly re-execute it:
$ eval $(apolo job generate-run-command <job-id>)

Options

Name	Description
--help	Show this message and exit.

kill

Kill job(s)

Usage

apolo job kill [OPTIONS] JOBS...

Kill job(s).

Options

Name	Description
--help	Show this message and exit.

logs

Print the logs for a job

Usage

apolo job logs [OPTIONS] JOB

Print the logs for a job.

Options

Name	Description
--help	Show this message and exit.
--since DATE_OR_TIMEDELTA	Only return logs after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time.
--timestamps	Include timestamps on each line in the log output.

ls

List all jobs

Usage

apolo job ls [OPTIONS]

List all jobs.

Examples

$ apolo ps -a
$ apolo ps -a --owner=user-1 --owner=user-2
$ apolo ps --name my-experiments-v1 -s failed -s succeeded
$ apolo ps --description=my favourite job
$ apolo ps -s failed -s succeeded -q
$ apolo ps -t tag1 -t tag2

Options

Name	Description
--help	Show this message and exit.
-a, --all	Show all jobs regardless the status.
--all-orgs	Show jobs in all orgs.
--all-projects	Show jobs in all projects.
--cluster CLUSTER	Show jobs on a specified cluster (the current cluster by default).
-d, --description DESCRIPTION	Filter out jobs by description (exact match).
--distinct	Show only first job if names are same.
--format COLUMNS	Output table format, see "apolo help ps-format" for more info about the format specification. The default can be changed using the job.ps-format configuration variable documented in "apolo help user-config"
--full-uri	Output full image URI.
-n, --name NAME	Filter out jobs by name.
--org ORG	Filter out jobs by org name (multiple option, the current org by default).
-o, --owner TEXT	Filter out jobs by owner (multiple option). Supports ME option to filter by the current user.
-p, --project PROJECT	Filter out jobs by project name (multiple option, the current project by default).
--recent-first / --recent-last	Show newer jobs first or last
--since DATE_OR_TIMEDELTA	Show jobs created after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time.
-s, --status [pending | suspended | running | succeeded | failed | cancelled]	Filter out jobs by status (multiple option).
-t, --tag TAG	Filter out jobs by tag (multiple option)
--until DATE_OR_TIMEDELTA	Show jobs created before a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time.
-w, --wide	Do not cut long lines for terminal width.

port-forward

Forward port(s) of a job

Usage

apolo job port-forward [OPTIONS] JOB LOCAL_PORT:REMOTE_RORT...

Forward port(s) of a job.

Forwards port(s) of a running job to local port(s).

Examples

# Forward local port 2080 to port 80 of job's container.
# You can use http://localhost:2080 in browser to access job's served http
$ apolo job port-forward my-fastai-job 2080:80

# Forward local port 2222 to job's port 22
# Then copy all data from container's folder '/data' to current folder
# (please run second command in other terminal)
$ apolo job port-forward my-job-with-ssh-server 2222:22
$ rsync -avxzhe ssh -p 2222 root@localhost:/data .

# Forward few ports at once
$ apolo job port-forward my-job 2080:80 2222:22 2000:100

Options

Name	Description
--help	Show this message and exit.

run

Run a job

Usage

apolo job run [OPTIONS] IMAGE [-- CMD...]

Run a job

IMAGE docker image name to run in a job.

CMD list will be passed as arguments to the executed job's image.

Examples

# Starts a container pytorch/pytorch:latest on a machine with smaller GPU resources
# (see exact values in `apolo config show`) and with two volumes mounted:
#   storage:/<home-directory>   --> /var/storage/home (in read-write mode),
#   storage:/neuromation/public --> /var/storage/neuromation (in read-only mode).
$ apolo run --preset=gpu-small --volume=storage::/var/storage/home:rw \
$ --volume=storage:/neuromation/public:/var/storage/home:ro pytorch/pytorch:latest

# Starts a container using the custom image my-ubuntu:latest stored in apolo
# registry, run /script.sh and pass arg1 and arg2 as its arguments:
$ apolo run -s cpu-small --entrypoint=/script.sh image:my-ubuntu:latest -- arg1 arg2

Options

Name	Description
--help	Show this message and exit.
--browse	Open a job's URL in a web browser
--cluster CLUSTER	Run job in a specified cluster
-d, --description DESC	Optional job description in free format
--detach	Don't attach to job logs and don't wait for exit code
--energy-schedule NAME	Run job only within a selected energy schedule. Selected preset should have scheduler enabled.
--entrypoint TEXT	Executable entrypoint in the container (note that it overwrites ENTRYPOINT and CMD instructions of the docker image)
-e, --env VAR=VAL	Set environment variable in container. Use multiple options to define more than one variable. See apolo help secrets for information about passing secrets as environment variables.
--env-file PATH	File with environment variables to pass
-x, --extshm / -X, --no-extshm	Request extended '/dev/shm' space [default: x]
--http-auth / --no-http-auth	Enable HTTP authentication for forwarded HTTP port [default: True]
--http-port PORT	Enable HTTP port forwarding to container [default: 80]
--life-span TIMEDELTA	Optional job run-time limit in the format '1d2h3m4s' (some parts may be missing). Set '0' to disable. Default value '1d' can be changed in the user config.
-n, --name NAME	Optional job name
--org ORG	Run job in a specified org
--pass-config / --no-pass-config	Upload apolo config to the job [default: no-pass-config]
--port-forward LOCAL_PORT:REMOTE_RORT	Forward port(s) of a running job to local port(s) (use multiple times for forwarding several ports)
-s, --preset PRESET	Predefined resource configuration (to see available values, run apolo config show)
--priority [low | normal | high]	Priority used to specify job's start order. Jobs with higher priority will start before ones with lower priority. Priority should be supported by cluster.
--privileged	Run job in privileged mode, if it is supported by cluster.
-p, --project PROJECT	Run job in a specified project.
--restart [never | on-failure | always]	Restart policy to apply when a job exits [default: never]
--schedule-timeout TIMEDELTA	Optional job schedule timeout in the format '3m4s' (some parts may be missing).
--share USER	Share job write permissions to user or role.
--tag TAG	Optional job tag, multiple values allowed
-t, --tty / -T, --no-tty	Allocate a TTY, can be useful for interactive jobs. By default is on if the command is executed from a terminal, non-tty mode is used if executed from a script.
-v, --volume MOUNT	Mounts directory from vault into container. Use multiple options to mount more than one volume. See apolo help secrets for information about passing secrets as mounted files.
--wait-for-seat / --no-wait-for-seat	Wait for total running jobs quota [default: no-wait-for-seat]
--wait-start / --no-wait-start	Wait for a job start or failure [default: wait-start]
-w, --workdir TEXT	Working directory inside the container

save

Save job's state to an image

Usage

apolo job save [OPTIONS] JOB IMAGE

Save job's state to an image.

Examples

$ apolo job save job-id image:ubuntu-patched
$ apolo job save my-favourite-job image:ubuntu-patched:v1
$ apolo job save my-favourite-job image://bob/ubuntu-patched

Options

Name	Description
--help	Show this message and exit.

status

Display status of a job

Usage

apolo job status [OPTIONS] JOB

Display status of a job.

Options

Name	Description
--help	Show this message and exit.
--full-uri	Output full URI.

top

Display GPU/CPU/Memory usage

Usage

apolo job top [OPTIONS] [JOBS]...

Display GPU/CPU/Memory usage.

Examples

$ apolo top
$ apolo top job-1 job-2
$ apolo top --owner=user-1 --owner=user-2
$ apolo top --name my-experiments-v1
$ apolo top -t tag1 -t tag2

Options

Name	Description
--help	Show this message and exit.
--cluster CLUSTER	Show jobs on a specified cluster (the current cluster by default).
-d, --description DESCRIPTION	Filter out jobs by description (exact match).
--format COLUMNS	Output table format, see "apolo help top-format" for more info about the format specification. The default can be changed using the job.top-format configuration variable documented in "apolo help user-config"
--full-uri	Output full image URI.
-n, --name NAME	Filter out jobs by name.
-o, --owner TEXT	Filter out jobs by owner (multiple option). Supports ME option to filter by the current user. Specify ALL to show jobs of all users.
-p, --project PROJECT	Filter out jobs by project name (multiple option).
--since DATE_OR_TIMEDELTA	Show jobs created after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time.
--sort COLUMNS	Sort rows by specified column. Add "-" prefix to revert the sorting order. Multiple columns can be specified (comma separated). [default: cpu]
-t, --tag TAG	Filter out jobs by tag (multiple option)
--timeout FLOAT	Maximum allowed time for executing the command, 0 for no timeout [default: 0]
--until DATE_OR_TIMEDELTA	Show jobs created before a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time.