# job Job operations ## Usage ```bash apolo job [OPTIONS] COMMAND [ARGS]... ``` Job operations. **Commands:** | Usage | Description | | ----------------------------------------------- | ------------------------------------------ | | [*attach*](#attach) | Attach terminal to a job | | [*browse*](#browse) | Opens a job's URL in a web browser | | [*bump-life-span*](#bump-life-span) | Increase job life span | | [*exec*](#exec) | Execute command in a running job | | [*generate-run-command*](#generate-run-command) | Generate command that will rerun given job | | [*kill*](#kill) | Kill job(s) | | [*logs*](#logs) | Print the logs for a job | | [*ls*](#ls) | List all jobs | | [*port-forward*](#port-forward) | Forward port(s) of a job | | [*run*](#run) | Run a job | | [*save*](#save) | Save job's state to an image | | [*status*](#status) | Display status of a job | | [*top*](#top) | Display GPU/CPU/Memory usage | ### attach Attach terminal to a job #### Usage ```bash apolo job attach [OPTIONS] JOB ``` Attach terminal to a job Attach local standard input, output, and error streams to a running job. #### Options | Name | Description | | ----------------------------------------- | --------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *--port-forward LOCAL\_PORT:REMOTE\_RORT* | Forward port(s) of a running job to local port(s) (use multiple times for forwarding several ports) | ### browse Opens a job's URL in a web browser #### Usage ```bash apolo job browse [OPTIONS] JOB ``` Opens a job's `URL` in a web browser. #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### bump-life-span Increase job life span #### Usage ```bash apolo job bump-life-span [OPTIONS] JOB TIMEDELTA ``` Increase job life span #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### exec Execute command in a running job #### Usage ```bash apolo job exec [OPTIONS] JOB -- CMD... ``` Execute command in a running job. #### Examples ```bash # Provides a shell to the container: $ apolo exec my-job -- /bin/bash # Executes a single command in the container and returns the control: $ apolo exec --no-tty my-job -- ls -l ``` #### Options | Name | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *-t, --tty / -T, --no-tty* | Allocate a TTY, can be useful for interactive jobs. By default is on if the command is executed from a terminal, non-tty mode is used if executed from a script. | ### generate-run-command Generate command that will rerun given job #### Usage ```bash apolo job generate-run-command [OPTIONS] JOB ``` Generate command that will rerun given job. #### Examples ```bash # You can use the following to directly re-execute it: $ eval $(apolo job generate-run-command ) ``` #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### kill Kill job(s) #### Usage ```bash apolo job kill [OPTIONS] JOBS... ``` Kill job(s). #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### logs Print the logs for a job #### Usage ```bash apolo job logs [OPTIONS] JOB ``` Print the logs for a job. #### Options | Name | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *--since DATE\_OR\_TIMEDELTA* | Only return logs after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time. | | *--timestamps* | Include timestamps on each line in the log output. | ### ls List all jobs #### Usage ```bash apolo job ls [OPTIONS] ``` List all jobs. #### Examples ```bash $ apolo ps -a $ apolo ps -a --owner=user-1 --owner=user-2 $ apolo ps --name my-experiments-v1 -s failed -s succeeded $ apolo ps --description=my favourite job $ apolo ps -s failed -s succeeded -q $ apolo ps -t tag1 -t tag2 ``` #### Options | Name | Description | | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *-a, --all* | Show all jobs regardless the status. | | *--all-orgs* | Show jobs in all orgs. | | *--all-projects* | Show jobs in all projects. | | *--cluster CLUSTER* | Show jobs on a specified cluster (the current cluster by default). | | *-d, --description DESCRIPTION* | Filter out jobs by description (exact match). | | *--distinct* | Show only first job if names are same. | | *--format COLUMNS* | Output table format, see "apolo help ps-format" for more info about the format specification. The default can be changed using the job.ps-format configuration variable documented in "apolo help user-config" | | *--full-uri* | Output full image URI. | | *-n, --name NAME* | Filter out jobs by name. | | *--org ORG* | Filter out jobs by org name (multiple option, the current org by default). | | *-o, --owner TEXT* | Filter out jobs by owner (multiple option). Supports `ME` option to filter by the current user. | | *-p, --project PROJECT* | Filter out jobs by project name (multiple option, the current project by default). | | *--recent-first / --recent-last* | Show newer jobs first or last | | *--since DATE\_OR\_TIMEDELTA* | Show jobs created after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time. | | *-s, --status \[pending \| suspended \| running \| succeeded \| failed \| cancelled]* | Filter out jobs by status (multiple option). | | *-t, --tag TAG* | Filter out jobs by tag (multiple option) | | *--until DATE\_OR\_TIMEDELTA* | Show jobs created before a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time. | | *-w, --wide* | Do not cut long lines for terminal width. | ### port-forward Forward port(s) of a job #### Usage ```bash apolo job port-forward [OPTIONS] JOB LOCAL_PORT:REMOTE_RORT... ``` Forward port(s) of a job. Forwards port(s) of a running job to local port(s). #### Examples ```bash # Forward local port 2080 to port 80 of job's container. # You can use http://localhost:2080 in browser to access job's served http $ apolo job port-forward my-fastai-job 2080:80 # Forward local port 2222 to job's port 22 # Then copy all data from container's folder '/data' to current folder # (please run second command in other terminal) $ apolo job port-forward my-job-with-ssh-server 2222:22 $ rsync -avxzhe ssh -p 2222 root@localhost:/data . # Forward few ports at once $ apolo job port-forward my-job 2080:80 2222:22 2000:100 ``` #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### run Run a job #### Usage ```bash apolo job run [OPTIONS] IMAGE [-- CMD...] ``` Run a job `IMAGE` docker image name to run in a job. `CMD` list will be passed as arguments to the executed job's image. #### Examples ```bash # Starts a container pytorch/pytorch:latest on a machine with smaller GPU resources # (see exact values in `apolo config show`) and with two volumes mounted: # storage:/ --> /var/storage/home (in read-write mode), # storage:/neuromation/public --> /var/storage/neuromation (in read-only mode). $ apolo run --preset=gpu-small --volume=storage::/var/storage/home:rw \ $ --volume=storage:/neuromation/public:/var/storage/home:ro pytorch/pytorch:latest # Starts a container using the custom image my-ubuntu:latest stored in apolo # registry, run /script.sh and pass arg1 and arg2 as its arguments: $ apolo run -s cpu-small --entrypoint=/script.sh image:my-ubuntu:latest -- arg1 arg2 ``` #### Options | Name | Description | | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *--browse* | Open a job's URL in a web browser | | *--cluster CLUSTER* | Run job in a specified cluster | | *-d, --description DESC* | Optional job description in free format | | *--detach* | Don't attach to job logs and don't wait for exit code | | *--energy-schedule NAME* | Run job only within a selected energy schedule. Selected preset should have scheduler enabled. | | *--entrypoint TEXT* | Executable entrypoint in the container (note that it overwrites `ENTRYPOINT` and `CMD` instructions of the docker image) | | *-e, --env VAR=VAL* | Set environment variable in container. Use multiple options to define more than one variable. See `apolo help secrets` for information about passing secrets as environment variables. | | *--env-file PATH* | File with environment variables to pass | | *-x, --extshm / -X, --no-extshm* | Request extended '/dev/shm' space *\[default: x]* | | *--http-auth / --no-http-auth* | Enable HTTP authentication for forwarded HTTP port *\[default: True]* | | *--http-port PORT* | Enable HTTP port forwarding to container *\[default: 80]* | | *--life-span TIMEDELTA* | Optional job run-time limit in the format '1d2h3m4s' (some parts may be missing). Set '0' to disable. Default value '1d' can be changed in the user config. | | *-n, --name NAME* | Optional job name | | *--org ORG* | Run job in a specified org | | *--pass-config / --no-pass-config* | Upload apolo config to the job *\[default: no-pass-config]* | | *--port-forward LOCAL\_PORT:REMOTE\_RORT* | Forward port(s) of a running job to local port(s) (use multiple times for forwarding several ports) | | *-s, --preset PRESET* | Predefined resource configuration (to see available values, run `apolo config show`) | | *--priority \[low \| normal \| high]* | Priority used to specify job's start order. Jobs with higher priority will start before ones with lower priority. Priority should be supported by cluster. | | *--privileged* | Run job in privileged mode, if it is supported by cluster. | | *-p, --project PROJECT* | Run job in a specified project. | | *--restart \[never \| on-failure \| always]* | Restart policy to apply when a job exits *\[default: never]* | | *--schedule-timeout TIMEDELTA* | Optional job schedule timeout in the format '3m4s' (some parts may be missing). | | *--share USER* | Share job write permissions to user or role. | | *--tag TAG* | Optional job tag, multiple values allowed | | *-t, --tty / -T, --no-tty* | Allocate a TTY, can be useful for interactive jobs. By default is on if the command is executed from a terminal, non-tty mode is used if executed from a script. | | *-v, --volume MOUNT* | Mounts directory from vault into container. Use multiple options to mount more than one volume. See `apolo help secrets` for information about passing secrets as mounted files. | | *--wait-for-seat / --no-wait-for-seat* | Wait for total running jobs quota *\[default: no-wait-for-seat]* | | *--wait-start / --no-wait-start* | Wait for a job start or failure *\[default: wait-start]* | | *-w, --workdir TEXT* | Working directory inside the container | ### save Save job's state to an image #### Usage ```bash apolo job save [OPTIONS] JOB IMAGE ``` Save job's state to an image. #### Examples ```bash $ apolo job save job-id image:ubuntu-patched $ apolo job save my-favourite-job image:ubuntu-patched:v1 $ apolo job save my-favourite-job image://bob/ubuntu-patched ``` #### Options | Name | Description | | -------- | --------------------------- | | *--help* | Show this message and exit. | ### status Display status of a job #### Usage ```bash apolo job status [OPTIONS] JOB ``` Display status of a job. #### Options | Name | Description | | ------------ | --------------------------- | | *--help* | Show this message and exit. | | *--full-uri* | Output full URI. | ### top Display GPU/CPU/Memory usage #### Usage ```bash apolo job top [OPTIONS] [JOBS]... ``` Display `GPU`/`CPU`/Memory usage. #### Examples ```bash $ apolo top $ apolo top job-1 job-2 $ apolo top --owner=user-1 --owner=user-2 $ apolo top --name my-experiments-v1 $ apolo top -t tag1 -t tag2 ``` #### Options | Name | Description | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *--help* | Show this message and exit. | | *--cluster CLUSTER* | Show jobs on a specified cluster (the current cluster by default). | | *-d, --description DESCRIPTION* | Filter out jobs by description (exact match). | | *--format COLUMNS* | Output table format, see "apolo help top-format" for more info about the format specification. The default can be changed using the job.top-format configuration variable documented in "apolo help user-config" | | *--full-uri* | Output full image URI. | | *-n, --name NAME* | Filter out jobs by name. | | *-o, --owner TEXT* | Filter out jobs by owner (multiple option). Supports `ME` option to filter by the current user. Specify `ALL` to show jobs of all users. | | *-p, --project PROJECT* | Filter out jobs by project name (multiple option). | | *--since DATE\_OR\_TIMEDELTA* | Show jobs created after a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time. | | *--sort COLUMNS* | Sort rows by specified column. Add "-" prefix to revert the sorting order. Multiple columns can be specified (comma separated). *\[default: cpu]* | | *-t, --tag TAG* | Filter out jobs by tag (multiple option) | | *--timeout FLOAT* | Maximum allowed time for executing the command, 0 for no timeout *\[default: 0]* | | *--until DATE\_OR\_TIMEDELTA* | Show jobs created before a specific date (including). Use value of format '1d2h3m4s' to specify moment in past relatively to current time. |