# Getting started ### Apolo Extras Documentation A set of tools and commands to extend the functionality of Apolo platform CLI client. ### Image Operations #### Copying Images Images can be copied between different projects, organizations, and clusters using the apolo-extras image transfer command. **Between Projects:** ```bash # Copy image from project1 to project2 within the same cluster apolo-extras image transfer \ image:/project/image:tag \ image:/target-project/image:tag ``` ```bash # Force overwrite existing image apolo-extras image transfer -F \ image:/project/image:tag \ image:/target-project/image:tag ``` **Between Clusters:** You need to image the image's full path including cluster, organization and project to be able to copy between clusters. ```bash # Copy image between different clusters apolo-extras image transfer \ image://cluster/organization/project/image:tag \ image://target-cluster/target-organization/target-project/image:tag ``` #### Building Images The platform provides two methods for building images: remote building using Kaniko (build) and local building using Docker (local-build). **Remote Build with layers cache:** ```bash # Basic build with default cache settings apolo-extras image build ./context image:: ``` ```bash # Disable build cache apolo-extras image build --no-cache ./context image:: ``` ```bash # Advanced cache control using Kaniko arguments apolo-extras image build --extra-kaniko-args="--cache-dir=/cache --cache-ttl=24h" ./context image:: ``` ```bash # Build with custom Dockerfile and build arguments apolo-extras image build -f custom.Dockerfile --build-arg VERSION=1.0 ./context image:: ``` ```bash # Build images using a specific hardware preset available in the cluster apolo-extras image build --preset ./context image:: ``` **Local Build:** ```bash # Basic local build apolo-extras image local-build ./context image:: ``` ```bash # Build with custom Dockerfile and verbose output apolo-extras image local-build -f custom.Dockerfile --verbose ./context image:: ``` **Working with External Registries** To work with external private registries, you'll need to configure authentication first. **Setting Up Registry Authentication:** You can generate an authentication config.json file for external registries using a username and password ```bash apolo-extras config build-registy-auth registry.external.com username password > docker.config.json ``` You can then use this config file to build images remotely and push to a private external registry ```bash apolo secret add registry_token @docker.config.json apolo-extras image build --preset cpu-small -e \ NE_REGISTRY_AUTH=secret:registry_token \ ./context \ //image:tag ``` You can also use your existing Docker Hub authentication to push images to a private Docker Hub repository. ```bash apolo secret add registry_token @${HOME}/.docker/config.json apolo-extras image build --preset cpu-small -e \ NE_REGISTRY_AUTH=secret:registry_token \ ./context \ //image:latest ``` It is possible to push images from your local machine to Apolo Platform Registry. Use the following command to save a config.json file with the authentication credentials. ```bash apolo-extras config save-registry-auth ${HOME}/.docker/config.json --cluster default ``` ### Data Operations #### Data Transfer Between Projects The apolo-extras data transfer command facilitates data movement between different internal storage locations. This is not supported with regular Apolo CLI apolo cp. Between directories in the same project: ```bash # Copy data between directories on the same project apolo-extras data transfer storage:folder1 storage:folder2 ``` Between Projects: ```bash # Copy data between projects on the same cluster apolo-extras data transfer storage:/project1/data storage:/project2/data # Copy data between disks apolo-extras data transfer disk:disk1:/data disk:disk2:/data ``` Between Clusters: ```bash # Copy data between clusters apolo-extras data transfer \ storage://cluster1/organization/project/directory \ storage://cluster2/organization/project/directory ``` #### External Storage Operations The platform supports various external storage systems with different authentication methods. **Google Cloud Storage (GCS)** Create a Apolo Secret containing GCP service account credentials to access the data on GCP. ```bash apolo secret add gcp-creds @path/to/credentials/file.json ``` You can then use it to start copy jobs to and from Google Cloud Storage. ```bash # Download and extract from GCS apolo-extras data cp -x -t \ -v secret:gcp-creds:/gcp-creds.txt \ -e GOOGLE_APPLICATION_CREDENTIALS=/gcp-creds.txt \ gs://bucket-name/dataset.tar.gz \ storage:/project/dataset # Upload to GCS with compression apolo-extras data cp -c \ -v secret:gcp-creds:/gcp-creds.txt \ -e GOOGLE_APPLICATION_CREDENTIALS=/gcp-creds.txt \ storage:/project/dataset \ gs://bucket-name/dataset.tar.gz ``` **AWS S3** Create a Apolo Secret containing AWS CLI tool credentials to access the data on AWS S3. ```bash apolo secret add s3-creds @path/to/credentials/file.json ``` You can then use it to start copy jobs to and from AWS S3. ```bash # Download and extract from S3 apolo-extras data cp -x -t \ -v secret:s3-creds:/s3-creds.txt \ -e AWS_SHARED_CREDENTIALS_FILE=/s3-creds.txt \ s3://bucket-name/dataset.tar.gz \ disk:disk-name:/project/dataset # Upload to S3 with compression apolo-extras data cp -c \ -v secret:s3-creds:/s3-creds.txt \ -e AWS_SHARED_CREDENTIALS_FILE=/s3-creds.txt \ disk:disk-name:/project/dataset \ s3://bucket-name/dataset.tar.gz ``` **Azure Blob Storage** Create a Apolo Secret containing credentials to access the data on Azure Blob Storage. ```bash apolo secret add azure-sas-token @path/to/credentials/file.json ``` You can then use it to start copy jobs to and from Azure Blob Storage. ```bash # Download and extract from Azure apolo-extras data cp -x -t \ -e AZURE_SAS_TOKEN=secret:azure-sas-token \ azure+https://storage-account.blob.core.windows.net/container/dataset.tar.gz \ storage:/project/dataset # Upload to Azure with compression apolo-extras data cp -c \ -e AZURE_SAS_TOKEN=secret:azure-sas-token \ storage:/project/dataset \ azure+https://storage-account.blob.core.windows.net/container/dataset.tar.gz ``` **HTTP/HTTPS Sources** ```bash # Download and extract from HTTP source apolo-extras data cp -x -t \ https://example.org/dataset.tar.gz \ disk:disk-name:/project/dataset # Download with custom preset for resource allocation apolo-extras data cp -s large-memory \ https://example.org/large-dataset.tar.gz \ storage:/project/dataset ``` #### Advanced Data Transfer Options Using Presets and Lifespans: ```bash # Use preset for resource allocation apolo-extras data cp -s high-bandwidth source destination # Set job lifespan apolo-extras data cp -l 3600 source destination # Combine preset and lifespan apolo-extras data cp -s high-bandwidth -l 7200 source destination ``` Working with Archives: ```bash # Extract specific archive types apolo-extras data cp -x source.tar.gz destination/ # For .tar.gz apolo-extras data cp -x source.zip destination/ # For .zip apolo-extras data cp -x source.tar.bz2 destination/ # For .tar.bz2 # Create archives with specific formats apolo-extras data cp -c source/ destination.tar.gz # Create .tar.gz apolo-extras data cp -c source/ destination.zip # Create .zip apolo-extras data cp -c source/ destination.tar.bz2 # Create .tar.bz2 ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.apolo.us/index/apolo-extras-cli/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.