Locked History Actions

Tool Shed

Galaxy Main Tool Shed

Contents

  1. What is the tool shed?
  2. Tool sheds supported by the core Galaxy development team
  3. Scripts for inspecting and maintaining the repositories contained within a tool shed
  4. History of the tool shed
  5. Related Pages
    1. Overview of tool shed repository features
    2. The contents of your tool shed repository
    3. Reviewing, rating and approving tool shed repositories
    4. Installing, maintaining and uninstalling tool shed repositories within a Galaxy instance
    5. Searching the tool shed
    6. The tool shed API
    7. Configuring and hosting your own local tool shed

What is the tool shed?

The Galaxy Tool Shed enables sharing of Galaxy tools across the Galaxy community.  The biggest misconception of the Galaxy tool shed is that it is a source code repository for developing Galaxy tools.  It is not!!  If you treat the tool shed as a source code repository for your tools while they are in development, you will undoubtedly experience unexpected  (and potentially undesirable) behavior by the tool shed.

The intent of the main Galaxy tool shed is to enable sharing of functionally correct (already developed) Galaxy tools between the many local Galaxy instances around the world. The mercurial repositories that are available in the main Galaxy tool shed can be "hg cloned" individually or "installed" individually as a means of making their contents (Galaxy tools, workflows, data, etc) available to your local Galaxy instance. This provides flexibility to those hosting their own local Galaxy instances in that they can install only those tools in which they have interest, and are not forced to get all of them in order to get any one of them.

If you are used to Galaxy, the tool shed will have a flavor of familiarity. Along with the top menu bar, the blue menu panel on the left provides access to the tool shed's features. Your initial visit to the tool shed will display the list of names by which tool shed repositories are categorized. You can browse the repositories in each category by clicking on the name link. You can browse all repositories by clicking the appropriate link in the left menu panel.

tool_shed.png

Although mercurial now provides the framework for the tool shed, a user can take advantage of virtually all of the tool shed's features without knowing anything about mercurial. This document provides information about using the various features of the tool shed. We'll start by creating a new mercurial repository to house a tool we've written. The intent of the tool shed (whether it is a local, proprietary tool shed or the public Galaxy tool shed hosted by Penn State University) is to provide a vehicle for sharing tools (and tool-related objects like workflows, data types, etc) that are determined to be functional within a Galaxy instance. The tools themselves should be implemented within a development environment that includes a Galaxy instance and when a tool is deemed functional, it can then be uploaded to the tool shed for sharing.

You can, however, tweak the primary intent of a tool shed to meet your needs. For example, you may be interested in using a local tool shed instance to share tools between developers during the development process. If this is the case, then your approach can be one where a developer creates a repository on the local tool shed so that multiple developers can clone it. The mercurial command line process for committing, pushing, pulling and updating the tool shed repository can be used to share updates to the tool code by multiple developers throughout the process. If a single developer is implementing the tool, however, it makes less sense to use the tool shed as part of the development process - just upload the tool to the tool shed repository when it is functional.

Tool sheds supported by the core Galaxy development team

The Galaxy Test Tool Shed is available as a sandbox environment allowing you to familiarize yourself with tool shed features. Feel free to mess around here as much as you want.

The Galaxy Main Tool Shed hosts production-ready Galaxy tools and tool components, and should not be used for testing or becoming familiar with tool shed features.

Scripts for inspecting and maintaining the repositories contained within a tool shed

The following scripts are available for inspecting and maintaining the repositories contained within a local tool shed that you are hosting.  You can configure cron to execute these scripts on a regular schedule or execute them manually against your local tool shed whenever you choose.  Each of these scripts is configured to execute regularly against both of the public tool sheds supported by the core Galaxy development team.

  • ~/lib/tool_shed/scripts/deprecate_repositories_without_metadata.py - this script will deprecate any repositories that are older than a specified number of days (set on the command line) and have been empty since creation.  The owner of the repository will receive an email alerting them that their repository has been marked as deprecated.  The owner can mark the repository an not deprecated at any time if they choose to use the repository.

  • ~/lib/tool_shed/scripts/check_repositories_for_functional_tests.py - this script will inspect all installable repository revisions in the tool shed.  For any installable repository revision that includes valid tools, the script inspects each tool configuration file to determine if at least one functional test has been defined for the tool.  If not, the repository is flagged as having invalid functional tests.  If all tools are found to have at least one functional test defined, but the repository does not contain a subdirectory named test-data that contains the test data file, the repository is flagged as having invalid functional tests.  When a repository is flagged as having invalid functional tests, the reason the functional tests are invalid is documented for the repository so that when a user is browsing the repository they will be able to see the information.  Repositories that do not contain any valid tools are not inspected by this script.

  • ~/install_and_test_tool_shed_repositories.sh - this shell script executes the test framework contained in ~/test/install_and_test_tool_shed_repositories.  You can set certain environment variables (all defined within the install_and_test_tool_shed_repositories.sh shell script) to direct the test framework to inspect repositories contained within a specified tool shed and install them into a temporary Galaxy environment created by the test framework.  The test framework will install only those repositories that have been flagged as having tools with valid functional tests by the ~/lib/tool_shed/scripts/check_repositories_for_functional_tests.py script discussed above.  The tool's functional tests will be executed and the results will be uploaded to the repository so that users browsing the repository can see them.

History of the tool shed

The Galaxy tool shed was originally introduced at the Galaxy Community Conference in 2010. At that time, the tool shed allowed uploading and downloading of tool archives (tar files) consisting of single tools or a suite of several tools. Uploading an archive required the contents of the archive to follow strict rules.

The tool shed was completely re-engineered for it's new introduction at the 2011 Galaxy Developer Conference, where it was referred to as the "next-gen tool shed", and the foundation of many of its features became mercurial (hg). Rather than simply storing tool archives as files on disk, hg repositories became the container, allowing for automatic version control among other useful features. In addition to uploading and downloading using a browser, hg pulls and pushes via http became possible.

Related Pages

Overview of tool shed repository features

  • Tool shed repository features - this document describes how to create a new repository in a tool shed.  An overview of the features that are available within the repository after it has been created and populated is included.

The contents of your tool shed repository

  • A single tool or a suite of tools per repository - this document discusses the pros and cons of restricting a repository to a single tool versus opening it up to include an entire suite of tools.

  • Adding change sets to a repository - with each change set committed and pushed to a repository, whether using hg from the command line or by uploading or deleting files using the tool shed UI features, metadata about the contents of the change set is generated.  This document provides the details about this process.

  • Tool shed features for Galaxy tools - the primary intent of the tool shed is for sharing Galaxy tools, workflows and other useful Galaxy utilities.  Galaxy tools are generally developed within a local Galaxy environment, proven to be functionally correct within that environment, and then uploaded to a tool shed for sharing.  With a couple of exceptions, tool features are defined within the Galaxy framework, and have nothing to do with the tool shed.  This document describes those exceptions.

  • Including installation information or 3rd-party tool dependency licensing information in your repository - this document provides the details for a simple feature of the tool shed that enables displaying the license information (or possibly other kinds of information) for tools included in a repository.

  • Tool shed features for Galaxy datatypes - this document provides information for how to include your proprietary Galaxy datatypes (upon which your Galaxy tools depend) in a repository in the tool shed.

  • Defining repository dependencies - this document provides information about how to define dependencies [for a repository] on any number of additional, separate repositories.

  • Including functional tests for your tools - this document provides details for including input and output datasets in your repository for functional tests defined in tool config files included in the repository.

  • Enabling workflow sharing - this document describes how the tool shed enables sharing Galaxy workflows.

Reviewing, rating and approving tool shed repositories

  • Reviewing tool shed repositories - this document discusses the goals for reviewing tool shed repositories, their specific contents, and the various components and features involved in their installation into local Galaxy instances and their use within them.

Installing, maintaining and uninstalling tool shed repositories within a Galaxy instance

Searching the tool shed

  • Tool shed search - this document describes the tool shed search feature, which enables you to find specific tools or workflows that are included in repositories contained in the tool shed.

The tool shed API

Configuring and hosting your own local tool shed

last edited 2013-05-08 12:14:32 by JoachimJacob