Table of Contents

guest
2022-09-24
       File Administrator Guide
         Files Web Part Administration
         File Root Options
           Troubleshoot Pipeline and Files
         File Terminology
         Transfer Files with WebDAV

File Administrator Guide


This section includes guides and background information useful to administrators working with files in LabKey Server.

File Administration Topics

Related Topics




Files Web Part Administration


Administrators can customize the Files web part in the following ways, allowing them to present the most useful set of features to their file browser users. To customize the Files web part, click the Admin button on the toolbar.

There are four tabs for the various customizations possible here:

Customize Actions Available

The Actions tab lets you control the availability of common file action and data import options such as importing datasets, creating assay designs, etc.

Note that file actions are only available for files in the pipeline directory. If a pipeline override has been set, or the file browser is showing non-file content (such as in a "@fileset" directory), these actions will not be available in the default file location.

You can change what a Files web part displays using the (triangle) menu; choose Customize. Select a file content directory to enable actions on this tab.

The Import Data button is always visible to admins, but you can choose whether to show it to non-admin users. For each other action listed, checking the Enabled box makes it available as a pipeline job. Checking Show in Toolbar adds a button to the Files web part toolbar.

Define File Tagging Properties

The File Properties tab lets you define properties that can be used to tag files. Each file browser can use the default (none), inherit properties from the parent container, or use custom file properties.

Once a custom property is defined, users are asked to provided property values when files are uploaded. There is a built in 'Comment' file property that is included when other custom properties are in use.

  • To define a property, select Use Custom File Properties and then click the Edit Properties button.
  • If no properties have been defined, you have the option to import field definitions from a prepared JSON file. Otherwise, click Manually Define Fields.
  • Click Add Field to add new properties. Details about customizing fields are available in this topic: Field Editor.
  • To reuse properties defined in the parent folder, select Use Same Settings as Parent.

Tagged files can be retrieved by searching on their property value. For more detail, see Step 3: Search the Repository.

Control Toolbar Buttons and Grid Settings

The Toolbar and Grid Settings tab controls the appearance of the file management browser.

Configure Toolbar Options: Toolbar buttons are in display order, from top to bottom; drag and drop to rearrange. Available buttons which are not currently displayed are listed at the end. Check boxes to show and hide text and icons independently.

Configure Grid Columns Settings (scroll down): lists grid columns in display order from top to bottom; drag and drop to rearrange. The columns can be reorganized by clicking and dragging their respective rows, and checkboxes can make columns Hidden or Sortable.

You can also change which columns are displayed directly in the Files web part. Pulldown the arrow in any column label, select Columns and use checkboxes to show and hide columns. For example, this screenshot shows adding the Download Link column:

Local Path to Files

Use the following procedure to find the local path to files that have been uploaded via the Files web part.

  • Go to (Admin) > Site > Admin Console.
  • Under Configuration, click Files.
  • Under Summary View for File Directories, locate and open the folder where your Files web part is located.
  • The local path to your files appears in the Directory column. It should end in @files.

General Settings Tab

Use the checkbox to control whether to show the file upload panel by default. The file upload panel offers a single file browse and upload interface and can be opened with the Upload Files button regardless of whether it is shown by default.

You may drag and drop files into the file browser region to upload them whether or not the panel is shown.

Note that in Premium Editions of LabKey Server, file upload can be completely disabled if desired. See below for details.

Configure Default Email Alerts

To control the default email notification for the folder where the Files web part resides, see Manage Email Notifications. Email notifications can be sent for file uploads, deletions, changes to metadata properties, etc. Admins can configure each project user to override or accept the folder's default notification setting at the folder level.

Project users can also override default notification settings themselves by clicking the (Email Preferences) button in the Files web part toolbar. SElect your own preference and click Submit.

Disable File Upload

Premium Feature — This feature is available in Premium Editions of LabKey Server. Learn more or contact LabKey.

Site Administrators can disable file upload across the entire site. When activated, file upload to both the file and pipeline roots will be disabled.

When file upload is disabled, the Upload button in the File Repository is hidden. Also, dragging and dropping files does not trigger file upload. Users (with sufficient permissions) can still perform the following actions in the File Repository, including: download, rename, delete, move, edit properties, and create folder.

Note that attaching files to issues, wikis, or messages is not disabled by this setting.

Changes to this setting are logged under the Site Settings Events log.

To disable file upload site-wide:

  • Go to (Admin) > Site > Admin Console.
  • Under Configuration, click Files.
  • Place a checkmark next to Disable file upload.
  • Click Save

Related Topics


Premium Resource Available

Subscribers to premium editions of LabKey Server can configure scanning of uploaded files for viruses. Learn more in this topic:


Learn more about premium editions




File Root Options


LabKey Server provides tools for securely uploading, processing and sharing your files. If you wish, you can override default storage locations for each project and associated subfolders by setting site-level or project-level file roots.

Topics

Summary View of File Roots and Overrides

You can view an overview of settings and full paths from the "Summary View for File Directories" section of the "Configure File System Access" page that is available through (Admin) > Site > Admin Console > Configuration > Files.

File directories, named file sets and pipeline directories can be viewed on a project/folder basis through the "Summary View." The 'Default' column indicates whether the directory is derived from the site-level file root or has been overridden. To view or manage files in a directory, double click on a row or click on the 'Browse Selected' button. To configure an @file or an @pipeline directory, select the directory and click on the 'Configure Selected' button in the toolbar.

If you add a pipeline override for any folder, we don't create a @files folder in the filesystem. The server treats an override as a user-managed location and will use the path specified instead.

Note that a @pipeline marker is used in the "Summary View for File Directories", available through (Admin) > Site > Admin Console > Configuration > Files. However, there is no corresponding @pipeline directory on the file system. The summary view uses the @pipeline marker simply to show the path for the associated pipeline.

Site-Level File Root

The site-level file root is the top of the directory structure for files you upload. By default it is under the LabKey Server installation directory, but you may choose to place it elsewhere if required for backup, permissions, or disk space reasons.

During server setup, a directory structure is created mirroring the structure of your LabKey Server projects and folders. Each project or folder is a directory containing a "@files" subdirectory. Unless the site-level root has been overridden at the project or folder level, files will be stored under the site-level root.

You can specify a site-level file root at installation or access the "Configure File System Access" page on an existing installation.

  • Select (Admin) > Site > Admin Console.
  • Under Configuration, click Files.

Change the Site-Level File Root

When you change the site-level file root for an existing installation, files in projects that use fileroots based on that site-level file root will be automatically moved to the new location. The server will also update paths in the database for all of the core tables. If you are storing file paths in tables managed by custom modules, the custom module will need register an instance of org.labkey.api.files.FileListener with org.labkey.api.files.FileContentService.addFileListener(), and fix up the paths stored in the database within its fileMoved() method.

Files located in projects that use pipeline overrides or in folders with their own project- or folder-level file roots will not be moved by changing the site-level file root. If you have set project-level roots or pipeline overrides, files in these projects and their subfolders must be moved separately. Please see Troubleshoot Pipeline and Files for more information.

Changes to file roots are audited under Project and Folder events.

Project-level File Roots

You can override the site-level root on a project-by-project basis. A few reasons you might wish to do so:

  • Separate file storage for a project from your LabKey Server. You might wish to enable more frequent backup of files for a particular project.
  • Hide files. You can hide files previously uploaded to a project or its subfolders by selecting the "Disable File Sharing" option for the project.
  • Provide a window into an external drive. You can set a project-level root to a location on a drive external to your LabKey Server.
From your project:
  • Select (Admin) > Folder > Project Settings.
  • Click the Files tab.

Changes to file roots are audited under Project and Folder events.

Folder-level File Roots

The default file root for a folder is a subfolder of the project file root plus the folder name. If the project-level root changes, this folder-level default will also change automatically to be under the new project-level root.

To set a custom file root for a single folder, follow these steps:

From your folder:
  • Select (Admin) > Folder > Management.
  • Click the Files tab.

Changes to file roots are audited under Project and Folder events.

Migrate Existing Files

When you change the site-level file root for an existing installation, the entire directory tree of files located under that site-level file root are automatically moved to the new location (and deleted in the previous location).

When you select a new project-level (or folder-level) file root, you will see the option "Proposed File Root change from '<prior option>'." Select what you want to happen to any existing files in the previous root location. The entire directory tree, including any subfolders within the file root are included in any copy or move.

Options are:

  • Not copied or moved: Default
  • Copied to the new location: The existing files stay where they are. A copy of the files is placed in the new file root and database paths are updated to point to the new location.
  • Moved to the new location: Files are copied to the new location, database paths are updated to point to this location, then the original files are deleted.
Note: All work to copy or move the entire directory tree of files is performed in a single pipeline job. The job will continue even if some files fail to copy, but if any file copy fails, no deletion will occur from the original location (i.e. a move will not be completed).

File Root Options

The directory exposed by the Files web part can be set to any of the following directories:

  • @files (the default)
  • @pipeline
  • @filesets
  • @cloud
  • any children of the above
Administrators can select which directory is exposed by clicking the (triangle) on the Files web part and selecting Customize. In the File Root pane, select the directory to be exposed and click Submit. The image below shows how to expose the sub-directory Folder A.

Alternative _webfiles Root

Administrators can enable an alternative WebDAV root for the whole server. This alternative webdav root, named "_webfiles", displays a simplified, file-sharing oriented tree that omits non-file content (like wikis), and collapses @file nodes into the root of the container’s node.

To access or mount this root go to a URL like the following (replacing my.labkeyserver.com with your real server domains):

This URL will expose the server's built-in WebDAV UI. 3rd party WebDAV clients can mount the above URL just like they can mount the default _webdav root.

To enable this alternative webdav root:

  • Select (Admin) > Site > Admin Console.
  • Under Configuration, click Files.
  • Under Alternative Webfiles Root, place a checkmark next to Enable _webfiles.
  • Click Save.

The _webfiles directory is parallel to the default _webdav directory, but only lists the contents under @files and its child containers. @pipeline, @filesets, and @cloud contents are not accessible from _webfiles.

Any name collisions between between containers and file system directories will be handled as follows:

Child containers that share names (case-insensitive) with file system directories will take precedence and be displayed with their names unchanged in the WebDAV tree. File system directories will be exposed with a " (files)" suffix. If there are further conflicts, we will append "2", "3", etc, until we find an unused name. Creating a subdirectory via WebDav always create a child file directory, never a child container.

Map Network Drive (Windows Only)

LabKey Server runs on a Windows server as an operating system service, which Windows treats as a separate user account. The user account that represents the service may not automatically have permissions to access a network share that the logged-in user does have access to. If you are running on Windows and using LabKey Server to access files on a remote server, for example via the LabKey Server pipeline, you'll need to configure the server to map the network drive for the service's user account.

Configuring the network drive settings is optional; you only need to do it if you are running Windows and using a shared network drive to store files that LabKey Server will access.

  • Select (Admin) > Site > Admin Console.
  • Under Configuration, click Files.
  • Under Map Network Drive, click Configure.

Drive letter: The drive letter to which you want to assign the network drive.

Path: The path to the remote server to be mapped using a UNC path -- for example, a value like "\\remoteserver\labkeyshare".

User: Provide a valid user name for logging onto the share; you can specify the value "none" if no user name or password is required.

Password: Provide the password for the user name; you can specify the value "none" if no user name or password is required.

Named File Sets

Named file sets are additional file stores for a LabKey web folder. They exist alongside the default file root for a web folder, enabling web sharing of files in directories that do not correspond exactly to LabKey containers. You can add multiple named file sets for a given LabKey web folder, displaying each in its own web part. The server considers named file sets as "non-managed" file systems, so moving either the site or the folder file root does not have any effect on named file sets. File sets are a single directory and do not include any subdirectories.

To add a named file root:

  • On the Files web part, click the (triangle) and select Customize.
  • On the Customize Files page, click Configure File Roots.
  • Under File Sets, enter a Name and a Path to the file directory on your local machine.
  • Click Add File Set.
  • Add additional file sets are required.
  • To display a named file set in the Files web part, click the (triangle) on the Files web part, and select Customize.
  • Open the "@filesets" node, select your named file set, and click Submit.
  • The Files web part will now display the files in your named file set.

For details on URL parameters used with named file sets, see Controlling File Display via the URL.

For an example showing how to display a named file set using the JavaScript API, see JavaScript API - Examples.

Related Topics




Troubleshoot Pipeline and Files


This topic includes some common troubleshooting steps to take when you encounter issues with uploading and importing data via the pipeline or file browser.

Troubleshooting Basics

When trying to identify what went wrong on your LabKey Server, a few general resources can help you:

  1. Check the error log.
  2. Review troubleshooting documentation.
  3. Check the labkey log.
  4. Review the audit log for actions taking place when you encountered the problem.

Pipeline Import Error Log

When you import files or archives using drag and drop or other pipeline import methods, you will see the Data Pipeline import page. Status messages will show progress. Upon completion you may either encounter an error or notice expected information is missing.

If there is an Error during import, the Status column will generally read "ERROR", though even if it does not, you may want to check these logs. There may be helpful details in the Info column. Click the value in the Status column for full details about the pipeline job.

The Folder import page provides a Job Status panel, followed by a log file panel giving a summary of the full log. Scan the panel for details about what went wrong, typically marked by the word Error or Warning and possibly highlighted in red. If this information does not help you identify the problem, click Show Full Log File for the full file. You can also click links to View or Download the log file in the Job Status web part.

XAR Import Errors

The log file is the first place to look if import of a XAR (xar.xml) file fails.

Notes specific to XAR files:

  • The most common problem is a duplicate LSID problem. In example 1 of the XAR Tutorial, the LSIDs have fixed values. This means that this xar.xml can only be imported in one folder on the whole site. If you are sharing access to a LabKey Server system with other user of that tutorial you will encounter this problem. Subsequent examples in the tutorial will show you how to address this conflict.
  • A second common problem is clashing LSID objects at the run level. If an object is created by a particular ProtocolApplication and then a second ProtApp tries to output an object with the same LSID, an error will result.
  • LabKey Server does not offer the ability to delete protocols or starting inputs in a folder, except for deleting the entire folder. This means that if you import a xar.xml in a folder and then change a protocol or starting input without changing its LSID, you won't see your changes. The XarReader checks first to see if the protocols in a xar.xml have already been defined, and if so will silently use the existing protocols rather than the (possibly changed) protocol descriptions in the xar.xml. See example 3 in the XAR Tutorial for a suggestion of how to avoid problems with this.
  • Sometimes a xar.xml will appear to import correctly but report an error when you try to view the summary graph. This is likely related to problems in referencing the Starting Inputs.

Files Not Visible

If you do not see the expected set of files in a Files web part, check the following. Some of these options are inherited by subfolders, so it may not be immediately clear that they apply to your context.

You can check for unexpected settings at each level as follows:
  • Files Web Part: Click the (triangle) menu in the web part corner and select Customize. See if the expected file root is being displayed.
  • Folder: (Admin) > Folder > Management > Files tab
  • Project: (Admin) > Folder > Project Settings > Files
  • Site: (Admin) > Site > Admin Console > Configuration > Files.

Import Data Button Not Available

If you are using a pipeline override for a folder that differs from the file directory, you will not see an Import Data button in the Files web part. You may either to change project settings to use the default site-level file root or you can import files via the Data Pipeline instead of the Files web part. To access the pipeline UI, go to: (Admin) > Go to Module > Pipeline.

Project-level Files Not Moved When Site-wide Root is Changed

When the site-wide root changes for an existing installation, files in projects that use fileroots based on that site-level file root will be automatically moved to the new location. The server will also update paths in the database for all of the core tables. If you are storing file paths in tables managed by custom modules, the custom module will need register an instance of org.labkey.api.files.FileListener with org.labkey.api.files.FileContenService.addFileListener(), and fix up the paths stored in the database within its fileMoved() method.

Files located in projects that use pipeline overrides or in folders with their own project- or folder-level file roots will not be moved by changing the site-level file root. If you have set project-level roots or pipeline overrides, files in these projects and their subfolders must be moved separately.

User Cannot Upload Files When Pipeline Override Set

In general, a user who has the Editor or Author role for a folder should be able to upload files. This is true for the default file management tool location, or file attachments for issues, wikis, messages, etc.

The exception is when you have configured a folder (or its parent folders) to use a pipeline override. In that case, you will need to explicitly assign permissions for the pipeline override directory.

To determine whether a pipeline override is set up, and to configure permissions if so, follow these steps:

  • Navigate to the folder in question.
  • Select (Admin) > Go to Module > Pipeline.
  • Click Setup.
  • If the "Set a pipeline override" option is selected, you have two choices:
    • Keep the override and use the choices under the Pipeline Files Permissions heading to set permissions for the appropriate users.
    • Remove the override and use normal permissions for the folder. Select "Use a default based on the site-level root" instead of a pipeline override.
  • Adjust folder permissions if needed using (Admin) > Folder > Permissions.

For further information, see Set a Pipeline Override.

Related Topics




File Terminology


This topic defines commonly used terminology for files in LabKey Server.

Root vs. Directory. A root generally implies some sort of inheritance throughout a tree of directories. A directory identifies just one spot in the file system.

LabKey installation directory. The default directory in the file system for LabKey Server that contains folders for files, modules, the webapp, etc. Sometimes referred to as [LABKEY_HOME]. (Example: /data/labkey or C:\labkey\)

Site-level file root. The directory in the LabKey Server's file system that contains your server's file directory structure (Example: /data/labkey/files). It can be set, typically at install time. This location is called a root, not just a directory, because it determines where the tree of file directories lives, not just the location of a single directory. The structure reflects your server's tree of projects and folders. See: File Root Options.

File directory. The specific location on the file system where files associated with a particular project or folder are placed. (Example: /data/labkey/files/project1/folder1/@files, where the folder of interest is folder1, a child of project1.) See: File Root Options.

File root. The directory in LabKey Server's file system that contains your project's file directory structure. (Example: /data/labkey/otherdata/project1/) This structure contains file directories in subfolders that match the structure of your project. See: File Root Options.

File root override. A custom destination for files for a given project. Can be set at the project level only. (Example: /data/labkey/otherdata/project1/myfiles). If a file root override is set, this root determines the the location of the tree of file directories for that project's subfolders. See: File Root Options.

Data processing pipeline. Provides a set of actions that can be performed against a given file. See: Data Processing Pipeline.

Pipeline override. An explicitly set location for files that are subject to actions. Also determines the location where files are uploaded when uploaded via the pipeline UI. Allows security to be set separately vs. the default, folder-specific permissions. See Set a Pipeline Override.




Transfer Files with WebDAV


Use a WebDAV client as an alternative to the native LabKey Server interfaces for accessing files on LabKey Server. WebDAV allows you to read, modify and delete files on the server. You can use either a 3rd party WebDAV client, such as Cyberduck, or, once properly configured, you can use Windows Explorer or OSX without installing any new software. You can also enable a site-wide WebDAV root that provides a more user-friendly user interface: see File Root Options.

Example Setup for Cyberduck WebDAV Client

To set up Cyberduck to access a file repository on LabKey Server, follow these instructions:

  • First, get the WebDAV URL for the target repository:
    • On LabKey Server, go to the target file repository.
    • Click the title of the Files web part.
    • The URL used by WebDAV appears at the bottom of the screen.
  • Alternatively, administrators can get the WebDAV URL directly from the Files web part as follows:
    • Open the Upload Files panel, then click the (file upload help) icon.
    • The File Upload Help dialog appears.
    • The URL used by WebDAV appears in this dialog. Copy the URL to the clipboard.
  • Set up Cyberduck (or another 3rd party WebDAV client).
    • Click Open Connection (or equivalent in another client).
    • Enter the URL and your username/password.
    • Click Connect.
    • You can now drag-and-drop clients into the file repository using the 3rd party WebDAV client.

Tested 3rd Party clients

  • CyberDuck: GUI WebDAV client.
  • WebDrive: Integrates with Explorer and allows you to mount the LabKey Server to a drive letter.
  • NetDrive: Integrates with Explorer and allows you to mount the LabKey Server to a drive letter.
  • cadaver: Command line tool. Similar to FTP

Native Windows WebDAV Client (WebDAV Redirector)

A WebDAV client called "WebDAV Redirector" is built into Windows 8 and Windows 10. Assuming your server is configured to use SSL, you can connect from Windows directly to a LabKey Server file repository. Configuring the WebDAV Redirector to work over non-SSL connections is not recommended.

Note that the WebDAV Redirector is limited to 50MB by default.

To connect, you can use the Window Explorer to map a network drive to the file repository URL, using the URL shown below.

To connect using a Windows Command prompt, use "net use". For example:

net use Y: https://hosted.labkey.com/labkey/_webdav/myProject/myFolder/@files/ /USER:johndoe@labkey.com * /PERSISTENT:YES

Explanation of the command above:

Command line itemDescription
Y:The drive letter that will allow the client to copy multiple files to the LabKey Server using familiar Windows commands . It can’t be in use at the time; if it is, either choose a different drive letter or issue a net use Y: /D command first to disconnect the Y: drive.
https://hosted.labkey.com/labkey/_webdav/myProject/myFolder/@files/The URL to the WebDAV root. Use double quotes if there are spaces in the URL. (To get this URL, see the screen shot above.)
_webdavThis component of the URL applies to all WebDAV connections into LabKey Server.
myProjectThe LabKey Server project name.
myFolderThe folder name within the project - the location of the Files web part.
@filesThe directory root for the file content. This folder is viewed by the Files web part in a LabKey Server folder. Files managed by the pipeline component appear under a root directory called @pipeline.
johndoe@labkey.comthe same user email you would use to sign into LabKey Server from a browser.
*Causes Windows to prompt for your LabKey password.
/PERSISTENT:YESCauses Windows to remember the drive letter mapping between restarts of the system.

Once you’ve mapped a drive letter to LabKey Server, you can use COPY, REN, XCOPY and other standard Windows command to move data files between the client and LabKey Server.

The mapped network drive feature is accessible in the Windows File Explorer There is a button for "Map Network Drive" above the files/folders list. ( On Windows 8, make sure the "This PC" node in the left panel Windows Explorer has selected the "This PC" node in the left hand pane. )

You can now use Windows Explorer to drag-and-drop files into the @files directory on the server.

Native OSX WebDAV Client

When using OSX, you do not need to install a 3rd party WebDAV client. You can mount a WebDAV share via the dialog at Go > Connect to Server. Enter a URL of the form:

https://<username%40domain.com>@<www.sitename.org>/_webdav/<projectname>/

To have the URL generated for you, see the instructions above for Cyberduck.

  • <username%40domain.com> - The email address you use to log in to your LabKey Server, with the @ symbol replaced with %40. Example: Use username%40labkey.com for username@labkey.com
  • <www.sitename.org> - The URL of your LabKey Server. Example: www.mysite.org
  • <projectname> - The name of the project you would like to access. If you need to force a login, this project should provide no guest access. Example: _secret

Linux WebDAV Clients

Tested clients:

  • Gnome Desktop: Nautilus file browser can mount a WebDAV share like an NFS share.
  • KDE Desktop: Has drop down for mounting a WebDAV share like an NFS share.
  • cadaver: Command line tool. Similar to FTP.

Related Topics