Send data files to ArrayExpress by FTP/Aspera

ArrayExpress provides a FTP repository for submitters to upload a large volume of data files associated with their experiment submissions.

Before you start transferring files, please note the following points:

  1. File preparation: Please check our guidelines on submitting files for a microarray experiment, sequencing experiment or array design. E.g. microarray data files should not be compressed, but fastq files from sequencing experiments must be individually compressed by gzip or bzip2.
  2. Data privacy: All ArrayExpress submitters use the same account to upload files (see below). As most of the files are un-/pre-published, we make all uploaded files "private", which means you will not be able to see any files or directories already on the FTP site, including those that you uploaded or created. You will only be able to see your files if you are uploading to your personal Annotare submission directory. The FTP server is a temporary storage space, where we will keep your files for two months, counting from the date of file upload, after which we will delete them without warning. This is to comply with the fair usage policy of the disc space which we share with the European Nucleotide Archive.
  3. No whitespaces or special characters in file names: Make sure the file names are constructed only from alphanumerals [A-Z,a-z,0-9], underscores [_] and dots [.], with no whitespaces, brackets, other punctuations or symbols.
  4. Finish your experiment submission (if applicable): Data files uploaded to ArrayExpress by FTP are only part of an experiment submission. The submission is only complete when experimental metadata (e.g. sample information and protocols) are also submitted to ArrayExpress using Annotare.
  5. Email us (for non-Annotare submitters): When the transfer is complete, please email us at annotare@ebi.ac.uk with the names of files transferred and a short note on what the files are for.

1. FTP upload instruction for Windows Explorer users
2. FTP upload instruction for terminal/command prompt users
3. Aspera upload instruction

 

1. Transferring files via FTP using Windows Explorer (drag and drop)

Make sure you are connected to the internet via a physical connection and not on a wireless connection.

Open a Windows Explorer window (and not an internet browser window as the latter is not designed for FTP file upload). If you need help how to open an Explorer window, here are instructions for Windows XP/Vista/7, Windows 8 and Windows 10.

If you are uploading files for an Annotare submission, you will have get the FTP address from your Annotare "Upload and assign data files" > "FTP upload..." window. This FTP address contains the path to your personal subdirectory. Please manually copy this FTP path from the Annotare dialogue using right-click "Copy link" and paste it into the Windows Explorer address bar. It should look similar to this: ftp://annotare:annotare1@ftp-private.ebi.ac.uk/prod/ibtd1rmo-20r7k3g747sup/. For all other submissions (e.g. array design), enter ftp://annotare:annotare1@ftp-private.ebi.ac.uk/prod/ in Windows Explorer address bar. The URLs already contain the username annotare and password annotare1, so you're logged in automatically after being connected to the FTP server.

If there are no error messages from Windows, that means connection is successful. (Unfortunately, Windows does not provide notification confirming successful connection to an FTP server.) When connected, you will see an almost "blank" Windows Explorer window which is similar to this:

Windows Explorer FTP login image

You can start dragging files from your computer's file system and dropping them onto the logged-on Windows Explorer window.

 

2. Transferring files via FTP using Mac/Unix/Windows terminal (command prompt)

Make sure you are connected to the internet via a physical connection and not on a wireless connection.

1. Open a Terminal window and change to the directory containing the data files.
2. Type: ftp -p ftp-private.ebi.ac.uk to connect to the FTP server.
3. Enter username annotare and password annotare1.
4. (Annotare submissions only) Change into the submission's subdirectory. You will find the path in Annotare by clicking on "FTP upload...". It should look similar to this: /prod/ibtd1rmo-20r7k3g747sup.
In the terminal window type cd followed by the path name.
5. Use the put command to place one file (or mput for multiple files) into the FTP directory.
To exit FTP, type quit. On exiting you will get a message printed to screen to tell you whether your transfer was successful.

Terminal session for FTP upload

Note: If you are using Mac, the file transfer may fail due to incompatible FTP settings of the standard FTP client on Mac (which is embedded in Finder). In this case you should be able to upload your files using an alternative FTP client, e.g. CyberDuck. If you have CyberDuck installed, you will be able to transfer files for your Annotare submission by clicking on the link with the FTP address in Annotare.

 

3. Transferring files using Aspera command line client

Aspera is a commercial file transfer protocol that provides better transfer speeds than FTP over long distances.

  1. Download the Aspera Transfer SDK from the following link: Aspera Transfer SDK - IBM API Hub - IBM Developer (choose the required platform)
  2. Unzip/untar into a suitable directory, for example, C:\aspera on Windows or /opt/aspera on Mac or Linux
  3. You can ignore most of the directories that the SDK writes to your machine. Please navigate to the ./aspera/bin directory. This is the location where you will find the ascp binary. This directory is the one you will use to initiate Aspera transfers.
  4. You can now simply execute ascp to start your transfer with the EMBL-EBI server.
    ascp -P 33001 -c aes128gcm <yourfile.txt> annotare@fasp.ebi.ac.uk:<your upload directory>/
  5. When prompted for a password enter annotare1.
  6. (Annotare submissions only) You can find your upload directory from Annotare submission page, by clicking on "FTP/Aspera upload" button. It should look similar to this: prod/abcd1efg-hijk2lmno3pqr.
    Proceed to import the uploaded files to your submission, as described here.
  7. For more information on how to use Aspera to transfer files, please follow this article: How to transfer files to and from EMBL-EBI using IBM Aspera