Examples of TransferManager

com.amazonaws.services.s3.transfer.TransferManager
can poll your transfer's status to check its progress if (myUpload.isDone() == false) { System.out.println("Transfer: " + myUpload.getDescription()); System.out.println(" - State: " + myUpload.getState()); System.out.println(" - Progress: " + myUpload.getProgress().getBytesTransferred()); } // Transfers also allow you to set a ProgressListener to receive // asynchronous notifications about your transfer's progress. myUpload.addProgressListener(myProgressListener); // Or you can block the current thread and wait for your transfer to // to complete. If the transfer fails, this method will throw an // AmazonClientException or AmazonServiceException detailing the reason. myUpload.waitForCompletion();
Note: Transfers are stored in memory. If the JVM is restarted, previous transfers are no longer accessible. If needed, clean up any multipart uploads that are incomplete.
org.syncany.plugins.transfer.TransferManager
The transfer manager synchronously connects to the remote storage. It is responsible for file upload, download and deletion.
All its operations are strictly synchronous and throw a {@code StorageException} if they fail. The implementations have to make surethat
- the repository is not corrupted, e.g. duplicate files or corrupt files
- files matching the specified file format are complete, i.e. fully uploaded
- methods that need an established connections re-connect if necessary
A transfer manager may organize files according to their type or name as it is optimal for the given storage. {@link RemoteFile}s can be classified by their sub-type. For network-transfer optimization reasons, it might be useful to place {@link MultichunkRemoteFile}s and {@link DatabaseRemoteFile}s in a separate sub-folder on the remote storage. @author Philipp C. Heckel

Examples of com.amazonaws.services.s3.transfer.TransferManager

            Date start = new Date();
            // compute s3 key
            s3Key = join(asList(installPath, fileName), S3Utils.SEPARATOR);


            // multi-part upload using S3 api to handle > 5G input stream
            TransferManager tm = new TransferManager(S3Utils.acquireClient(s3));


            // download using S3 API
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentLength(remoteSize);
            if ( contentType != null ){
                metadata.setContentType(contentType);
            }
            PutObjectRequest putObjectRequest = new PutObjectRequest(s3.getBucketName(), s3Key, in, metadata);
            // check if RRS is enabled
            if (s3.getEnableRRS()){
                putObjectRequest = putObjectRequest.withStorageClass(StorageClass.ReducedRedundancy);
            }
            // register progress listenser
            putObjectRequest.setProgressListener(new ProgressListener() {
                @Override
                public void progressChanged(ProgressEvent progressEvent) {
                    // s_logger.debug(progressEvent.getBytesTransfered()
                    // + " number of byte transferd "
                    // + new Date());
                    totalBytes += progressEvent.getBytesTransfered();
                    if (progressEvent.getEventCode() == ProgressEvent.COMPLETED_EVENT_CODE) {
                        s_logger.info("download completed");
                        status = TemplateDownloader.Status.DOWNLOAD_FINISHED;
                    } else if (progressEvent.getEventCode() == ProgressEvent.FAILED_EVENT_CODE) {
                        status = TemplateDownloader.Status.UNRECOVERABLE_ERROR;
                    } else if (progressEvent.getEventCode() == ProgressEvent.CANCELED_EVENT_CODE) {
                        status = TemplateDownloader.Status.ABORTED;
                    } else {
                        status = TemplateDownloader.Status.IN_PROGRESS;
                    }
                }


            });
            // TransferManager processes all transfers asynchronously,
            // so this call will return immediately.
            Upload upload = tm.upload(putObjectRequest);


            upload.waitForCompletion();


            // finished or aborted
            Date finish = new Date();

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

    return new AmazonS3Client(provider);
  }


  private boolean upload(AmazonS3 s3, File sourceFile) throws MojoExecutionException
  {
    TransferManager mgr = new TransferManager(s3);


    Transfer transfer;
    if (sourceFile.isFile()) {
      transfer = mgr.upload(bucketName, destination, sourceFile);
    } else if (sourceFile.isDirectory()) {
      transfer = mgr.uploadDirectory(bucketName, destination, sourceFile, recursive);
    } else {
      throw new MojoExecutionException("File is neither a regular file nor a directory " + sourceFile);
    }
    try {
      getLog().debug("Transferring " + transfer.getProgress().getTotalBytesToTransfer() + " bytes...");

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

    return new AmazonS3Client(provider);
  }


  private static boolean upload(AmazonS3 s3, String bucketName, String destinationFile, File source)
  {
    TransferManager mgr = new TransferManager(s3);
    Upload upload = mgr.upload(bucketName, destinationFile, source);


    try {
      upload.waitForUploadResult();
    } catch (InterruptedException e) {
      return false;

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

        Properties props = Utils.readConfig(config);
        AmazonS3Client s3service = Utils.openService(props);
        Backend backend = ds.getBackend();
        String bucket = ((S3Backend)backend).getBucket();
        LOG.info("delete bucket [" + bucket + "]");
        TransferManager tmx = new TransferManager(s3service);
        if (s3service.doesBucketExist(bucket)) {
            for (int i = 0; i < 3; i++) {
                tmx.abortMultipartUploads(bucket, startTime);
                ObjectListing prevObjectListing = s3service.listObjects(bucket);
                while (prevObjectListing != null ) {
                    List<DeleteObjectsRequest.KeyVersion> deleteList = new ArrayList<DeleteObjectsRequest.KeyVersion>();
                    for (S3ObjectSummary s3ObjSumm : prevObjectListing.getObjectSummaries()) {
                        deleteList.add(new DeleteObjectsRequest.KeyVersion(s3ObjSumm.getKey()));
                    }
                    if (deleteList.size() > 0) {
                        DeleteObjectsRequest delObjsReq = new DeleteObjectsRequest(bucket);
                        delObjsReq.setKeys(deleteList);
                        s3service.deleteObjects(delObjsReq);
                    }
                    if (!prevObjectListing.isTruncated()) break;
                    prevObjectListing = s3service.listNextBatchOfObjects(prevObjectListing);
                }
            }
            s3service.deleteBucket(bucket);
            LOG.info("bucket: " + bucket + " deleted");
            tmx.shutdownNow();
            s3service.shutdown();
        }
    }

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

            String writeThreadsStr = prop.getProperty(S3Constants.S3_WRITE_THREADS);
            if (writeThreadsStr != null) {
                writeThreads = Integer.parseInt(writeThreadsStr);
            }
            LOG.info("Using thread pool of [{}] threads in S3 transfer manager.", writeThreads);
            tmx = new TransferManager(s3service,
                (ThreadPoolExecutor) Executors.newFixedThreadPool(writeThreads,
                    new NamedThreadFactory("s3-transfer-manager-worker")));
            
            int asyncWritePoolSize = 10;
            String maxConnsStr = prop.getProperty(S3Constants.S3_MAX_CONNS);

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

            String writeThreadsStr = prop.getProperty(S3Constants.S3_WRITE_THREADS);
            if (writeThreadsStr != null) {
                writeThreads = Integer.parseInt(writeThreadsStr);
            }
            LOG.info("Using thread pool of [{}] threads in S3 transfer manager.", writeThreads);
            tmx = new TransferManager(s3service,
                (ThreadPoolExecutor) Executors.newFixedThreadPool(writeThreads,
                    new NamedThreadFactory("s3-transfer-manager-worker")));
            
            int asyncWritePoolSize = 10;
            String maxConnsStr = prop.getProperty(S3Constants.S3_MAX_CONNS);

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

             * not set, invocation first goes us standard region, which
             * redirects it to correct location.
             */
            s3service.setEndpoint(endpoint);
            LOG.info("S3 service endpoint: " + endpoint);
            tmx = new TransferManager(s3service, createDefaultExecutorService());
            LOG.debug("  done");
        } catch (Exception e) {
            LOG.debug("  error ", e);
            throw new DataStoreException("Could not initialize S3 from "
                + config, e);

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

             * not set, invocation first goes us standard region, which
             * redirects it to correct location.
             */
            s3service.setEndpoint(endpoint);
            LOG.info("S3 service endpoint: " + endpoint);
            tmx = new TransferManager(s3service, createDefaultExecutorService());
            LOG.debug("  done");
        } catch (Exception e) {
            LOG.debug("  error ", e);
            throw new DataStoreException("Could not initialize S3 from "
                + config, e);

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

        ClientConfiguration clientConfig = new ClientConfiguration();


        clientConfig.setUserAgent("AwsGradlePlugin");


        s3 = new AmazonS3Client(credentials);
        tm = new TransferManager(s3);
    }

View Full Code Here

Examples of com.amazonaws.services.s3.transfer.TransferManager

            if (writeThreadsStr != null) {
                writeThreads = Integer.parseInt(writeThreadsStr);
            }
            LOG.info("Using thread pool of [" + writeThreads
                + "] threads in S3 transfer manager");
            tmx = new TransferManager(s3service,
                (ThreadPoolExecutor) Executors.newFixedThreadPool(writeThreads,
                    new NamedThreadFactory("s3-transfer-manager-worker")));
            String renameKeyProp = prop.getProperty(S3Constants.S3_RENAME_KEYS);
            boolean renameKeyBool = (renameKeyProp == null || "".equals(renameKeyProp))
                    ? true

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.