9. Resource handling

The Spring Framework provides a org.springframework.core.io.ResourceLoader abstraction to load files from the filesystem, servlet context and the classpath. Spring Cloud AWS adds support for the Amazon S3 service to load and write resources with the resource loader and the s3 protocol.spring-doc.cn

The resource loader is part of the context module, therefore no additional dependencies are necessary to use the resource handling support.spring-doc.cn

9.1. Configuring the resource loader

Spring Cloud AWS does not modify the default resource loader unless it encounters an explicit configuration with an XML namespace element. The configuration consists of one element for the whole application context that is shown below:spring-doc.cn

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
       xsi:schemaLocation="http://www.springframework.org/schema/cloud/aws/context
       http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd">

    <aws-context:context-credentials>
            ...
        </aws-context:context-credentials>

    <aws-context:context-resource-loader/>
</beans>

9.2. Downloading files

Downloading files can be done by using the s3 protocol to reference Amazon S3 buckets and objects inside their bucket. The typical pattern is s3://<bucket>/<object> where bucket is the global and unique bucket name and object is a valid object name inside the bucket. The object name can be a file in the root folder of a bucket or a nested file within a directory inside a bucket.spring-doc.cn

The next example demonstrates the use of the resource loader to load different resources.spring-doc.cn

public class SimpleResourceLoadingBean {

    @Autowired
    private ResourceLoader resourceLoader;

    public void resourceLoadingMethod() throws IOException {
        Resource resource = this.resourceLoader.getResource("s3://myBucket/rootFile.log");
        Resource secondResource = this.resourceLoader.getResource("s3://myBucket/rootFolder/subFile");

        InputStream inputStream = resource.getInputStream();
        //read file
    }
}

9.3. Uploading files

Since Spring Framework 3.1 the resource loader can also be used to upload files with the org.springframework.core.io.WritableResource interface which is a specialization of the org.springframework.core.io.ResourceLoader interface. Clients can upload files using the WritableResource interface. The next example demonstrates an upload of a resource using the resource loader.spring-doc.cn

public class SimpleResourceLoadingBean {

    @Autowired
    private ResourceLoader resourceLoader;

    public void writeResource() throws IOException {
        Resource resource = this.resourceLoader.getResource("s3://myBucket/rootFile.log");
        WritableResource writableResource = (WritableResource) resource;
        try (OutputStream outputStream = writableResource.getOutputStream()) {
            outputStream.write("test".getBytes());
        }
    }
}

9.3.1. Uploading multi-part files

Amazon S3 supports multi-part uploads to increase the general throughput while uploading. Spring Cloud AWS by default only uses one thread to upload the files and therefore does not provide parallel upload support. Users can configure a custom org.springframework.core.task.TaskExecutor for the resource loader. The resource loader will queue multiple threads at the same time to use parallel multi-part uploads.spring-doc.cn

The configuration for a resource loader that uploads with 10 Threads looks like the followingspring-doc.cn

<beans ...>
  <aws-context:context-resource-loader task-executor="executor" />
  <task:executor id="executor" pool-size="10" queue-capacity="0" rejection-policy="CALLER_RUNS" />
</beans>

Spring Cloud AWS consumes up to 5 MB (at a minimum) of memory per thread. Therefore each parallel thread will incur a memory footprint of 5 MB in the heap, and a thread size of 10 will consume therefore up to 50 mb of heap space. Spring Cloud AWS releases the memory as soon as possible. Also, the example above shows that there is no queue-capacity configured, because queued requests would also consume memory.spring-doc.cn

9.3.2. Uploading with the TransferManager

The Amazon SDK also provides a high-level abstraction that is useful to upload files, also with multiple threads using the multi-part functionality. A com.amazonaws.services.s3.transfer.TransferManager can be easily created in the application code and injected with the pre-configured com.amazonaws.services.s3.AmazonS3 client that is already created with the Spring Cloud AWS resource loader configuration.spring-doc.cn

This example shows the use of the transferManager within an application to upload files from the hard-drive.spring-doc.cn

public class SimpleResourceLoadingBean {

    @Autowired
    private AmazonS3 amazonS3;

    public void withTransferManager() {
        TransferManager transferManager = new TransferManager(this.amazonS3);
        transferManager.upload("myBucket","filename",new File("someFile"));
    }
}

9.4. Searching resources

The Spring resource loader also supports collecting resources based on an Ant-style path specification. Spring Cloud AWS offers the same support to resolve resources within a bucket and even throughout buckets. The actual resource loader needs to be wrapped with the Spring Cloud AWS one in order to search for s3 buckets, in case of non s3 bucket the resource loader will fall back to the original one. The next example shows the resource resolution by using different patterns.spring-doc.cn

public class SimpleResourceLoadingBean {

    private ResourcePatternResolver resourcePatternResolver;

    @Autowired
    public void setupResolver(ApplicationContext applicationContext, AmazonS3 amazonS3){
        this.resourcePatternResolver = new PathMatchingSimpleStorageResourcePatternResolver(amazonS3, applicationContext);
    }

    public void resolveAndLoad() throws IOException {
        Resource[] allTxtFilesInFolder =  this.resourcePatternResolver.getResources("s3://bucket/name/*.txt");
        Resource[] allTxtFilesInBucket =  this.resourcePatternResolver.getResources("s3://bucket/**/*.txt");
        Resource[] allTxtFilesGlobally =  this.resourcePatternResolver.getResources("s3://**/*.txt");
    }
}

Resolving resources throughout all buckets can be very time consuming depending on the number of buckets a user owns.spring-doc.cn

9.5. Using CloudFormation

CloudFormation also allows to create buckets during stack creation. These buckets will typically have a generated name that must be used as the bucket name. In order to allow application developers to define static names inside their configuration, Spring Cloud AWS provides support to resolve the generated bucket names. Application developers can use the org.springframework.cloud.aws.core.env.ResourceIdResolver interface to resolve the physical names that are generated based on the logical names.spring-doc.cn

The next example shows a bucket definition inside a CloudFormation stack template. The bucket will be created with a name like integrationteststack-sampleBucket-23qysofs62tc2spring-doc.cn

{
    "Resources": {
        "sampleBucket": {
            "Type": "AWS::S3::Bucket"
        }
    }
}

Application developers can resolve that name and use it to load resources as shown in the next example below.spring-doc.cn

public class SimpleResourceLoadingBean {

    private final ResourceLoader loader;
    private final ResourceIdResolver idResolver;

    @Autowired
    public SimpleResourceLoadingBean(ResourceLoader loader, ResourceIdResolver idResolver) {
        this.loader = loader;
        this.idResolver = idResolver;
    }

    public void resolveAndLoad() {
        String sampleBucketName = this.idResolver.
            resolveToPhysicalResourceId("sampleBucket");
        Resource resource = this.loader.
            getResource("s3://" + sampleBucketName + "/test");
    }
}