Use LazyFileReader for reading file contents

Simple ioutil.ReadFile is used for reading file contents but it reads
all of the file contents and copies them into the memory and is run in a
single goroutine. It causes much memory consumption at copying media
files in content directory to publish directory and it is not good at
performance.

This improves the both issue by replacing ReadFile with LazyFileReader.

It postpones reading the file contents until it is really needed. As the
result, actual file read is run in parallelized goroutine. It improves
performance especially in a really big site.

In addition, if this reader is called from io.Copy, it does not copy the
file contents into the memory but just copies them into destination
file. It improves much memory consumption issue when the site has many
media files.

Fix #1181

diff --git a/source/filesystem.go b/source/filesystem.go
index 7b7ebb6b7fcac35ce00ea605a7107f532b5555f8..05167701b6597c6c1078bb62b08a0bbe2ad93d1b 100644 (file)
--- a/source/filesystem.go
+++ b/source/filesystem.go
@@ -14,10 +14,8 @@
 package source
 
 import (
-       "bytes"
        "github.com/spf13/viper"
        "io"
-       "io/ioutil"
        "os"
        "path/filepath"
        "regexp"
@@ -114,11 +112,11 @@ func (f *Filesystem) captureFiles() {
                if isNonProcessablePath(filePath) {
                        return nil
                }
-               data, err := ioutil.ReadFile(filePath)
+               rd, err := NewLazyFileReader(filePath)
                if err != nil {
                        return err
                }
-               f.add(filePath, bytes.NewBuffer(data))
+               f.add(filePath, rd)
                return nil
        }