Reduce memory usage by avoiding full file reads

[xswordsx]

What does this PR do?

Prior to this change the application would copy all of the binaries in memory, use the first 12 bytes and copy the rest verbatim into the new output file.

How was this tested?

I built the binary from both the current master (7ddd0e4) and this branch, applying a simple print out of the memory allocated for both.

memory print-out patch

index 9ff733a..17b4a2e 100644
--- a/makefat.go
+++ b/makefat.go
@@ -8,6 +8,7 @@ import (
        "fmt"
        "io"
        "os"
+       "runtime"
 )

 const (
@@ -25,6 +26,11 @@ func main() {
                os.Exit(2)
        }

+
+       var ms1, ms2 runtime.MemStats
+       runtime.GC()
+       runtime.ReadMemStats(&ms1)
+
        // Read input files.
        type input struct {
                file   *os.File
@@ -134,6 +140,7 @@ func main() {
        // Write each contained file.
        for _, i := range inputs {
                if offset < i.offset {
+                       fmt.Println("adjusting offset:", i.offset-offset)
                        _, err = out.Write(make([]byte, i.offset-offset))
                        if err != nil {
                                panic(err)
@@ -153,4 +160,9 @@ func main() {
        if err != nil {
                panic(err)
        }
+
+       runtime.ReadMemStats(&ms2)
+       fmt.Println("total:   ", ms2.TotalAlloc - ms1.TotalAlloc, "bytes")
+       fmt.Println("adjusted:", ms2.HeapAlloc - ms1.HeapAlloc, "bytes")
+       fmt.Println("mallocs: ", ms2.Mallocs - ms1.Mallocs)
 }

I then ran the built binaries against a simple "Hello, world!" Go binary built for amd64 and arm64 and fed them to both makefat versions.

Dummy binary

// filename: hello.go
import "fmt"

func main() {
	fmt.Println("Hello, world!")
}

# Building the binaries
GOARCH=amd64 go build -o world_amd64 ./hello.go
GOARCH=arm64 go build -o world_arm64 ./hello.go

$ ls -la world_*
-rwxr-xr-x  1 the_mac  staff  2569248 Apr  6 19:20 world_amd64
-rwxr-xr-x  1 the_mac  staff  2492514 Apr  6 19:09 world_arm64

What are the results of the changes

makefat now doesn't scale its memory usage with binary input size.

makefat (master)

adjusting offset: 16336
adjusting offset: 3040
total:    5093048 bytes
adjusted: 5092512 bytes
mallocs:  28

-----------------------

makefat (new)
adjusting offset: 16336
adjusting offset: 3040
total:    87800 bytes
adjusted: 87800 bytes
mallocs:  31

I've also tried it with a much larger Go binary (~25MB) and the new results are similar

$ ls -la larger_{amd64,arm64}
-rwxr-xr-x@ 1 the_mac  staff  25569440 Apr  6 19:43 larger_amd64
-rwxr-xr-x@ 1 the_mac  staff  24414194 Apr  6 19:43 larger_arm64

$ ./makefat_master testing_both larger_{amd64,arm64}
adjusting offset: 16336
adjusting offset: 5984
total:    50021432 bytes
adjusted: 50021128 bytes
mallocs:  30

$ ./makefat_new testing_both larger_{amd64,arm64}
adjusting offset: 16336
adjusting offset: 5984
total:    91032 bytes
adjusted: 91032 bytes
mallocs:  31

Caveats

• The io.Copy call does not pass in a buffer so Go allocates a new 32kb one on each call