Large file upload optimization solution

I have done an AI medical image segmentation project before, and the technology stack uses vue3+Django. Medical imaging files are relatively large, ranging from 20M to 50M. The solution at that time directly used form uploading, without considering optimization at all. If the upload failed, it was manually retransmitted. Now I know that there are some common optimization solutions in the industry. After all, file upload is a relatively common scenario.

The most direct and simple file upload

Upload files directly through the form and convert the files into byte streams, which is also the solution I used in my previous projects. When the file is slightly larger, the efficiency is low and the upload is easy to fail.

<el-upload


  ref="uploadRef"


  drag


  class="upload-demo"


  action="api/upload/"


  :limit="100"


  :on-exceed="handleExceed"


  :auto-upload="false"


  multiple = "true"


  accept = ".dcm, .nii, .nii.gz, .raw"


>


  <el-icon class="el-icon--upload"> <upload-filled /></el-icon>


  <div class="el-upload__text">


    <em>Click or drag the file</em>


  </div>





  <template #tip>


    <div class="el-upload__tip text-red">


      To reduce server load, a maximum of 100 files can be uploaded at one time


    </div>


  </template>


</el-upload>

Multipart upload

Split a large file into several small files (called Parts), upload them one after another, and rejoin the original files on the server. It is suitable for uploading large files, and can also achieve breakpoint resume upload on this basis.

Implementation process

1. The front end calculates MD5 value for large files

2. The front end cuts the large file into N small files, and assigns a serial number to each small file.

3. The backend accepts each uploaded small file and puts it in the cache directory

4. If the front-end fails to upload a certain small file, it sends a failure request to the back-end and asks the back-end to delete the small file.

5. If the front-end sends a merge request to the back-end after sending all the small files, it means that all the small files have been uploaded.

6. The backend reads all small files and merges them in sequence number.

7. The back-end calculates the MD5 value of the merged large file and compares it with the front-end MD5 value to prevent tampering during the transmission process.

Asynchronously upload files to control the number of concurrencies

Upload slices. If there are too many slices, too many asynchronous requests will be initiated at the same time. If too many TCP connections are applied for at the same time, the browser will also cause lag, so the number of concurrent asynchronous requests needs to be controlled.

The implementation idea is to put the requests in a queue. For example, if the number of concurrency is 4, then 4 requests will be initiated at the same time. Then when a request is completed, the next request can be initiated. The idea is clear. The specific code is as follows

Encapsulation implementation

Because this is a very common requirement, many excellent frameworks in the industry have been encapsulated and can be called directly to avoid reinventing the wheel.

Front-end implementation

import React, { useState } from 'react';


import axios from 'axios';


import { useDropzone } from 'react-dropzone';





const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB





const FileUpload: React.FC = () => {


  const [selectedFile, setSelectedFile] = useState<File | null>(null);


  const [uploadProgress, setUploadProgress] = useState<number>(0);


  const [errorMessage, setErrorMessage] = useState<string | null>(null);


  const [uploadedChunks, setUploadedChunks] = useState<Set<number>>(new Set());





  const onDrop = (acceptedFiles: File[]) => {


    setSelectedFile(acceptedFiles[0]);


    setErrorMessage(null);


  };





  const { getRootProps, getInputProps } = useDropzone({ onDrop });





  const queryUploadedChunks = async (fileName: string) => {


    try {


      const response = await axios.get(`http://localhost:8080/upload/status?fileName=${fileName}`);


      setUploadedChunks(new Set(response.data.uploadedChunks));


    } catch (error) {


      setErrorMessage('Failed to query uploaded chunks');


    }


  };





  const uploadChunk = async (chunk: Blob, chunkIndex: number, totalChunks: number) => {


    const formData = new FormData();


    formData.append('file', chunk);


    formData.append('chunkIndex', chunkIndex.toString());


    formData.append('totalChunks', totalChunks.toString());


    formData.append('fileName', selectedFile!.name);





    try {


      await axios.post('http://localhost:8080/upload', formData, {


        headers: {


          'Content-Type': 'multipart/form-data'


        },


        onUploadProgress: (event) => {


          const progress = (event.loaded / event.total!) * 100;


          setUploadProgress(((chunkIndex / totalChunks) * 100) + (progress / totalChunks));


        }


      });


    } catch (error) {


      throw new Error(`Failed to upload chunk ${chunkIndex + 1}: ${error.message}`);


    }


  };





  const handleFileUpload = async () => {


    if (!selectedFile) return;





    await queryUploadedChunks(selectedFile.name);


    const totalChunks = Math.ceil(selectedFile.size / CHUNK_SIZE);





    for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) {


      if (uploadedChunks.has(chunkIndex)) continue; // Resume upload from breakpoint





      const start = chunkIndex * CHUNK_SIZE;


      const end = Math.min(start + CHUNK_SIZE, selectedFile.size);


      const chunk = selectedFile.slice(start, end);





      try {


        // This is a concurrent upload. There is no upper limit for concurrent uploads. If optimized, the number of concurrent uploads can be limited to 5.


        await uploadChunk(chunk, chunkIndex, totalChunks);


        setUploadedChunks(new Set([...uploadedChunks, chunkIndex]));


      } catch (error) {


        setErrorMessage(error.message);


        break;


      }


    }





    if (uploadedChunks.size === totalChunks) {


      console.log('File uploaded successfully');


    }


  };





  return (


    <div>


      <div {...getRootProps()} style={{ border: '2px dashed #000', padding: '20px', cursor: 'pointer' }}>


        <input {...getInputProps()} />


        <p>Drag 'n' drop some files here, or click to select files</p>


      </div>


      {selectedFile && (


        <div>


          <button onClick={handleFileUpload}>Upload</button>


          <progress value={uploadProgress} max="100">{uploadProgress}%</progress>


          {errorMessage && <div style={{ color: 'red' }}>{errorMessage}</div>}


        </div>


      )}


    </div>


  );


};





export default FileUpload;

Backend implementation

import org.springframework.http.HttpStatus;


import org.springframework.http.ResponseEntity;


import org.springframework.web.bind.annotation.*;


import org.springframework.web.multipart.MultipartFile;





import java.io.FileOutputStream;


import java.io.IOException;


import java.nio.file.Files;


import java.nio.file.Path;


import java.nio.file.Paths;


import java.util.HashSet;


import java.util.Set;





@RestController


@RequestMapping("/upload")


public class FileUploadController {





    private static final String TEMP_UPLOAD_DIR = "temp-uploads/";


    private static final String FINAL_UPLOAD_DIR = "uploads/";





    @PostMapping


    public ResponseEntity<String> handleChunkedFileUpload(


            @RequestParam("file") MultipartFile file,


            @RequestParam("chunkIndex") int chunkIndex,


            @RequestParam("totalChunks") int totalChunks,


            @RequestParam("fileName") String fileName) {





        fileName = sanitizeFileName(fileName);


        Path tempFilePath = Paths.get(TEMP_UPLOAD_DIR + fileName + ".part" + chunkIndex);





        try {


            Files.createDirectories(tempFilePath.getParent());


            Files.write(tempFilePath, file.getBytes());





            if (chunkIndex == totalChunks - 1) {


                mergeChunks(fileName, totalChunks);


                return ResponseEntity.status(HttpStatus.OK).body("File uploaded successfully: " + fileName);


            }





            return ResponseEntity.status(HttpStatus.OK).body("Chunk " + (chunkIndex + 1) + " uploaded successfully");





        } catch (IOException e) {


            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error uploading chunk: " + e.getMessage());


        }


    }





    @GetMapping("/status")


    public ResponseEntity<Set<Integer>> getUploadedChunks(@RequestParam("fileName") String fileName) {


        fileName = sanitizeFileName(fileName);


        Path tempFilePath = Paths.get(TEMP_UPLOAD_DIR + fileName);





        Set<Integer> uploadedChunks = new HashSet<>();


        try (Stream<Path> files = Files.list(tempFilePath.getParent())) {


            uploadedChunks = files


                    .filter(file -> file.getFileName().toString().startsWith(fileName))


                    .map(file -> {


                        String fileNameWithPart = file.getFileName().toString();


                        String partIndex = fileNameWithPart.substring(fileNameWithPart.lastIndexOf(".part") + 5);


                        return Integer.parseInt(partIndex);


                    })


                    .collect(Collectors.toSet());


        } catch (IOException e) {


            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(uploadedChunks);


        }





        return ResponseEntity.status(HttpStatus.OK).body(uploadedChunks);


    }





    private void mergeChunks(String fileName, int totalChunks) throws IOException {


        Path finalFilePath = Paths.get(FINAL_UPLOAD_DIR + fileName);


        Files.createDirectories(finalFilePath.getParent());





        try (FileOutputStream outputStream = new FileOutputStream(finalFilePath.toFile(), true)) {


            for (int i = 0; i < totalChunks; i++) {


                Path chunkPath = Paths.get(TEMP_UPLOAD_DIR + fileName + ".part" + i);


                Files.copy(chunkPath, outputStream);


                Files.delete(chunkPath); // Clear space


            }


        } catch (IOException e) {


            throw new IOException("Error merging chunks: " + e.getMessage());


        }


    }





    private String sanitizeFileName(String fileName) {


        return fileName.replaceAll("[^a-zA-Z0-9\\.\\-]", "_");


    }


}