Hot questions for Using Transmission Control Protocol in file transfer

Question:

I am new to Java networking, and having looked for a resolution for my problem for a while now, I figured why not ask some advice from some more qualified people on this matter?

I currently have made a small tool which manages a server of mine, and another small client tool. My goal is for the tool to be able to send commands from the client to the server computer. This way I can perform certain actions on the server computer from another machine, including sending a zip archive with updated files.

I have the basics setup: a TCP connection that sends a command from client to server (server replies with a confirmation) and then I would like the supposed action to take place. My question now is this:

When sending a file (.zip) from the client to server, should I send it over TCP or use something like FTP? I would not only like to send the file to the server, but also when it arrived to extract and replace the existing files.

Kind regards, Alex

EDIT: This is what I used for transferring a file from the client to server, however the file doesn't reach the destination in full size.. D:

Server

package server.control.net.impl;

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;

public class Update implements Runnable {

    private final static int serverPort = 5555;
    private final static String fileInput = "C:\\Users\\Alexander\\Documents\\update.zip";

    public static void main(String args[]) throws IOException{
        ServerSocket servsock = new ServerSocket(serverPort);
        File myFile = new File(fileInput);
        while (true) {
          Socket sock = servsock.accept();
          byte[] mybytearray = new byte[(int) myFile.length()];
          BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
          bis.read(mybytearray, 0, mybytearray.length);
          OutputStream os = sock.getOutputStream();
          os.write(mybytearray, 0, mybytearray.length);
          os.flush();
          sock.close();
        }
    }

    public static void start(){
        Update upd = new Update();  
        Thread tupd = new Thread(upd);  
        tupd.start(); 
    }

    @Override
    public void run() {

    }
}

Client

package server.control.net;

import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.Socket;
import java.net.UnknownHostException;

public class TCPClient {

    private final static String serverIP = "127.0.0.1";
    private final static int serverPort = 5555;
    private final static String fileOutput = "C:\\Users\\Alexander\\Documents\\updateoutput.zip";

    public static void main(String args[]) throws UnknownHostException, IOException {
        Socket sock = new Socket(serverIP, serverPort);
        byte[] mybytearray = new byte[1024];
        InputStream is = sock.getInputStream();
        FileOutputStream fos = new FileOutputStream(fileOutput);
        BufferedOutputStream bos = new BufferedOutputStream(fos);
        int bytesRead = is.read(mybytearray, 0, mybytearray.length);
        bos.write(mybytearray, 0, bytesRead);
        bos.close();
        sock.close();
    }
}

Answer:

You did not get InputStream from the socket after serverSocket.accept(). Open InputStream on the socket.

    clientSocket = serverSocket.accept();

    InputStream in = clientSocket.getInputStream();

    // Writing the file to disk
    // Instantiating a new output stream object
    OutputStream output = new FileOutputStream("YourFile.zip");

    byte[] buffer = new byte[1024];
    while ((bytesRead = in.read(buffer)) != -1) {
        output.write(buffer, 0, bytesRead);
    }
    // Closing the FileOutputStream handle
    output.close();

Refer to working example at : Write and Read File over Socket

Question:

I'm trying to send a file from a client to a server with Sockets in Java. It works fine when I am testing on the same machine, but when I test across different machines, I lose large chunks of data, resulting in a corrupted file. If I try to send a very small file (<20 bytes), it doesn't even reach the println inside the server's while loop.

Here is my code:

Server.java

package edu.mst.cs.sensorreceiver;

import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;

public class Server {
    private static final int PORT = 51111;
    private static final int CHUNK_SIZE = 1024;
    private static final File _downloadDir = new File("downloads/");

    public static void main(String[] args) {
        if (!_downloadDir.exists()) {
            if (!_downloadDir.mkdirs()) {
                System.err.println("Error: Could not create download directory");
            }
        }

        Socket socket = null;
        try {
            ServerSocket server = new ServerSocket(PORT);

            while (true) {
                System.out.println("Waiting for connection...");
                socket = server.accept();

                BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));

                String name = in.readLine();
                File file = new File(_downloadDir, name);

                String size = in.readLine();
                int fileSize;
                try {
                    fileSize = Integer.parseInt(size);
                } catch (NumberFormatException e) {
                    System.err.println("Error: Malformed file size:" + size);
                    e.printStackTrace();
                    return;
                }

                System.out.println("Saving " + file + " from user... (" + fileSize + " bytes)");
                saveFile(file, socket.getInputStream());
                System.out.println("Finished downloading " + file + " from user.");
                if (file.length() != fileSize) {
                    System.err.println("Error: file incomplete");
                }
            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (socket != null) {
                try {
                    socket.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }

    private static void saveFile(File file, InputStream inStream) {
        FileOutputStream fileOut = null;
        try {
            fileOut = new FileOutputStream(file);

            byte[] buffer = new byte[CHUNK_SIZE];
            int bytesRead;
            int pos = 0;
            while ((bytesRead = inStream.read(buffer, 0, CHUNK_SIZE)) >= 0) {
                pos += bytesRead;
                System.out.println(pos + " bytes (" + bytesRead + " bytes read)");
                fileOut.write(buffer, 0, bytesRead);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (fileOut != null) {
                try {
                    fileOut.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
        System.out.println("Finished, filesize = " + file.length());
    }
}

Client.java

package edu.mst.cs.sensorreceiver;

import java.io.*;
import java.net.Socket;

public class Client {
    private static final String HOSTNAME = "131.151.163.153";
    private static final int PORT = 51111;
    private static final int CHUNK_SIZE = 1024;

    public static void main(String[] args) {
        sendFile(args[0]);
    }

    private static void sendFile(String path) {
        if (path == null) {
            throw new NullPointerException("Path is null");
        }

        File file = new File(path);
        Socket socket = null;
        try {
            System.out.println("Connecting to server...");
            socket = new Socket(HOSTNAME, PORT);
            System.out.println("Connected to server at " + socket.getInetAddress());

            PrintStream out = new PrintStream(socket.getOutputStream(), true);

            out.println(file.getName());
            out.println(file.length());

            System.out.println("Sending " + file.getName() + " (" + file.length() + " bytes) to server...");
            writeFile(file, socket.getOutputStream());
            System.out.println("Finished sending " + file.getName() + " to server");
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (socket != null) {
                try {
                    socket.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }

    private static void writeFile(File file, OutputStream outStream) {
        FileInputStream reader = null;
        try {
            reader = new FileInputStream(file);
            byte[] buffer = new byte[CHUNK_SIZE];
            int pos = 0;
            int bytesRead;
            while ((bytesRead = reader.read(buffer, 0, CHUNK_SIZE)) >= 0) {
                outStream.write(buffer, 0, bytesRead);
                outStream.flush();
                pos += bytesRead;
                System.out.println(pos + " bytes (" + bytesRead + " bytes read)");
            }
        } catch (IndexOutOfBoundsException e) {
            System.err.println("Error while reading file");
            e.printStackTrace();
        } catch (IOException e) {
            System.err.println("Error while writing " + file.toString() + " to output stream");
            e.printStackTrace();
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

I've been working on this for hours and I have made almost no progress. I thought I had a pretty good understanding of how reading/writing from streams works, but clearly I'm missing something here.

Again, everything works perfectly when I am sending and receiving from the same machine. But if I try to send a file between two computers, even if the two are on the same LAN, I lose a lot of the data that was sent.

Can anybody figure out my problem? I've already tried everything I could think of.


Answer:

You appear to be mixing chunked data and line oriented operation. I suggest you use a DataInputStream on the server, and a DataOutputStream. Starting on the client, something like

private static void sendFile(String path) {
    if (path == null) {
        throw new NullPointerException("Path is null");
    }

    File file = new File(path);
    Socket socket = null;
    try {
        System.out.println("Connecting to server...");
        socket = new Socket(HOSTNAME, PORT);
        System.out.println("Connected to server at "
                + socket.getInetAddress());

        try (DataOutputStream dos = new DataOutputStream(
                new BufferedOutputStream(socket.getOutputStream()));) {
            dos.writeUTF(file.getName());
            dos.writeLong(file.length());

            System.out.println("Sending " + file.getName() + " ("
                    + file.length() + " bytes) to server...");
            writeFile(file, dos);
            System.out.println("Finished sending " + file.getName()
                    + " to server");
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (socket != null) {
            try {
                socket.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

Then on the server

socket = server.accept();

DataInputStream dis = new DataInputStream(socket.getInputStream());
String name = dis.readUTF();
File file = new File(_downloadDir, name);
long fileSize = dis.readLong();
System.out.println("Saving " + file + " from user... ("
        + fileSize + " bytes)");

Question:

I have an issue with data transfer rate when sending files larger than 15GB. I have 3 servers and one client. When sending a file from client to servers, I split a file into blocks (each block is typically 256MB) and each block is replicated on 2 server. Replication happens in pipeline method. When sending a block, each block is sliced into smaller packets (each packet is typically 128 KB), sent to the servers, and merged on the server side to be stored in hard drive. Everything is fine here. I tested the system for 5GB to 50GB files with the increment of 5GB. Average write is around 600MB/sec for all files. See below chart. Here I am comparing with HDFS.

The problem happens when reading same files from servers. Files are distributed over multiple servers. For instance I can read block1 from server1, block2 from server2, and so on. Intuitively, read must be faster than write because client reads from 3 servers in parallel. when reading files smaller than 15GB {5GB, 10GB, 15GG}, the performance is around 1.1GB/sec. The problem arises when reading files larger than 20GB {20GB, 25GB, ...., 50GB}. Performance decreases as the file size increases.

The above picture shows a benchmark test for reading 50GB file. Each black dot shows an individual block reading time. As you can see, the performance starts to decrease after 60th - 70th block. Interestingly, this happens for all files larger than 15GB, slowing down right around same place (around 65th block). As the size of file increases, the slow part is dominating, and the performance is getting worse. I feel like there is some obstacle around 16GB. The only hint I see that may help is that 3 servers send blocks randomly in parallel until around 65th. So blocks' transfer are overlapping. After that, one server sends at a time in round-robin order. I can see this from the log outputs. There is still some overlapping here but not as much as before 65th block.

I am using java 1.8 for this project, and netty 4.1.8. as tcp server. OS is CentOS 7. Each server has two CPU (Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz ) = 40 cores 64GB RAM 10 GBit Ethernet.

I have spent lots of time and could not find the root cause of problem. The problem may arise from Java VM, Netty, OS, OS TCP defaults, or another reason.

Server Side BlockSenderManager

@Override
    public void run(){

        while(nodeManager.isRunning()){
            try
            {
                BlockRequest br = blockSenders.take();
                if(br != null){
                    executor.execute(new BlockSender( br, this));
                }

                if(wait.take())
                    System.out.println(br.getBlockId()+" Delivered");
            }
            catch (InterruptedException e)
            {
                e.printStackTrace();
            }
        }

BlockSender at Server side:

    @Override
        public void run()
        {
            FileInputStream fis = null;

            try
            {
                java.io.File file = new java.io.File(path+"/" + blockRequest.getBlockId());

                fis = new FileInputStream(file);
                fSize = file.length();
                long rem = fSize;

                sendBlockInfo();
                int bufSize;
                if (fSize < (long) packetSize)
                    bufSize = (int) fSize;
                else
                    bufSize = packetSize;
                int read = 0, packetOrder = 1;

                byte[] data;
                if(bufSize <= rem)
                    data = new byte[bufSize];
                else
                    data = new byte[(int)rem];
                while ((read = (fis.read(data))) > 0)
                {
                    if (read < 1)
                        break;

                    BlockPacket bp = new BlockPacket();

                    bp.setRequestId(blockRequest.getRequestId());
                    bp.setBlockId(blockRequest.getBlockId());
                    bp.setData(data);
                    bp.setPacketSeqNo(packetOrder);
                    if(read < bufSize)
                    {
                        bp.setIsLastPacket(true);
                    }

                    executor.execute(new Sender(bp));

                    packetOrder++;
                    if(rem > bufSize)
                        rem = rem - bufSize;

                    if(bufSize <= rem)
                        data = new byte[bufSize];
                    else
                    {
                        data = new byte[(int)rem];
                    }
                }

                fis.close();
                executor.shutdown();
            }
            catch (FileNotFoundException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            catch (IOException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

public class Sender implements Runnable
    {
        private final BlockPacket bp;
        private final FileBlock fb;
        private DataClient dc;

        public Sender(BlockPacket bp)
        {
            this.bp = bp;
            this.fb = null;
            dc = getDataClient(requestClient);
        }

        public Sender(FileBlock fb)
        {
            this.bp = null;
            this.fb = fb;
            dc = getDataClient(requestClient);
        }

        @Override
        public void run()
        {

            if (dc != null)
            {
                if (bp != null)
                {
                    dc.send(bp);
                }
                else if (fb != null)
                {
                    dc.send(fb);
                }
            }

        }
    }

ReceivedPacketProcessor at Client side

public void processBlockPacket(BlockPacket bp)
    {
        ByteBuffer buf = ByteBuffer.wrap(bp.getData());
        try
        {
            inChannel.write(buf);  
        }
        catch (IOException e)
        {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

    @Override
        public void run()
        {
            try
            {
                aFile = new RandomAccessFile(path+"/"+fileName, "rw");
                inChannel = aFile.getChannel();
                //java.io.File f = new java.io.File(path+"/"+fileName);
                //fop = new FileOutputStream(f);
                String reqId = file.getFileID();
                currentBlockId = reqId + "_" + currentBlockSeq;
                while (true)
                {
                    BlockPacket bp = null;
                    if (numberOfBlocks > 0)
                    {
                        try
                        {
                            bp = this.blockingQueue.take();
                        }
                        catch (InterruptedException e)
                        {
                            e.printStackTrace();
                        }
                        if (bp.getBlockId().equals(currentBlockId))
                        {
                            if (currentPacket == bp.getPacketSeqNo())
                            {

                                if(fileBlocks.containsKey(currentBlockId))
                                {
                                    processBlockPacket(bp);
                                    if(currentPacket < fileBlocks.get(currentBlockId).getNoOfPackets())
                                        currentPacket++;
                                    else
                                    {
                                        if (fileBlocks.get(currentBlockId).getPackets().size() < 1)
                                        {
                                            removeFileBlock(currentBlockId);
                                            currentBlockSeq++;
                                            currentBlockId = reqId + "_" + currentBlockSeq;
                                            currentPacket = 1;
                                            numberOfBlocks--;
                                        } 
                                    } 
                                }
                                else
                                {
                                    tempList.add(bp); 
                                }

                                for(int k =numberOfBlocks; k>0; k--)
                                {
                                    if(fileBlocks.containsKey(currentBlockId))
                                    {
                                        int pCount = fileBlocks.get(currentBlockId).getNoOfPackets();
                                        int i;
                                        for (i = currentPacket; i <= pCount; i++)
                                        {
                                            if (fileBlocks.get(currentBlockId).getPackets().containsKey(i))
                                            {
                                                processBlockPacket(fileBlocks.get(currentBlockId).getPackets().remove(i));
                                                currentPacket++;
                                            }
                                            else
                                            {
                                                break;
                                            }
                                        }
                                        if(i <= pCount)
                                        {
                                            break;
                                        }
                                        else
                                        {
                                            if (fileBlocks.get(currentBlockId).getPackets().size()  < 1)
                                            {
                                                removeFileBlock(currentBlockId);
                                                currentBlockSeq++;
                                                currentBlockId = reqId + "_" + currentBlockSeq;
                                                currentPacket = 1;
                                                numberOfBlocks--;
                                            } 
                                        }
                                    }
                                }
                            }
                        }
                        else
                        {
                            if(fileBlocks.containsKey(bp.getBlockId())){
                                fileBlocks.get(bp.getBlockId()).getPackets().put(bp.getPacketSeqNo(), bp);
                            }else{
                                tempList.add(bp);
                            }

                        }
                    }
                    else{
                        break;
                    }
                    for(int i=0; i<tempList.size(); i++){
                        if(fileBlocks.containsKey(tempList.get(i).getBlockId())){
                            BlockPacket temp = tempList.remove(i);
                            fileBlocks.get(temp.getBlockId()).getPackets().put(temp.getPacketSeqNo(), temp); 
                        }
                    }
                }
                System.out.println("CLOSING FILE....");
                this.isCompleted.put(true);
                inChannel.force(true);
                inChannel.close();
            }
            catch (FileNotFoundException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            catch (IOException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
            catch (InterruptedException e)
            {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

With -XX:+PrintGCDetails turned on, here is a sample log.

Any comment/help is appreciated.


Answer:

This is because the dirty page ration in memory. Since the incoming data rate is higher than the local IO flush throughput, the data is accumulated in memory. Once the max allowed dirty page ratio is reached, the receiver does not accept more data. Thus, the system is bounded with local IO, not the network in this case. So the diminishing of return happens at about 15GB. You can change some setting in

/etc/sysctl.conf

Such as:

vm.dirty_background_ratio = 2
vm.dirty_ratio = 80
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500

This might be a useful reading.

The system performance is still bounded by local IO and max allowed dirty page ratio. You can increase increase the dirty page ratio to only postpone the diminishing of return time. It will reach again this point if the file/data is larger. New Results:

Question:

Despite hours of researching this problem, I have made very little progress. According to my professor, the code should be working as written...

I have a server that stays open, and a client that requests a file. Once the client receives the file, the client closes.

When I open the server, I am able to transfer a complete .jpg image file. The client then closes while the server remains open. I start up another client and try to transfer the same image, and only a portion of the bytes are transferred/written to the disk. The file transfer is only completely successful for the first file transferred by the server!

Additionally strange, a simple .txt text file never successfully transfers. I believe the cause is on the server side because it remains open as opposed to the client, which starts over each time.

Server Code:

import java.io.*;
import java.net.*;
import java.util.Arrays;

class ft_server {

    public static void main(String args[]) throws Exception {

        /*
         * Asks user for port number and listens on that port
         */
        BufferedReader portFromUser = new BufferedReader(new InputStreamReader(System.in));
        System.out.println("Enter the port you'd like to use: ");
        int portNumber = Integer.valueOf(portFromUser.readLine());

        if (portNumber < 1 || portNumber > 65535) {
            System.out.println("Please choose a port number between 1 and 65535.");
            return;
        }
        portFromUser.close();


        ServerSocket listenSocket = new ServerSocket(portNumber);
        /*
         * Finished with user input
         */

        /*
         * Continuously listens for clients:
         */
        while (true) {
            Socket clientSocket = listenSocket.accept();
            BufferedReader inFromClient = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
            DataOutputStream outToClient = new DataOutputStream(clientSocket.getOutputStream());

            String clientIP = clientSocket.getRemoteSocketAddress().toString();
            System.out.println("The client " + clientIP + " connected!");

            String clientMessage = inFromClient.readLine();
            System.out.println("The client requested file: " + clientMessage);

            // Get file. If doesn't exist, let's client know.
            // Otherwise informs client of file size.
            File myFile = new File(clientMessage);

            if (!myFile.exists()) {
                outToClient.writeBytes("File does not exist!\n");
                return;
            } else {
                outToClient.writeBytes(String.valueOf((int)myFile.length()) + "\n");
            }

            // Create array for storage of file bytes:
            byte[] byteArray = new byte[(int)myFile.length()];
            BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));

            // Read file into array:
            bis.read(byteArray, 0, byteArray.length);

            // Send the file:
            outToClient.write(byteArray, 0, byteArray.length);

            outToClient.close();
            clientSocket.close();
        }
    }
}

Client Code:

import java.io.*;
import java.net.*;

class ft_client {

    public static void main(String args[]) throws Exception {

        int byteSize = 2022386;
        int bytesRead;

        /*
         * Asks user for IP and port:
         */
        BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
        System.out.println("Enter an IP address: ");
        String ipAddress = inFromUser.readLine();
        System.out.println("Enter a port: ");
        String port = inFromUser.readLine();

        Socket clientSocket;

        try {
            // Makes socket, port, and calls connect. Assumes it's TCP:
            clientSocket = new Socket(ipAddress, Integer.valueOf(port));
        } catch (Exception e) {
            System.out.println(e.getMessage());
            return;
        }

        // Creates InputStream from server to get file size and other messages:
        BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));

        // Anything written to this will be sent to the server:
        DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());

        // Asks for a file name to download from the server:
        System.out.println("What file do you want?: ");
        String message = inFromUser.readLine();
        outToServer.writeBytes(message + "\n");
        inFromUser.close();

        // Listens for confirmation from server.
        // If the file exists, the file size is delivered here:
        String response = inFromServer.readLine();
        System.out.println("File size: " + response);
        if (response.equals("File does not exist!")) {
            return;
        }

        // Receives file from server:
        byteSize = (int) Integer.valueOf(response);
        byte[] byteArray = new byte[byteSize];
        InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???
        FileOutputStream fos = new FileOutputStream(message);
        BufferedOutputStream bos = new BufferedOutputStream(fos);       

        // Continuously writes the file to the disk until complete:
        int total = 0;
        while ((bytesRead = is.read(byteArray)) != -1) {
            bos.write(byteArray, 0, bytesRead);
            total += bytesRead;
        }

        bos.close();
        System.out.println("File downloaded (" + total + " bytes read)");

        clientSocket.close();
    }
}

Are buffered readers interfering with output streams? Is there a better way to transfer files?


Answer:

It's worth checking, in your server code, what value comes back from the file read() call, so:

int bytesRead = bis.read(byteArray, 0, byteArray.length);
System.out.println("File bytes read: " + bytesRead + " from file size: " + myFile.length());

The read() method is under no obligation to fill the byteArray - only to return something and to tell you how many bytes it read. From the docs, it:

Reads up to len bytes of data from this input stream into an array of bytes. If len is not zero, the method blocks until some input is available; otherwise, no bytes are read and 0 is returned.

You need to keep reading in a loop. I'd do this (actually, same as your client!):

int n;
while ((n = bis.read(byteArray, 0, byteArray.length)) != -1) {
    // Send the chunk of n bytes
    outToClient.write(byteArray, 0, n);
}
bis.close();
outToClient.close();

or something similar. I've closed the file too: it'd close on GC/finalize, but that could be a while, and meanwhile you're holding the file open.

EDIT

The specific problem with your image-read in this case is in your client code. You read the file size near the top of the code:

    // Creates InputStream from server to get file size and other messages:
    BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));

and then you access the client again:

    InputStream is = clientSocket.getInputStream(); // calling clientSocket.getInputStream() twice???

and as your comment suggests, this is bad! Thank you to @EJP for highlighting this!

This causes a problem of buffer over-ingestion: the BufferedReader consumes more bytes into its belly than you extract from it, so when you visit the clientSocket inputstream the second time, the read-pointer has moved on. You never look again at what the BufferedReader consumed.

As a general rule, once you plug buffering code onto something, you must be careful to read only from that buffer. In this case, it's difficult, because you can't read image (raw binary) data from a Reader, because it will busily interpret the binary values as characters and read them as UTF-8 or something.

Even without buffers, it's a minor sin to mix Readers (text oriented) and binary data (DataStreams) on the same stream. HTTP and email does this, so you are in good company, but they get away with it by being very tightly specified. Problem is, you can easily get snarled with questions of local/default character encoding at each end, whether you're reading Unix "LF" vs Windows "CR/LF" line endings etc.

In this case, try not using BufferedReaders at all, and try using DataInput/Output streams all the way. Try writeUTF(s) and readUTF() for transferring the String data. Ideally, create them like this:

    DataInputStream inFromServer = new DataInputStream (new BufferedInputStream(clientSocket.getInputStream()));

so you still get the benefits of buffering.

EDIT 2

So seeing the new client code:

        byteSize = (int) Integer.valueOf(response);
        byte[] byteArray = new byte[byteSize];
        FileOutputStream fos = new FileOutputStream(message);

        int readBytes = inFromServer.read(byteArray);

        // Continuously writes the file to the disk until complete:
        int total = 0;
        for (int i=0; i<byteArray.length; i++) {
            fos.write(byteArray[i]);
            total++;
        }

        fos.close();

Here, we're assuming that because the byteArray array is set to the right size, that the inFromServer.read(byteArray) will populate it - it won't. It's good to assume that any and all read operations will return you just as much data as the system has to hand: in this case, it's probably going to return as soon as it gets the first packet or two, with an underfilled array. This is same as C and Unix read behaviour too.

Try this - I'm repeatedly reading and writing a 4K buffer, until the byte count is reached (as determined by summing the return values of the reads):

        byteSize = (int) Integer.valueOf(response);
        byte[] byteArray = new byte[4096];
        FileOutputStream fos = new FileOutputStream(message);
        int total = 0;
        // Continuously writes the file to the disk until complete:
        while (total < byteSize && (readBytes = inFromServer.read(byteArray)) != -1) {
            fos.write(byteArray, 0, readBytes);
            total += readBytes;
        }
        fos.close();

A variant is this - same thing, but byte at a time. Might be a bit clearer. It's going to be slow - all those reads and writes are hitting the OS, but if you put a BufferedInputStream/BufferedOutputStream around the socket/file streams, it'll iron that out. I've added them:

    DataInputStream inFromServer = 
            new DataInputStream(new BufferedInputStream(clientSocket.getInputStream()));
    ...         
        byteSize = (int) Integer.valueOf(response);

        OutputStream fos = new BufferedOutputStream(FileOutputStream(message));
        int total = 0;
        int ch;
        // Continuously writes the file to the disk until complete:
        while (total < byteSize && (ch = inFromServer.read()) != -1) {
            fos.write(ch);
            total ++;
        }
        fos.close();

And finally! the simplest answer is this. Your code, but changed to:

        int readBytes = inFromServer.readFully(byteArray);

Yes! Those nice people in 1990's Javasoft added a DataInput.readFully method, which does what you want! - basically wraps the code above. It's the simplest solution, and arguably most correct approach: "use existing libraries where possible". OTOH, it's the least educational, and the time you spend getting used to read/writes like this is not deducted from your life-expectancy!

And in fact, the readFully approach has severe limitations. Try pointing it at a 1GB file and see what happens (after you've fixed up the array size at the top): you'll a) run out memory, and b) wish that while you were ingesting a huge blob, you could at least be spooling it out to disk. If you try a 2.5G file, you'll notice that some of those ints should become longs to cope with numbers >= 2^31.

If it was me, I'd do the 4K buffer one. (BTW I'm writing this on a laptop with no Java compiler installed, so I haven't actually run the above! DO respond if there are any difficulties.)

Question:

I'm trying to make a video file transfer but am having problems getting the server to start sending bytes.

The first step is for the client to connect, the socket gets accepted. Then the client sends the video file name but the server never reads this.

This is the code for the server up until it blocks:

 try(ServerSocket serverSocket = new ServerSocket(4005))
    {
        Socket socket = serverSocket.accept();
        System.out.println("accepted");
        OutputStream os = socket.getOutputStream();

        BufferedReader receiveReader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
        System.out.println("This gets printed");
        String request = receiveReader.readLine();//never passes this line
        System.out.println("This doesn't get printed");

and this is the client up until it blocks waiting for the server to send the video bytes:

 try(Socket socket = new Socket(IPAddress, 4005))
            {
                byte[] messageBytes = new byte[10000];

                DataOutputStream outputStream = new DataOutputStream(socket.getOutputStream());
                outputStream.writeBytes("REQUEST;"+videoPath);//This is the line that should send the bytes for the server to read, so it won't block.

                String home = System.getProperty("user.home");
                String path = home+"\\Downloads" + videoName; 
                path = path.trim();
                FileOutputStream fos = new FileOutputStream(path);
                BufferedOutputStream bos = new BufferedOutputStream(fos);
                InputStream is = socket.getInputStream();

                int bytesRead = 0;
                System.out.println("Downloading file...");
                while((bytesRead = is.read(messageBytes))!=-1)//This blocks here

Why on earth isn't the server reading the "Request" + videoPath bytes that the server is sending? I tried outputStream.flush() as well, no luck.


Answer:

Usual problem. You're reading a line but you aren't writing a line. Add a line terminator to the sent message.

When you fix this you will then discover that you can't mix buffered streams and readers on the same socket. I suggest you do all the I/O via the DataInput/OutputStream classes, using read/writeUTF() for the name.

If you're sending multiple files see my answer there.

Question:

I've been writing a simple file transfer program which opens a TCP connection to a server, negotiates the terms of the file transfer (over TCP) and streams the file from the client to the server. The problem is large files almost always corrupt. I logged the socket InputStream and OutputStream and found that while the data matches at first, after several kilobytes an interesting desync occurs.

At a certain point the receiving socket starts reading 0 while the sending socket continues sending the file data.

Example:

Server: ..., -75, 82, 34, -109, 50, -51, 52, 9, -14, -70,...
Client: ..., -75, 82, 34, -109, 50,   0,  0, 0,   0,   0,...

However After a bit it recovers

..., 0, 0, 0, 0, 0, -51, 52, 9, -14, -70,...

You'll note it resumes where it left off. The only problem is now I have a bunch of zeros in my file and the end of the stream will not be recorded into the file (as the file length is sent ahead of time).

NOTE: this only seems to happen some of the time (more often than not)

The streaming code itself:

Sender:

<header stuff, I checked it works fine>

long dataLeft = 0;
while(dataLeft < packet.payloadSize){
    packet.remaining = packet.payloadSize-dataLeft;
    byte[] temp = new byte[packet.remaining<8192?(int)packet.remaining:8192];
    packet.inStream.read(temp);
    stream.write(temp);
    dataLeft+=temp.length;
}

stream.flush();

<Rest of the code>

Receiver:

public boolean streamPayload(OutputStream out) throws IOException{
    long dataLeft = 0;
    while(dataLeft < payloadSize){
        remaining = payloadSize-dataLeft;
        byte[] temp = new byte[remaining<8192?(int)remaining:8192];
        inStream.read(temp);
        out.write(temp);
        dataLeft+=temp.length;
    }
    return true;
}

long dataLeft acts as the index btw.

Does anyone know why this might be happening and how I can fix it/compensate for the issue? I know I can't tell if a zero is intended so I can't just listen for the issue by watching the output.


Answer:

I don't know if this will solve your problem but one thing you are doing wrong is ignoring the amount of data transferred.

InputStream in = //from somewhere
OutputStream out = //to somewhere
byte[] buffer = new byte[8192];
int read = -1;
while ((read = in.read(buffer)) > -1) {
    out.write(buffer, 0, read);
}
out.close();
int.close();

You could be reading data without receiving any and then writing 8192 byte buffer that is empty because you didn't check if anything came through.