Hot questions for Using Transmission Control Protocol in packet

Question:

I am developing a simple client server program using this link. My server is a Mac machine and client is a Windows machine (Win 10).

The client-server communication works fine. When I examined the bytes sent using Wireshark, in the first TCP packet only one character gets sent. And there is another TCP packet where the rest of the data gets sent.

i.e. If I send "qwerty", client send "q", server responds, then client sends "werty" . Similarly Server sends "Q", client responds then server sends "WERTY".

Here is my client server code. I am flushing the data after doing writeBytes().

How can I enforce the data to be sent in a single TCP packet?

Client Code:

import java.io.*;
import java.net.*;

class Client
{
    public static void main(String argv[]) throws Exception
    {
        String sentence;
        String modifiedSentence;
        BufferedReader inFromUser = new BufferedReader( new InputStreamReader(System.in));
        Socket clientSocket = new Socket("192.1.162.65", 6789);
        DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
        BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
        System.out.println("Enter some text: " + inFromServer);
        sentence = inFromUser.readLine();
        outToServer.writeBytes(sentence + '\n');
        outToServer.flush();
        modifiedSentence = inFromServer.readLine();
        System.out.println("FROM SERVER: " + modifiedSentence);
        clientSocket.close();
    }
}

Server Code:

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.ServerSocket;
import java.net.Socket;

public class Server {

    public static void createListener(ServerSocket serverSocket) throws Exception {

        String clientSentence;
        String capitalizedSentence;
        @SuppressWarnings("resource")
        ServerSocket welcomeSocket = new ServerSocket(6789);

        while (true) {
            @SuppressWarnings("resource")
            Socket connectionSocket = welcomeSocket.accept();
            BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
            DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());
            clientSentence = inFromClient.readLine();
            System.out.println("Received: " + clientSentence);
            capitalizedSentence = clientSentence.toUpperCase() + '\n';
            outToClient.writeBytes(capitalizedSentence);
            outToClient.flush();
        }
    }

    public static void main(String[] args) throws Exception {
        Server.createListener(null);
    }
}

Edit 1: The reason behind this question is, I am trying to use the same server code to send data to another client that is not in my control. In that case, the above code sends just the first TCP packet. I do not see the second TCP packet on Wireshark. Any suggestion on how I can go about debugging it?


Answer:

To my knowledge, you can't and more importantly: you should not.

The point is: those libraries intend to create an abstraction to you. The TCP protocol is actually complex; and most likely; you do not want to deal with all the subtle details.

So the non-answer here: unless you encounter real issues with this implementation (like an un-acceptable performance hit) - you should rather look into writing clear, readable code; instead of fiddling with TCP stack implementation details!

Example: your code doesn't deal with exceptions, you just let them pass through. Your code isn't closing streams; your code is containing suppress warnings annotations. And obviously: it was never written with unit-tests in mind.

Such things matter. Not if the JVM sends one TCP package or two!

Question:

i have some problem with receive byte packets by TCP in JAVA. My TCPServer class sending 207 - byte packets. When I send one packet, program display in console "Read 207 byte packet." and stops. With next packet it continue execution, display "Multiple Measurement" and "Read 1868767867 byte packet.". After that receiving is stopped forever. I don't know why 1868767867 bytes it receive. I check it in wireshark and server sending always 207 bytes.

This is my TCPClient class:

public class TCPClient extends Thread {

private ServerSocket serverSocket; 
private Socket connectionSocket;
private InputStream inputStream;
private DataInputStream dataInputStream;


public TCPClient() throws IOException {
    try {
        serverSocket = new ServerSocket(Config.TCP_PORT);
    } catch (IOException e) {
        e.printStackTrace();
    }
}

@Override
public void run() {

    try {
        connectionSocket = serverSocket.accept();
        inputStream = connectionSocket.getInputStream();
        dataInputStream = new DataInputStream(inputStream);
    } catch (IOException e) {
        e.printStackTrace();
    }
    while(true) {
        try {
            JsonObject json = getJsonFromTcp();
            if (json != null) {
                    String command = json.getAsJsonPrimitive("command").getAsString();
                    if(command.equals("multipleMeasurement")) {
                        executeMultipleMeasurement();
                    }
                }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

private JsonObject getJsonFromTcp() throws IOException {    

    byte[] buff = new byte[4];
    for(int i = 0; i < 4; i++) {
        buff[i] = dataInputStream.readByte();
    }

    int len = (((buff[3] & 0xff) << 24) | ((buff[2] & 0xff) << 16) | ((buff[1] & 0xff) << 8) | (buff[0] & 0xff));

    if(len > 0) {
        System.out.println("Read " + len + " byte packet.");
        byte[] data = new byte[len];
        dataInputStream.readFully(data);
        String jsonString = new String(data, "UTF-8");
        JsonParser jsonParser = new JsonParser();
        JsonObject json = jsonParser.parse(jsonString).getAsJsonObject();
        return json;
    }
    return null;
}

private void executeMultipleMeasurement() {
    System.out.println("Multiple Measurement");
}
}

Anyone know solution?


Answer:

Looking at the number 1868767867, its bytes are

"%c%c%c%c" % (0x7b,0x22,0x63,0x6f)
'{"co'

So you could be reading the first four bytes of the next message as the length of the message. The most likely explanation given the claim that the server sends exactly 207 bytes each time is that the server includes the length of the length prefix (4 bytes) in the total message length. Depending on the intended protocol, it might be appropriate to read (length - 4) bytes as the body of the packet.

// Account for the length of the header
len -= 4;

if(len > 0) {
    System.out.println("Read " + len + " byte packet.");
    byte[] data = new byte[len];
    dataInputStream.readFully(data);

A second possibility is that the server is measuring the number of characters in a string and then using that length as the length of the utf-8 converted byte buffer that it will send, including some non-ascii characters that cause the resulting buffer to be longer.

Without seeing the server code, it's impossible to be certain which is happening here.

Question:

I have a TCP server in Java and a client written in Python. The python client simply sends 10 packets waiting 2 seconds in between each send. However, the java server doesn't seem to recognize the packets until the python script terminates, in which it finally receives all the messages as if they came in at the same time. I have watched with wireshark and verified that the client is sending all the packets correctly.

ServerSocket serverSocket = new ServerSocket(6789); 

Socket clientSocket = null; 
System.out.println ("Waiting for connection.....");

try { 
     clientSocket = serverSocket.accept(); 
} 
catch (IOException e) { 
     System.err.println("Accept failed."); 
     System.exit(1); 
} 

System.out.println ("Connection successful");
System.out.println ("Waiting for input.....");

String data; 

PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true); 
BufferedReader in = new BufferedReader( 
        new InputStreamReader(clientSocket.getInputStream())); 

while ((data = in.readLine()) != null) { 
    System.out.println ("Server: " + data);         
} 

Python code: import socket import time

TCP_IP = '192.168.1.127'
TCP_PORT = 6789

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.connect((TCP_IP, TCP_PORT))

for i in range(10):
    sock.send("Packet #: %s" % str(i))
    time.sleep(2)

Output I'm seeing:

Connection successful
Waiting for input.....
Server: Packet: #: 0Packet: #: 1Packet: #: 2Packet: #: 3Packet: #: 4Packet: #: 5Packet: #: 6Packet: #: 7Packet: #: 8Packet: #: 9

Answer:

You are using BufferedReader.readLine() in the server... but never sending a new line symbol from the client.

As such, the server doesn't know whether it is going to receive more data in that line, and so has to wait until it is sure that no more data will be sent on that line: closing the socket is one way to indicate that.

Add \n to the end of the packet you send from the client.

sock.send("Packet #: %s\n" % str(i))

Question:

So I wanna communicate between a C++ client and a Java server. So far so good (yeah I know about the Endiness prob) but I would like to read a defined structure on my serverside which was defined on clientside.

Let's say my client sends a packet in this structure:

char *pLoginData = new char[512];
// ...
char *packet = pLoginData;
*(WORD*)pLoginData = (usernameLength + 1 + passwordLength + 1);

pLoginData += 2;

strcpy((char*)(pLoginData), username.c_str());
pLoginData += usernameLength + 1;

strcpy((char*)(pLoginData), password.c_str());
pLoginData += passwordLength + 1;

// now send it
m_iResult = send(connectSocket, packet, *(WORD*)(packet) + 2, 0);

On my serverside how far I got is:

// init socket, init DataInputStream..
// ...
messageByte[0] = in.readByte(); // defined as: byte[] messageByte = new byte[1000];
System.out.println("Packet len: " + messageByte[0]);

bytesRead = in.read(messageByte);
msg += new String(messageByte, 0, bytesRead); // defined as: String msg = null;

System.out.println(msg);

Output is unfortunately:

Packet len: 16
null testuser testpw 

(there is a space after testpw, why? because look on how the client constructs the packet) As you can see, my output contains the word "null" (dunno why?), but you see aswell that the word "null" isn't counted, cause packet len returns 16 which is: username (space) pw (space) (testuser testpw )

Hope you guys could help me out here :) Thanks in advance!


Answer:

As you said in a comment in the code, you're defining msg as a null string and then appending to it.

You should declare msg as an empty string.

Question:

I'm trying to send a TCP packet. It sends correctly to the server but sender is not getting response (server is sending response back correctly). Client doesn't even process code afeter sending the packet...

 Socket socket = new Socket (ip, port);

 PrintWriter mOut = new PrintWriter(socket.getOutputStream(), true);  
 mOut.print("DSPSYSSTS");

 //Everything works fine until here

 BufferedReader mIn = new BufferedReader (new InputStreamReader (socket.getInputStream ()));
 String fromClient = mIn.readLine();
 out.println ("Client Message: " + fromClient);

 mOut.close();
 mIn.close ();      
 socket.close ();

The JSP doesn't print the input and it remains loading forever. What's wrong?

Returning String of systemRequest.request in below code

ReadSpoolFile readSplf = new ReadSpoolFile(splfArray.get(0));
String splfContent = readSplf.read();

GetSystemStatus getSysSts = new GetSystemStatus();
String systemStatus = getSysSts.get(splfContent);
return systemStatus + "\r\n";

Server side Response:

String response = systemRequests.request(message, SystemRequests.SILENT_OFF);

ChannelBuffer mCbResponse;

if(response != null){
mCbResponse = ChannelBuffers.copiedBuffer(response.getBytes());
mChannel.write(mCbResponse);    //<------Write response

Answer:

Try this:

mOut.print("DSPSYSSTS");
mOut.flush();
...

Question:


Answer:

You should send network requests in a background thread. Use AsyncTask or Thread.

new Thread(new Runnable(){
    public void run(){
        mTcpClient.sendMessage("test");
    }
}).start();

Question:

While SocketIn/OutputStreams in Java seem stream based to an API user, TCP packets are packet based. One can write only one byte or an array of bytes. But one could also write more bytes than a TCP packet could carry.

So how does Java form TCP packets from the write methods?

Creates calling write(singleByte) 4 times 4 TCP packets? Or does java join the bytes together?

Does java join and split bytes to form the TCP packets?


Answer:

No, Java, or more precisely the Java Virtual Machine, has no reason to get into these details.

What the JVM does, is that it opens a native Socket just like any native program would. It then allows the Java code to interact with the native socket through the various Java API. This leaves the networking details to the operating system's network stack (TCP/IP implementation).

Question:

I develop a Peer-to-Peer Project, i try to communicate two machines (A my laptop connected by WiFi and B my NAS connected by Ethernet) on the same network. The communication work well but i have an issue when i send a packet of size 1696bytes.

So in my sender A i write the block of 1696bytes to B, when i look on Wireshark i see two packets are send one of size 1448bytes and another of size 248bytes (1448+248=1696).

So i think the packet are cut by one protocol or something between the network and physical layer because 1448 can correspond to a MTU. And so in my receiver B something weird happen, if i send two times the block of 1696, the first time i will read one block of 1696 even if in Wireshark i see the two splitted blocks, and the second time i read two times for the two splitted blocks...

For me it's a problem because i need the two blocks together to decrypt them.

And when i send from B to A in A and i receive always one block of size 1696bytes, and in Wireshark i see also one block of 1696bytes. So maybe the issue come from Ethernet...

So i don't know which solution is the best, i could read bytes one to one until a size defined like 2048 and so apply a padding to my packet in the sender to have a 2048 packet size. Or maybe you have other fix to give me.

I hope i was clear,

Thank you,

The Receiver class :

public abstract class DataReceiver implements Runnable {
protected ConnectionType    type;
protected AsymmetricEncryption as;
protected Socket            socket;
protected FIFOQueue         buffer;
protected int               dataPacketSize;

protected abstract IDataStructure dataFormatter(byte[] data, int len) throws NoSuchPaddingException, NoSuchAlgorithmException, IllegalBlockSizeException, InvalidKeyException, InvalidAlgorithmParameterException, BadPaddingException;

public DataReceiver(ConnectionType type, Socket socket, FIFOQueue buffer, AsymmetricEncryption as) {
    this.type = type;
    this.socket = socket;
    this.buffer = buffer;
    this.dataPacketSize = 2048;
    this.as = as;
}

public int waitData() {
    try {
        System.out.println(socket.getPort() + " : I wait data from " + socket.getInetAddress().getHostAddress());
        byte[] data = new byte[dataPacketSize];
        int len;
        IDataStructure dataStructure;
        while ((len = socket.getInputStream().read(data, 0, data.length)) != -1) {
            try {
                if ((dataStructure = dataFormatter(data, len)) == null) {
                    System.err.println("Error: bad data format.");
                } else if (type == ConnectionType.EXTRA) {
                    ((ExtraDataStructure)dataStructure).getContent().setParam();
                    buffer.putData(dataStructure);
                } else if (type == ConnectionType.INTRA) {
                    ((IntraDataReceiver)this).getRqManager().doRequest((IntraDataStructure)dataStructure, buffer, socket);
                }
                Arrays.fill(data, (byte)0);
            } catch (NoSuchPaddingException | NoSuchAlgorithmException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException | JsonSyntaxException | InvalidAlgorithmParameterException e) {
                e.printStackTrace();
            }
            System.out.println("New data received from " + socket.getInetAddress().getHostAddress());
            //dumpBuffer();
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
    return (0);
}

    @Override
    public void run() {
        waitData();
    }
}

The sender class :

public class ExtraDataDispatcher extends DataDispatcher {

public ExtraDataDispatcher(AsymmetricEncryption as, NodeIdentifier nodeId) {
    super(ConnectionType.EXTRA, as, nodeId);
}

@Override
public boolean dispatchData(IDataStructure data, Socket destSocket) throws IOException {
    System.out.println("HERE");
    destSocket.getOutputStream().write(new Gson().toJson(((ExtraDataStructure)data).getContent()).getBytes());
    return false;
}
}

Answer:

TCP is a stream protocol.

Writing to a TCP socket is like writing to a file.

Reading from a TCP socket is like reading from a file.

A single read() call doesn't correspond to a single send(). Instead, it just reads whatever is available in the TCP buffer at the moment. Whether it's half a message, a whole message or a hundred messages.

If you want individual messages written to a file to be separate from one another, you'd need some way of telling when a message starts, and when it ends. There are plenty of ways of doing that, here are a couple:

  • In text files, one can use a delimiter such as"\n to separate parts of the file. In a binary file, this is trickier unless you can guarantee that the delimiter won't show up in the middle of a message. The same is true in TCP. You could delimit messages using some special value (e.g. \0). Then, all you'd need to do on the other side is read from the socket until you see the delimiter. Don't do this if you can't guarantee that your message body doesn't contain the delimiter for obvious reasons.

  • Count. Prefix each message with, say, an integer (4 bytes) that indicates the length of the message. So if you want to send 01 02 03 aa bb cc dd (hex), send 00 00 00 07 01 02 03 aa bb cc dd. The receiver would read the first 4 bytes of every message, and figure out the number of bytes it has to read to get the entire message. This does however require the sender to know the length of its message in advance. It's not a problem in most cases, though.

Question:

I am using spring-integration TCP connections. From the Serializer implementation i write some bytes directly to the given OutputStream.

The first byte seems to be sent on its own (in it's own TCP packet, which is by the way reported malformed by wireshark) and then the remaining bytes in the next packet.

Changing the first two bytes to be written in one call (two byte long array) causes these two bytes to be sent in a seperate packet (not malformed) and the remaning in the next packet. This outcome is reproducible.

Is this something I can (or should) affect? It is causing problems on the reading end, because that equipment apparently expects the entire message to be contained in a single TCP packet.


Answer:

Is this something I can (or should) affect? It is causing problems on the reading end, because that equipment apparently expects the entire message to be contained in a single TCP packet.

Then the author of this application has no idea of TCP. TCP is a stream based and not a packet based protocol. This means that a single write can result into multiple packets on the wire or multiple writes can result in a single packet.

...which is by the way reported malformed by wireshark...

If you capture the packets on the sender system you probably see the effects of offloading the checksum computing to the network card. In this case wireshark will report the checksums wrong. If you capture the packets in-between or on the receiving end you should see the correct checksum because the network card on the senders machine calculated the checksum before forwarding the packet.

Question:

Me asking one of those questions again :P Since I'm porting my C++ server to a Java server (and it's nearly done) I'm missing only one thing: Sending files. I can't wrap my head around on how to construct the packet in Java to send it via DataOutputStream. In C++ I prepared the packet in this way (first 4bytes reserved for the file_size, rest the file itself):

char *pData = new char[file_size];

memcpy(pData, charArray.data(), file_size);

char *packet = new char[file_size + 4];
memset(packet, 0, file_size + 4);

*(int*)(packet) = file_size;
memcpy((char*)(packet + 4), pData, file_size);

int r = file_size + 4;
sendall(stream, packet, &r);

I hope you can help me out here, I'm able to construct simple packets but this one is giving me a headache :P Do I merge the bytes or how would I accomplish the C++ code in Java x..x Thanks in advance!

sendall func:

int sendall(TCPStream *s, char *buf, int *len)
{
    int total = 0;
    int bytesleft = *len;
    int n;

    while(total < *len)
    {
        n = s->send(buf+total, bytesleft);
        if (n == -1) break;
        total += n;
        bytesleft -= n;
    }

    *len = total;

    return n ==-1 ? -1 : 0;
}

Answer:

The Java equivalent of that C code is:

int file_size = ...;
byte[] file_data = ...;

byte[] packet = new byte[file_size + 4];
ByteBuffer.wrap(packet).order(ByteOrder.LITTLE_ENDIAN).putInt(file_size);
System.arraycopy(file_data, 0, packet, 4, file_size);

You need to confirm whether the file size was sent with high or low byte first, and change to BIG_ENDIAN if needed.

Since C code just added file size as int, and was likely run on Intel processor, the result was LITTLE_ENDIAN, which is why I specified that in the code above. But as @paulsm4 said, you should test this.

Question:

I have a client(one machine) which sent the request to server(another machine).

Client sends request to server Server processes request Server sends response to client

In this scenario, when there is a TCP reset observed from client side, will there be a scenario where in my server wont be able to conjure/generate a response towards the client ?

I am using Java program.


Answer:

Since client has reset the connection, meaning the old session between client and server has been closed. In this case, it's impossible for server to send response back to the client within closed session.

Server can only send back response for an alive connection, which means the reset connection.

Question:

It seems that I can't figure out, why am I getting first packet splitted, and the rest of the packets I receive as one.

The first received thing is IMEI (17 bytes), the netty server sends back 01 respond and starts sending the packets, which I respond to them with another respond.

But why do I keep getting first packet in two parts ? While others are being send as one (which is ok). It always receives up to 1024 bytes and then the rest of 251 bytes . The whole package is up to 1275 bytes..


Answer:

Generally speaking there is not guarantee if a packet is split or not when using TCP. So you can not make any assumptions on this.

That said what you see may be the result of using AdaptiveRecvByteBufAllocator (which is the default) as it starts with small allocation sizes and then increase these if needed.

You could use a different RecvByteBufAllocator if you want to change the behaviour. But again this is nothing you can depend on.

Question:

currently I am working with an inhouse protocol where I send a request to our hardware and receive the answer with netty. In the message which I receive are several bytes which tell me how many bytes the answer will contain. In my channelRead method I wait until the readable bytes of the recieved message are equal or greater than the expected bytes to make sure I get all data.

if (((ByteBuf) msg).readableBytes() >= dataSize) {
        //do something with the bytes
        ctx.close();
        ((ByteBuf) msg).release();
}

This works fine if I receive exactly one tcp package from the hardware. Sometimes the hardware splits the TCP frame into several packages and my channelRead waits for ever.

Is there a simple way in netty to reassemble these packets in the channelRead method?


Answer:

Just extend ByteToMessageDecoder. This will handle all the buffering for you. Check the javadocs for more details and an example.

Question:

So I have an Android app and a socket server. The communication seems fine when sending the first packet from the client. I get the response fine.

However, from the second time on, when I send a packet, nothing happens. No Logcat nor anything in the server. I tried printing everything that comes into the server.

So I tried to catch the packets with Wireshark. I saw that on the second packet, this happens.

The packet seems to be coming in, but I don't get anything in the server program running.

I tried running another app after one gets stuck like above, and the second app gets the response fine the first time, then the same thing.

So I guess the server program isn't stuck anywhere or something.

Here's the listening part of my code.

serverSocket = new ServerSocket(port);
while (true) { // socket
    Socket clientSocket = serverSocket.accept();

    out = new DataOutputStream(clientSocket.getOutputStream());
    in = new DataInputStream(clientSocket.getInputStream());

    byte[] read = new byte[512];
    int readcount = in.read(read);
    System.out.println(bytesToHex(read)); // doesn't print from the second one

    if (readcount != -1) {
        if (read[0] == something) {
            HandleMsg doIt = new HandleMsg(connection, out, Arrays.copyOfRange(read, 1, readcount));
            doIt.start();
        }
        else {
            System.out.println("Got Something Unknown!");
            System.out.println(bytesToHex(Arrays.copyOfRange(read, 0, readcount)));
        }
    }
}

connection is a DB connection.

Also, when I restart the app or the connection, it works fine the first time, and get stuck again.

Thank you for your help.


Answer:

You're only doing one read from each accepted socket, so of course the second packet from the same source doesn't get printed. You have to loop.

Question:

I am trying to figure out how to make this piece of code into TCP instead of UDP

DatagramPacket answerDP = null;
answerDP = new DatagramPacket(new byte[110], 110);  

What do I use in TCP instead of DatagramPacket?

The same goes for DatagramSocket, what do I use in TCP instead?

DatagramSocket socket = null;    
socket = new DatagramSocket(); 
socket.send(packet);
socket.setSoTimeout(5000); //wait for answar max. 1 sec.
socket.receive(answerDP);     

Answer:

DatagramSocket in TCP is basically ServerSocket

So for example, to initialise it:

ServerSocket welcomeSocket = new ServerSocket(6789);

And the client socket would be something like:

Socket clientSocket = new Socket("localhost", 6789);

Set timeout works the same

socket.setSoTimeout(5000);

See a simple example here

Question:

I have developing a simple client in java (I use a Windows 7 machine) to communicate with a server. The problem was that the server never understood my request. So I have analysed the communication with Wireshark and have noticed that only one byte is send in a first TCP packet, and 40ms after the remaining bytes are send in a other packet.

In fact, we communicate with binary frames, so all the frames must begin with the total length of the frame on 2 bytes. So it is normal that the server will never understand me. All my frames never exceed 10 bytes, so it's a insignificant amount of data. I know that TCP packets can be segmented, but for me it has no sense to segment a tiny frame after only one byte.

After unsuccessful hours of research, I tried casually to send bytes in a other way using write method instead of writeBytes method. Now I send all the data in only one TCP packet and the communication works fine, but I have never find the explanation. If someone knows, I will be happy to learn it.

Here is my code :

import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;

public class Client {

public static void main(String argv[]) {

    try {
        Socket socket = new Socket();
        socket.connect(new InetSocketAddress("10.2.1.1", 1003), 1000);
        DataOutputStream outToServer = new DataOutputStream(socket.getOutputStream());
        DataInputStream inFromServer = new DataInputStream(socket.getInputStream());

        // Hexadecimal frame to send to server
        String hexFrame = "0004FF9D3175";

        // Build bytes frame
        String bytesFrame = "";
        for (int i=0; i<hexFrame.length()/2; i++) {
            bytesFrame += (char) Integer.parseInt(hexFrame.substring(i*2, (i+1)*2), 16);
        }

        // This generates 2 TCP packets
        // outToServer.writeBytes(bytesFrame);
        // This generates only 1 TCP packet
        outToServer.write(bytesFrame.getBytes());

        // Read answer from server
        hexFrame = "";
        short frame_length = inFromServer.readShort();
        for (int i=0; i<frame_length; i++) {
            hexFrame += String.format("%2s", Integer.toHexString(inFromServer.readUnsignedByte())).replace(" ", "0");
        }

        System.out.println("Receive : " + hexFrame);

        socket.close();
    } catch (IOException e) {
        e.printStackTrace();
    }

}

}


Answer:

Neither Java nor TCP makes any guarantees about this. TCP can segment the data any way it likes, and you have no business relying on any two bytes being delivered consecutively. The problem here is really at the reading end, that makes incorrect assumptions.

In fact, we communicate with binary frames

In fact you are communicating over a byte-stream protocol. No frames, no message boundaries, nothing.

However, if you want a little more control over this you should use a BufferedOutputStream between the DataOutputStream and the socket output stream, and similarly a BufferedInputStream at the receiving end. Flush the stream when you want the data to be sent, typically just before the next read.

Question:

I need to merge two TCP packets into one. I wrote a socket emulator which reads a line of data from a csv file and outputs each line of data into two 99 byte binary packets every second. I now need to write another emulator that will merge these two 99 byte packets into one 198 byte packet.

This is what I put together so far and it basically forwards both 99 byte packets from one emulator and relays it to the client as two 99 byte packets. I have tried a couple different things but cannot seem to figure out how to merge the two into one 198 byte packet. Sounds simple, but i cannot wrap my head around it, suggestions will be greatly appreciated. Thanks

  package PacketFuser;

    import java.io.ByteArrayOutputStream;
    import java.io.DataInputStream;
    import java.io.IOException;
    import java.net.InetAddress;
    import java.net.ServerSocket;
    import java.net.Socket;

    public class PacketFuser {

    public static void main(String args[]) throws IOException{
        //Start server
        Socket socket = null;
        final ServerSocket ss  = new ServerSocket(6666);  
        System.out.println("Waiting on connection...");
        while (ss.isBound()){
            try {
                socket = ss.accept();
                System.out.println("Connected to port: " +socket.toString());
                }
                catch (IOException e){
                }

        //Start Client Socket
        InetAddress address=InetAddress.getLocalHost();
        Socket c1=null;
        boolean client = false;

        while (client == false){
        try{
        System.out.println("waiting on Emulator");
        Thread.sleep(1000);
        c1=new Socket(address, 31982);
        client = true;
        }
        catch (IOException e){} 
        catch (InterruptedException ex) {}
        }
        System.out.println("Emulator Connected");


    //I need to figure out here how to catch two packets and merge them into one 198 byte packets here.

        DataInputStream in = new DataInputStream(s1.getInputStream());
        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
        int pread;
        byte[] p1 = new byte[99];

        while ((pread = in.read(p1, 0, p1.length)) != -1 ) {
        buffer.write(p1, 0, pread);
        buffer.flush();
        socket.getOutputStream().write(p1);
        }

            }
        }
    }

Answer:

Change new byte[99] for new byte[198].

Question:

I am trying to figure out how to push some manually created TCP header data to a byte array. I will be able to use JnetPcap.send(byte[] packet) and see it with wireshark, all I need now is to create the packet as a byte array so I can send it along. I want to do this manually as there are plenty of library functions I could use to create the header.

I simply can't find any explanation on how to format the values or load them to the array. I am also not sure if the values I'm using will be valid. This is what I'm thinking of using...

public static int sourcePort = 1234; //16 bits
public static int destinationPort = 80; //16 bits
public static int sequenceNum = 0; //32 bits - Can this be arbitrary?
public static int ackNum = 0; //32 bits - sequenceNumber + next data start
public static int dataOffset = 5; //4 bits - Minimum value of 5
public static int reserved = 0; //4 bits - Always 0
public static int controlFlags = 0; //8 bits - Not sure if I need any
public static int windowSize = 0; //16 bits Can this be arbitrary?
public static int checkSum = 0; //16 bits - ?use TCP.calculateChecksum() 
public static int urgent = 0; //16 bits

byte[] packet = new byte[160];
//Now load the values into the byte[]

(I am also using JnetPcap built in get ethernet and get Ipv header functions)

UPDATE: I found this snippet which looks like it is the utility I need to put the hex values to byte array:

byte[] pktBytes = FormatUtils.toByteArray("0015c672234c90e6ba92661608004500002d358c4000800600000a000b050a090028c26e270fb8b256e3a2009f785018faf01f550000746573740a");
JMemoryPacket packet = new JMemoryPacket(pktBytes);

So how do I translate my values to go in here. Would it be literally the hex translations appended to each other?

So my 16 bit destinationPort = 80; becomes 0050 ... and 32 bit sequenceNum = 0; becomes 0000 0000 ... 4 bit dataOffset = 5; becomes 5. That seems like it could work, I'm going to give it a try.

(They have 118 hex digits, is that right for TCP header? my values would leave me with 40 hex digits, maybe they have a payload or IP/Ethernet header as well?)


Answer:

The usual way is via a ByteArrayOutputStream wrapped in a DataOutputStream.

Question:

I am researching how to build an Intrusion Prevention System (IPS) application using Java, however I don't know how to redirect Internet packets from a PC to my application and read the packets. The data format I'm looking to get from the packets is similar to the data that Wireshark produces. How would I go about getting low-level packet data like this in a Java application?


Answer:

Unfortunately, Java does not have access to "raw sockets"--the kind of access you need to get the details you'd expect from a Wireshark dump or the similar.

If you need to use Java for the rest of your application, consider using JNI (link is to a good JNI tutorial). It allows you to use C/C++/Fortran code within a Java application, so you can make system calls and get the kind of access you need.

Question:

i've a QT/C++ server and a Java client. The client asks a file to the server, and the server send to the client the stream. The problem is that in the TCP transmission (also in localhost) i lost some packets. Sometimes, the client receive 280705 bytes of 288890.

There is the server :

MyTcpServer::MyTcpServer(QObject *parent) :
    QTcpServer(parent)
{
}

void MyTcpServer::startServer(int port)
{
    if(!this->listen(QHostAddress::Any, serverPort))
    {
        qDebug() << "Could not start server";
    }
    else
    {
        qDebug() << "Listening to port " << serverPort << "...";
    }
}

void MyTcpServer::incomingConnection(qintptr socketDescriptor)
{
    SocketThread *thread = new SocketThread(socketDescriptor, this);
    connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    thread->start();
}

This is the SocketThread:

SocketThread::SocketThread(qintptr ID, QObject *parent) :
QThread(parent)
{
    this->socketDescriptor = ID;
}

void SocketThread::run()
{
    socket = new QTcpSocket();
    if(!socket->setSocketDescriptor(this->socketDescriptor))
    {
        emit error(socket->error());
        return;
    }
    connect(socket, SIGNAL(readyRead()),    this, SLOT(readyRead()), Qt::DirectConnection);
    connect(socket, SIGNAL(disconnected()), this, SLOT(disconnected()));

    socket->write("Welcome to the Server\r\n"); //SEND AN HELLO MESSAGE
    socket->waitForBytesWritten();
    exec();
}

void SocketThread::readyRead()
{       
    QByteArray socketByteArray = socket->readAll();

    int number = 0;
    QDataStream socketDataStream(socketByteArray);
    socketDataStream >> number; //RECEIVE A NUMBER (I WANT 1)
    if (number == 1)
    {
       QFile file("C:\\temp\\test.txt");
       file.open(QIODevice::ReadWrite);

        socket->write(QString("%1\n").arg(file.size()).toStdString().c_str()); //SEND THE FILESIZE AS STRING
        socket->waitForBytesWritten();
        QByteArray buffer = file.readAll();
        long byteSent = socket->write(buffer); //SEND THE FILE

        socket->flush();
        file.close();
    }
    socket->close();
}

void SocketThread::disconnected()
{
    socket->deleteLater();
    exit(0);
}

And this is the JAVA client :

Socket MyClient = null;

boolean connect()
{
  try
  {
    MyClient = new Socket(remoteIP, remotePort);
    MyClient.setSoTimeout(60000);
    if (MyClient != null) {
        inFromServer = new BufferedReader(new InputStreamReader(MyClient.getInputStream()));

        serverWelcomeMessage = inFromServer.readLine(); //RECEIVE THE WELCOME MESSAGE
    }
  }
  catch (IOException e) {
    ...
  }
}

void requestFile()
{
    try {

        FileOutputStream fos = null;

        BufferedOutputStream bos = null;

        DataOutputStream outToServer = new DataOutputStream(MyClient.getOutputStream());

        outToServer.write(encodeIntToByteArray(1)); //SEND THE 1

        outToServer.flush();

        InputStream is = MyClient.getInputStream();

        int remoteFileSize = Integer.parseInt(inFromServer.readLine()); //RECEIVE THE FILESIZE AS STRING

        fos = new FileOutputStream(output);

        bos = new BufferedOutputStream(fos);

        int byteCount = 0;

        int totalByteCount = 0;

        byte[] bytes = new byte[1400];

        while ((byteCount = is.read(bytes)) > 0) {  //RECEIVE THE FILE

            bos.write(bytes, 0, byteCount);

            totalByteCount += byteCount;

        }

        System.out.println("Byte Received "+totalByteCount+" of "+remoteFileSize);

        bos.close();

        fos.close();

        is.close();
    }
catch(...) {

} }

The file test.txt is a file with a number on each line:

0
1
2
3
4
...much numbers...
50000

Sometimes, the client receive the entire file, sometimes receive the file without the first part like this:

60
1860
1861
1862
...much numbers...
50000

Starts from 60, jump to 1860 and ends to 50000.

I try to iterate the request 1000 times, and 90% of times the code works, transfer all data.

Can someone help me to understand ?


Answer:

The problem is in usage of IO streams. You cannot use different instances without proper understanding of side-effects: inFromServer & is. Your exact problem is java.io.BufferedReader#defaultCharBufferSize.

I would recommend you to initialize streams & readers on connection. And use exactly them everywhere across your class.

private Socket socket;
private OutputStream outputStream;
private Writer outputWriter;
private InputStream inputStream;
private Reader inputReader;

public void connect() throws IOException {
    socket = new Socket(..., ...);
    socket.setSoTimeout(60000);
    outputStream = new BufferedOutputStream(socket.getOutputStream()); // Buffered 
    outputWriter = new OutputStreamWriter(outputStream);               // Non-buffered - !important
    inputStream = new BufferedInputStream(socket.getInputStream());    // Buffered
    inputReader = new InputStreamReader(inputStream);                  // Non-buffered - !important
}

And it would be better to use Java naming conventions for Java code.

Question:

I'm currently writing a Java application where I'm running a primary sever along with a backup one (using VMs). In order to get the backup server to take over as the primary (if the primary crashes) while being completely transparent to the client, I need to get the same application running on both the primary and the backup and both must have the same state.

To do this I have to synchronize the backup with the primary so that it receives the same sequence of packets from the client and sends the same (suppressed) replies. For this I need to get the initial TCP sequence number of the backup to match up with the initial sequence number of the primary.

In order to achieve this I thought of intercepting the packets to and from the backup server and modifying the sequence number in their TCP header so it matches up with that of the primary. Thus I need an application that -

  • Lets me intercept TCP packets on the fly
  • Allows me to modify them and then forward them to their original destination
  • Ensures that the original unmodified packet is dropped
  • Something that can be embedded in Java applications is a huge plus but this isn't a major requirement

Is there any way to achieve this?


Answer:

To anyone who's facing the same problem, I finally managed to do this using ettercap - http://ettercap.github.io/ettercap/

More specifically, I used the ARP poisoning mode to intercept the packets to and from the server and used a custom ettercap plugin to modify them appropriately

Question:

I'm trying to write a Java server so that an Android client can send a string to it, and the server would reply with its own string. The first part of this works, where the client sends a string to the server, but the server sending a message to the does not work: the packet makes it out of the server, but the Android client does not pick it up. Does anyone have suggestions on how to fix this?

This entire process worked previously on a Python server, but I am changing to Java because of library support (Java has better support for NAT traversal)

Server (Java):

public class TCPServer implements Runnable {
    private Thread t = null;

    @Override
    public void run() {
        // TODO Auto-generated method stub
        String receive;
        String response = "1||2||3||4\n";
        ServerSocket tcpServer = null;
        Socket tcpClient = null;
        try {
            tcpServer = new ServerSocket(4999);
            System.out.println("   TCP open for connections");
            while(true) {
                tcpClient = tcpServer.accept();
                BufferedReader inStream = new BufferedReader(new InputStreamReader(tcpClient.getInputStream()));
                receive = inStream.readLine();
                System.out.println("Server <<< " + receive);

                DataOutputStream outStream = new DataOutputStream(tcpClient.getOutputStream());
                outStream.write(response.getBytes("UTF-8"));
                outStream.flush();
                System.out.println("Server >>> " + response);
            }
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
            System.out.println("IOException: " + e.toString());
        } finally {
            if (tcpServer != null) {
                try {
                    tcpServer.close();
                } catch (IOException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                    System.out.println("IOException: " + e.toString());
                }
            }
            if (tcpClient != null) {
                try {
                    tcpClient.close();
                } catch (IOException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                    System.out.println("IOException: " + e.toString());
                }
            }
        }
    }

    public void start() {
        // TODO Auto-generated method stub
        if(t == null) {
            t = new Thread(this);
            t.start();
            System.out.println("Start TCP Server");
        }

    }
}

Client (Android):

public class AsyncTCPSend extends AsyncTask<Void, Void, Void> {
            String address = "";
            String message = "";
            String response = "";
            AsyncTCPSend(String addr, String mes) {
                address = addr;
                message = mes + "\n";
            }

            @Override
            protected Void doInBackground(Void... params) {
                Socket socket = null;
                try {
                    socket = new Socket(address, 4999);
                    socket.getOutputStream().write(message.getBytes());

                    ByteArrayOutputStream writeBuffer = new ByteArrayOutputStream(1024);
                    byte[] buffer = new byte[1024];

                    int bytesRead;
                    InputStream writeIn = socket.getInputStream();

                    while((bytesRead = writeIn.read(buffer)) != -1) {
                        writeBuffer.write(buffer,0,bytesRead);
                        response += writeBuffer.toString("UTF-8");
                    }
                } catch (UnknownHostException e){
                    e.printStackTrace();
                    response = "Unknown HostException: " + e.toString();
                    System.out.println(response);
                } catch (IOException e) {
                    response = "IOException: " + e.toString();
                    System.out.println(response);
                } finally {
                    if (socket != null) {
                        try {
                            socket.close();
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    }
                }
                return null;
            }

            @Override
            protected void onPostExecute(Void result) {
                recieve.setText(response);
                super.onPostExecute(result);
            }
        }

Answer:

I don't know why you would call an InputStream 'writeIn', but the problem is that the client is reading the socket until end of stream, and the server is never closing the accepted socket, so end of stream never occurs.

Question:

Is it possible to access the sequence number of an incoming TCP-packet in netty? I want to uniquely identify each packet, even with the same payload.


Answer:

It's not possible in netty as it is not exposed in java.nio.* classes and we also not expose it in our JNI transport.

Question:

I have read all around the Stack, but I cannot find a working solution to determine the packet size and set the array of bytes to match that length to receive a correct amount of bytes for one message at once.

DataInputStream in = new DataInputStream(socket.getInputStream());
while(true) {
    short myShortStreamSize = in.readShort();
    byte[] payload = new byte[myShortStreamSize];
    in.readFully(payload);
    Log.i("Foo", new String(payload));
}

That doesn't work for me.

My TLS server is made with Node.js. Normally Node.js TLS module handles the packets in the way if I use the same module to connect to the server it's enough to use socket.write("message") and the Node.js client understands the sizes already so you don't need to define it yourself. It will write that message to the console without issues. As I mentioned that my server is TLS. I'm not sure if these have something to do with my issue, but would be awesome to make it work.

My goal is to receive one message at once without cutting it into pieces. If I send "Good morning, today is a sunny day!" I want my Java/Android client to read it from start to end without mixing it up with other packets.


Answer:

... size of TLS packet ... determine the packet size ... receive one message at once

There is no "TLS packet". There is a TLS record which has the length in the first 2 bytes. This TLS record might be split into multiple TCP fragments or TCP fragments might also contain multiple TLS records in full, in half or whatever. These TCP fragments then might even be cut into multiple IP packets although TCP tries hard to avoid this.

And your plain text "message" itself might end up being spread over multiple TLS records since the size of such a record might be smaller than the message. Depending on your application and TLS stack it might also be mixed together with other "messages".

In other words: what you want is impossible. Unless you somehow know already the size of your message before reading it you cannot rely on reading the full message and also only this message. Neither TCP nor TLS preserve any message "boundaries" you might have imagined when crafting your writes but which have no actual representation in the payload.