Hot questions for Using Transmission Control Protocol in bytearray

Question:

I have one client connected to a server. The communication in the TCP session is bidirectional. If I try to send multiple messages simultaneously from the sever to the client they are mixed up in one array. The message is already corrupted when it reaches the ByteArrayDecoder. Here is my pipeline:

    ChannelPipeline channelPipeline = ch.pipeline();

    channelPipeline.addLast("byteArrayEncoder", new ByteArrayEncoder());
    channelPipeline.addLast("myRequestEncoder", new MyRequestEncoder());
    channelPipeline.addLast("mySecondEncoder", new MySecondEncoder());

    channelPipeline.addLast("byteArraydDecoder", new ByteArrayDecoder());
    channelPipeline.addLast("myResponseDecoder", new MyResponseDecoder());
    channelPipeline.addLast("mySecondDecoder", new MySecondDecoder());

    channelPipeline.addLast("mytHandler", myHandler);

For example I have noticed that the byte array in ByteArrayDecoder has length more than 500 instead of 230. And I have seen that the contents of the array are corresponding to two or more messages that have been concatenated.

I have only one channel. I have tried to use more than one EventExecutorGroup in the pipeline but this keeps happening and I end up to miss several inbound messages.

However with 1tps everything seems to work OK.

Is netty supposed to have such behavior? Am I missing anything?


Answer:

TCP is a stream based protocol, you application should work from that concept. Netty also receives the bytes as a stream from the networking layer, and its the job of the application to convert the stream of bytes in something useful.

You are failing to correctly convert the incoming streaming bytes in correct blocks of data that your application can handle.

There are multiple ways to introduce a "reframer" for the bytes:

Make every packet a fixed length

If every packet is a fixed length, you can just read X amount of bytes, and use that as a simple framing solution, this is really simple using netty:

channelPipeline.addLast("framer", new FixedLengthFrameDecoder(512));

This splits all the incoming bytes up in blocks of 512 bytes.

The arguments to FixedLengthFrameDecoder are:

  • Length of the incoming packets
Prepend the length of the packet before sending

Instead of difficulty making every packet the same size, you can also add a length to a packet, so you can split the packets later. Netty also comes with useful utility classes for this:

channelPipeline.addLast("framer", new LengthFieldBasedFrameDecoder(Short.MAX_VALUE,0,2,0,2));
channelPipeline.addLast("framer-prepender", new LengthFieldPrepender(2, false));

The arguments for LengthFieldBasedFrameDecoder are:

  • Max frame length
  • Offset in receiving data for frame length
  • Length in bytes of our length field
  • Bytes to strip from the received message

The arguments for LengthFieldPrepender are:

  • Length in bytes of our length field

Question:

I have a Java application that is reading data from a TCP socket that is receiving XML of varying size. The first 5 bytes of a given packet are supposed to indicate the size of the remaining message. I can read the message and xml successfully if I manually create a large byte[] and read the data.

Here are the instructions from the manual for the application that is generating the data:

Each message is preceded by the message size indicator which is a 32-bit unsinged integer using the network bytes order method. For example: \x05\x00\x00\x00\x30\x31\x30\x32\x00 indicates the message size of an ack which is 5 bytes included the fifth message byte '\0'. The size indicator specifies everything following the size indicator itself.

However I can't figure out how to decode the first 5 bytes to an integer that I can use to correctly size a byte[] for reading the rest of the message. I get random results:

Here is the code I'm using to parse the message:

DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedInputStream inFromServer = new BufferedInputStream(clientSocket.getInputStream());

byte[] data = new byte[10];
inFromServer.read(data);
String result = new String(data, "ISO-8859-1");

Logger.info(data+"");

//PROBLEM AREA: Tried reading different byte lengths but no joy
//This should be a number but it never is. Often strange symbols
byte[] numeric = Arrays.copyOfRange(data,1,5);
String numericString = new String(numeric, "ISO-8859-1");

//Create a huge array to make sure everything gets captured. 
//Want to use the parsed value from the start here
byte[] message = new byte[1000000];
inFromServer.read(message);

//This works as expected and returns correctly formatted XML
String fullMessage = new String(message, "ISO-8859-1");

Logger.info("Result "+result+ " Full message "+fullMessage);

Answer:

The length looks like it's little endian. You can still use DataInputStream but you have to swap the bytes. If you used NIO's SocketChannel and a ByteBuffer you could set the byte order, but this is likely to be harder to use.

// only do this once per socket.
DataInputStream in = new DataInputStream(
                                  new BufferedInputStream(clientSocket.getInputStream()));

// for each message.
int len0 = in.readInt();
int len = Integer.reverseBytes(len0);
assert len < 1 << 24;

byte[] bytes = new byte[len];
in.readFully(bytes);

String text = new String(bytes, "ISO-8859-1").trim();
int number = Integer.parseInt(text);

Question:

Currently I'm trying to communicate between from Client to Server and vice versa in Java.

So as of now, I tried with the following

Attempt 1) Trying to send through string in bytearray form.

Client

OutputStream os = clientSock.getOutputStream();
byte[] sndMsg = new String("test").getBytes();
os.write(sndMsg);
os.flush();

Server

InputStream is = serverChild.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));

String rcvRequest = br.readLine();
System.out.println(rcvRequest);

Apparently when it does not print out test.

Attempt 2) However, when I change the code from Client side to

PrintWriter out = new PrintWriter(new OutputStreamWriter(clientSock.getOutputStream()));
out.println("test");
out.flush();

It works. Why is this so? Is it any way I can use attempt 1 to try and send byte array over and receive it as a string in the server side? Thank you.


Answer:

BufferedReader.readLine waits for NewLine character, you should add it to "test" in attempt 1. println in attempt 2 adds it automatically

Question:


Answer:

You're misreading the file length in the receiver. You are getting zero, so you're constructing a zero length byte array, so read() returns zero.

You need to send the length via DataOutputStream.writeLong() and read it via DataInputStream.readLong(). And then your sending and receiving code is all wrong as well. See my answer here for complete code.