Hot questions for Using Transmission Control Protocol in json

Question:

I am trying to pass a json from C# server to Java server using TCP, problem is that the first time Java server seems to receive an empty json. Second time and onwards it works fine, see output bellow. Any ideas or suggestions are welcome, thanks in advance.

Output:

Starting server...
Waiting for a connection..
Connected!
Reading...
Received empty object
Received: 
Connected!
Reading...
Received: "Request:gethotellist"
Connected!
Reading...
Received: "Request:gethotellist"

Here is the C# code snippet for sending json:

public void GetHotelList()
{
TcpClient clientSocket = new TcpClient();
clientSocket.Connect("127.0.0.1", 6767);

NetworkStream ns = clientSocket.GetStream();

string jsonRequest = "Request:gethotellist";

string jsonToSend = JsonConvert.SerializeObject(jsonRequest);

byte[] dataBytes = Encoding.UTF8.GetBytes(jsonToSend);

ns.Write(dataBytes, 0, dataBytes.Length);

ns.Close();
}

Java server:

public class JHotelServer
{

public static void main(String[] args) throws IOException
{
  final int PORT = 6767;

  System.out.println("Starting server...");     

  @SuppressWarnings("resource")
  ServerSocket welcomeSocket = new ServerSocket(PORT);  

  System.out.println("Waiting for a connection..");

  while(true)
  {             
     try
     {
        Socket connectionSocket = welcomeSocket.accept();  

        System.out.println("Connected!");

        Thread connectionThread = new Thread(new TcpConnectionManager(connectionSocket));
        connectionThread.start();
     }

     catch(IOException ioe)
     {
        ioe.printStackTrace();
     }         
  }
}
}

And here is the Tcp Communication manager:

public class TcpConnectionManager implements Runnable
{   
private DataInputStream inFromDotNet;

public TcpConnectionManager(Socket socket) throws IOException
{
  inFromDotNet = new DataInputStream(socket.getInputStream());
}

@Override
public void run()
{
  try
  {               
     System.out.println("Reading...");

     byte[] rvdMsgByte = new byte[inFromDotNet.available()];

     // Collecting data into byte array
     for (int i = 0; i < rvdMsgByte.length; i++)
     {
         rvdMsgByte[i] = inFromDotNet.readByte();
     }

     if (rvdMsgByte.length == 0)
     {
        System.out.println("Received empty object");
     }

     // Converting collected data in byte array into String.
     String rvdMsgTxt = new String(rvdMsgByte);

     System.out.println("Received: " + rvdMsgTxt);       
  }

  catch(IOException ioe)
  {
     ioe.printStackTrace();
  }
}
}

Answer:

Based on the code shown, it would be entirely expected to sometimes get an empty payload, because if the payload packets haven't arrived yet: inFromDotNet.available() will be zero.

Basically, .available() (and equivalent) should never be used to determine anything about the content, other than "there are bytes currently buffered" (perhaps to choose between sync read and async read).

There are two approaches commonly used here:

  • use EOF to indicate the end of the payload, i.e. read until the socket says it is closed and everything has been read; how you detect this depends on the specific socket API (for example, in .NET, it would be when Receive returns a non-positive number)
  • implement some kind of basic framing protocol

The first option is relevant if you will only ever send one message per socket; the second option is necessary if you will be sending multiple messages per socket. A basic framing protocol could be as simple as "messages are separated by newline characters", but in many cases you may need binary-safe framing, such as a length prefix. For example, your JSON could contain newlines, which could incorrectly be interpreted as the end of a frame.

In either case: with sockets you almost always need to read in a loop, because the data can be split over multiple packets, and/or your receive buffer size may be too small. So usually, you would loop and buffer all the payload until you detect either an EOF or the end of a frame, and only then start trying to process the content. In particular, you should not usually try to decode text until you know you have the entire thing, because multi-byte characters may span multiple "read"/"receive" calls, so: unless you're using a stateful text decoder, you may incorrectly decode such characters.

The behaviour you're seeing would be common to almost all platforms and languages; it isn't specific to Java/C#.

Question:

I need help troubleshooting the following. I got a really simple program sending TCP command that succesfully sends json data to my Yeelight RGB light from my laptop, but not from the raspberry pi.

import java.io.*;
import java.net.*;

class TCPClient
{
        public static void main(String args[]) throws Exception
        {
                String modifiedSentence;
                Socket clientSocket = new Socket(args[0], 55443);
                DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());
                PrintWriter printWriterw = new PrintWriter(outToServer);
                BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
                String command = "{\"id\":1,\"method\":\"set_rgb\",\"params\":[13631232, \"smooth\", 500]}";
                printWriterw.println(command);
                printWriterw.flush();
                modifiedSentence = inFromServer.readLine();
                System.out.println("FROM SERVER: " + modifiedSentence);
                clientSocket.close();
        }
}

I also have a program that acts as a TCPServer on my laptop so I can see if I can send and receive the same message to my laptop ip.

Rpi, Laptop, Yeelight -> all in same network

  • run above program on laptop to Yeelight IPadress -> success
  • run above program on RPI to Yeelight IPadress -> keeps waiting for response, timeout.
  • run above program on laptop to Yeelight IPadress -> success
  • run above program on rpi to laptop IPadress -> success (just check if json arrives)
  • run above program on laptop to laptop localhost -> success receiving message is exactly the same.
  • using telnet on the rpi to send the json to the yeelight ip + port -> success...

Im really lost on why the program is not working when running from my RPI.

Hope someone can help.


Answer:

I believe the problem lies in the println function. The open API for yeelight states that the end of commands must have \r\n. This is default on a windows machine (which I suppose you are using on your laptop), yet the raspberry pi is probably linux, which does newlines with \n only. Instead of println(), manually append \r\n to the command. I believe this will fix your issue.

For reference:

Documentation for yeelight API is found here: http://www.yeelight.com/download/Yeelight_Inter-Operation_Spec.pdf

On page 8 it states " All messages must be delivered in defined JSON format on a TCP connection and each individual message must be terminated by "\r\n"."

Question:

I currently have a TCP server (java desktop application) and client (android application) connected.

I can send json string from client to server and from server to client, my issue is sometimes i need to send from the server a large string which gets cut.

This is my code:

SERVER SIDE

public void sendResponse(String response){
  PrintWriter writer = new PrintWriter(BufferedWriter(
                       new OutputStreamWriter(mClientSocket.getOutputStream())), true);

  if(writer != null && !writer.checkError()){
     writer.println(response);
     writer.flush();
  }
}

CLIENT SIDE

BufferedReader input = new BufferedReader(new InputStreamReader(mSocket.getInputStream()));
String response;
while((response = input.readLine()) != null && mRun){
    Data data = new Gson().fromJson(response, Data.class);
    //when i try to convert the response to Data object using Gson
    //it gives me a com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated array at line 1 column 106498

}

as you can see the error on client side happens because the string was to large so it got cut off, can someone please tell me what to do in this case?

thanks in advance.


Answer:

I suggest you use a DataOutputStream instead of trying to read lines. Something like,

public void sendResponse(String response) {
    OutputStream os = mClientSocket.getOutputStream();
    DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(os));
    dos.writeUTF(response);
    dos.flush();
}

Then, on the client

InputStream is = mSocket.getInputStream();
DataInputStream dis = new DataInputStream(is);
String response = dis.readUTF(); // <-- not just one line. the entire string.
Data data = new Gson().fromJson(response, Data.class);

And, if it's a really large JSON response you might consider adding a GZIPInputStream and GZIPOutputStream.

DataInputStream dis = new DataInputStream(new GZIPInputStream(is));

and

DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os));

Edit

public void sendResponse(String response) {
    OutputStream os = mClientSocket.getOutputStream();
    DataOutputStream dos = new DataOutputStream(new GZIPOutputStream(os));
    byte[] buff = response.getBytes("UTF-8");
    dos.writeInt(buff.length);
    dos.write(buff);
    dos.flush();
}

And then to read,

InputStream is = mSocket.getInputStream();
DataInputStream dis = new DataInputStream(new GZIPInputStream(is));
int len = dis.readInt();
byte[] buff = new byte[len];
dis.readFully(buff);
String response = new String(buff, "UTF-8");
Data data = new Gson().fromJson(response, Data.class);

Question:

Hi let me get straight to the problem. I have a big JSON packet that the server sends to this client once the client is authenticated

But the packet comes back in a weird way like it's split or something example:

The JSON should be:

Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}

When it comes through:

Received: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,
Received: "ItemSlots":30}

Why does it split the packet or something when it comes to the client and how can I fix this anyway?

Java Receive Code:

   private class ClientThread extends Thread {
        public void run() {
            try {
                while (selector.select() > 0) {
                    for (SelectionKey sk : selector.selectedKeys()) {

                        selector.selectedKeys().remove(sk);
                        if (sk.isReadable()) {

                            SocketChannel sc = (SocketChannel)sk.channel();
                            ByteBuffer buff = ByteBuffer.allocate(1024);
                            String content = "";

                            while (sc.read(buff) > 0) {
                                sc.read(buff);
                                buff.flip();
                                content += charset.decode(buff);
                                buff.clear();
                            }

                            System.out.println("Recieved: " + content);
                            sk.interestOps(SelectionKey.OP_READ);
                        }
                    }
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

Thanks have a wonderful day.


Answer:

Hi lemme get straight to the problem so i got a big JSON packet that the server sends to this client once the client is authenticated

You mean you have a big JSON message. Packets are things that network protocols used to exchange information.

But the packet comes back in a weird way like its split or something example:

Unless you're looking at the wire, you aren't looking at packets. You're looking at the bytes you got from your end of the TCP connection.

The JSON should be:

Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0,"ItemSlots":30}

When it comes through:

Recieved: {"UserID":1,"PlayerID":2,"EXP":0,"Lvl":1,"Coins":0, Recieved: "ItemSlots":30}

Excellent. You got the same bytes. Now make a JSON parser that figures out where the message ends and parses it.

Why does it split the packet or something when it comes to the client

It splits the message into packets because that's how TCP gets the message to the other side. TCP is not a message protocol and it doesn't know or care what the application considers to be a message -- that's the application's job.

and how i can i fix this anyway?

Write a JSON parser to figure out where the messages end. You haven't implemented any code to receive JSON over TCP yet, so that won't work until you do.

TL;DR: If you want an application-level message protocol, you need to implement one. TCP is not one.