Hot questions for Using Transmission Control Protocol in logstash

Question:

I'm running into some issues sending log data to my logstash instance from a simple java application. For my use case, I'm trying to avoid using log4j logback and instead batch json events on separate lines through a raw tcp socket. The reason for that is I'm looking to send data through a aws lambda function to logstash which means storing logs to disk may not work out.

My logstash configuration file looks like the following:

input {
  tcp {
        port => 5400
        codec => json
  }
}
filter{
  json{
    source => "message"
  }
}
output {
   elasticsearch {
      host => "localhost"
      protocol => "http"
      index => "someIndex"
   }
}

My java code right now is just opening a tcp socket to the logstash server and sending an event directly.

Socket socket = new Socket("logstash-hostname", 5400);
DataOutputStream os = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
os.writeBytes("{\"message\": {\"someField\":\"someValue\"} }");
os.flush();
socket.close();

The application is connecting to the logstash host properly (if logstash is not up an exception is thrown when connecting), but no events are showing up in our ES cluster. Any ideas on how to do this are greatly appreciated!

I don't see any relevant logs in logstash.err, logstash.log, or logstash.stdout pointing to what may be going wrong.


Answer:

The problem is that your data is already deserialized on your input and you are trying to deserialize it again on your filter. Simply remove the json filter.

Here is how I recreated your scenario:

# the json input
root@monitoring:~# cat tmp.json 
{"message":{"someField":"someValue"}}


# the logstash configuration file
root@monitoring:~# cat /etc/logstash/conf.d/test.conf
input {
  tcp {
    port => 5400
    codec => json
  }
}

filter{
}

output {
   stdout {
     codec => rubydebug
   }
}


# starting the logstash server
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf


# sending the json to logstash with netcat
nc localhost 5400 <  tmp.json

# the logstash output via stdout
{
       "message" => {
        "someField" => "someValue"
    },
      "@version" => "1",
    "@timestamp" => "2016-02-02T13:31:18.703Z",
          "host" => "0:0:0:0:0:0:0:1",
          "port" => 56812
}

Hope it helps,

Question:

I am sending TCP String messages from Java to Logstash. I do not want to create a new TCP connection for every message but I want to reuse TCP connection. Is it safe to keep one TCP connection and send messages via Socket.write and Socket.flush until one of the method throws IOException and then open new connection?


Answer:

We have been running this in production for over a year and so far no problems.

Question:

The code below runs in a thread, takes messages off of a queue, and sends them to logstash. I've a unit test that proves that the json (which is the content of the messages) makes it into the socket and gets to the other end.

However, when I point it at actual logstash, nothing ever turns up. I can 'nc' the same content and it shows up. Of course, nc will close the connection at the end of sending the one message. There's a bit of complexity here to reconnect if we ever lose the connection.

The logstash config is pretty simple:

input {
 tcp {
     port => 9191
     codec => json
     }
}
output {
    stdout {
       codec => rubydebug
    }
}


         while (true) {
                byte[] message = messages.take();
                executor.getWithRetry(() -> {
                    try {
                        if (socket == null) {
                            socket = new Socket(sinkAddress, port);
                        }
                        socket.getOutputStream().write(message);
                        return null;
                    } catch (IOException e) {
                        socket.close();
                        socket = null;
                        throw e;
                    }
                }).join();
            }

Answer:

Make sure you are sending a newline character at the end of the message. I found that sending "\n" works fine even from a windows box to logstash running on linux.