Hot questions for Using Transmission Control Protocol in http

Question:

I am hosting a simple PHP echo server locally. I am trying to send a message to the server in Java, and use a GET request to print the response but am getting a 'malformed HTTP request' error. Can anyone tell me how to correctly format the GET request?

//Client code:

import java.io.*;
import java.net.*;

public class TCPclient {

    public static void main(String argv[]) throws Exception {
    String sentence, modifiedSentence;
    BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in));
    Socket clientSocket = new Socket("localhost", 8000);  
    DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());

    BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
            sentence = inFromUser.readLine();

            outToServer.writeBytes(sentence + "\n");
            outToServer.writeChars("GET /echo.php HTTP/1.1" +"\n"); 


            modifiedSentence = inFromServer.readLine();
            System.out.println("FROM SERVER: " + modifiedSentence);
            inFromServer.close();
            outToServer.close();
            inFromUser.close();
            clientSocket.close();
            }

}

//PHP Server code:

<?php

/* Simple php echo page 
*/

//      ini_set('display_errors', 'On');
//      error_reporting(E_ALL | E_STRICT);
      if(isset($_GET['source'])) {
        if ($_GET['source'] == "raw")
           echo file_get_contents(basename($_SERVER['PHP_SELF']));
        else
           echo "<pre>" . htmlspecialchars(file_get_contents(basename($_SERVER['PHP_SELF']))) . "</pre>";
      } else if (isset($_GET['message'])){
        echo strtoupper( htmlspecialchars($_GET['message'])) . '\n';
      } else {
?>
      <html>
        <head>
          <title>Error: No input message</title>
        </head>
        <body>
          <h1> No message</h1>
          <p>Echo server called without sending parameter</p>
        </body>
      </html>
<?php
      }
?>

Answer:

An http server understand words like GET, POST, PUT, DELETE. So here's a problem:

        outToServer.writeBytes(sentence + "\n");
        outToServer.writeChars("GET /echo.php HTTP/1.1" +"\n"); 

The first statement there is telling something to the http server that it doesn't understand: the sentence. Comment out that line and your request becomes valid, and you will get a response.

I guess you want to post the sentence as a parameter to the echo service. You can put it in the query string:

        outToServer.writeChars("GET /echo.php?message=" + sentence + " HTTP/1.0\n\n"); 

I also changed the HTTP version and appended an extra \n as pointed out by the comment of Steffen Ullrich:

Also you must add empty line to mark the end of the header. And the line ending must be \r\n not \n. And you should better do a HTTP/1.0 request not HTTP/1.1 since your code is neither able to deal with HTTP keep-alive nor with HTTP chunked encoding

However, this is not quite enough. You also need to encode some characters, such as spaces. The URL in the get request must be encoded, as pointed out by @zapl in a comment.

I also recommend to test your echo service first using simple telnet:

telnet localhost 8000

There, you can type your message, "GET /echo.php?message=something" and verify it works. Once you find something that works as intended, you can update the Java code accordingly.

Question:

My application uses Tor successfully, but I would like to send a NEWNYM request to obtain a new identity. I could kill and restart the process, or possibly use a shell script, but I'd prefer to do it in native Java.

There is a Python solution, but I'm not quite sure how to map it to Java connection types. I imagine it's fairly similar to my Tor HTTP code, but I'm not familiar enough with Java connection types to make it happen.

// My starter code.
public static String torNewIP() throws Exception {
   Proxy proxy = new Proxy(Proxy.Type.HTTP, 
              new InetSocketAddress(TOR_IP, TOR_CONTROL_PORT));     
   // This is how I open a web page using Tor, but the control
   // port is probably not HTTP.
   // HttpURLConnection connection = (HttpURLConnection) url.openConnection(proxy);
   // connection.connect();
   // Something like connection.send("NEWNYM");
}   

Answer:

Double check to make sure Tor is run with the option ControlPort [TOR_CONTROL_PORT]

public static void torNewIP() throws Exception {
   Socket socket = new Socket();
   socket.connect(new InetSocketAddress(TOR_IP, TOR_CONTROL_PORT));
   socket.getOutputStream().write(new String("SIGNAL NEWNYM\r\n").getBytes());
   socket.close();
}

Not entirely sure why, but it may take some time to function as expected - perhaps the 10 minute IP rotation. Sorry I can't provide better details, but it worked for me.

Question:

I'm running a very simplistic Undertow configuration. Whole code included:

public class HelloWorldUndertow {
    public static void main(String[] args) {
        Undertow undertow = Undertow.builder()
                .addHttpListener(9090, "0.0.0.0")
                .setHandler(new HttpHandler() {
                    @Override
                    public void handleRequest(HttpServerExchange exchange) throws Exception {
                        exchange.setResponseCode(200);
                        exchange.getResponseSender().send("hello!");
                    }
                }).build();
        undertow.start();
    }
}

I try to test how many requests per second it can handle. I'm using apache ab tool for measurements:

ab -n 100000 -c 10 http://localhost:9090/test

The biggest number I managed to get is around 10000 3/sec:

Concurrency Level:      10
    Time taken for tests:   10.664 seconds
    Complete requests:      100000
    Failed requests:        0
    Total transferred:      10000000 bytes
    HTML transferred:       600000 bytes
    Requests per second:    9377.69 [#/sec] (mean)
    Time per request:       1.066 [ms] (mean)
    Time per request:       0.107 [ms] (mean, across all concurrent requests)
    Transfer rate:          915.79 [Kbytes/sec] received

    Connection Times (ms)
    min  mean[+/-sd] median   max
    Connect:        0    0   0.2      0       1
    Processing:     0    1   0.5      1      11
    Waiting:        0    1   0.5      1      11
    Total:          0    1   0.5      1      11

    Percentage of the requests served within a certain time (ms)
    50%      1
    66%      1
    75%      1
    80%      1
    90%      2
    95%      2
    98%      2
    99%      2
    100%     11 (longest request)

I get that it is BIG, but I'm curoius what is a limiting factor in this case. It's not processor (the app is running below 20% when sampled using Java Mission Control) and not memory. I'm running it on Windows - maybe that is the reason?


Answer:

I can't tell you how to make it perform better when running in Windows, but if you run it in Linux you can do the tuning recommended for Jetty here: http://www.eclipse.org/jetty/documentation/current/high-load.html In summary:

sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"
sysctl -w net.core.somaxconn=4096
sysctl -w net.core.netdev_max_backlog=16384
sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sysctl -w net.ipv4.tcp_syncookies=1
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w net.ipv4.tcp_tw_recycle=1
sysctl -w net.ipv4.tcp_congestion_control=cubic

Include the -server JVM param when you run your code.

Then try: ab -n 100000 -c 100 http://localhost:9090/test

(make 100,000 requests using 100 connections)

When I try that in a Linux vm running in VMWare Player on my Windows7 laptop (3 years old) I get over 100,000 requests per second.

Question:

This is an assignment for a course we are having and i need some help.

I am having problems for example, trying to request a file that does not exist, it works that a 404 file not found page comes up, but when i look in the web tool for Safari i can see that the response code is 200, OK, which is definialty wrong, it should be the code that is the error.

But why i don't see, i send the error code header when a error occurs, but it´still doesn't work. Can somebody point me at the right direction or maybe just say what the problem is and i can fix it :D ?

Main:

import java.io.File;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketAddress;

public class WebServer {

    private static int PORT = 8888;
    private static String ROOT_DIR = "";

    public static void main(String[] args) {


        if (isCorrect(args) == true) {

            boolean isRunning = true;

            try {

            /* Creates a new server socket */
                ServerSocket serverSocket = new ServerSocket();

            /* Binds the port to the server */
                SocketAddress localBindPoint = new InetSocketAddress(PORT);

                serverSocket.bind(localBindPoint);

                System.out.println("==============================================" +
                        "\n| HTTP Web Server |" +
                        "\n===================" +
                        "\n| Configuration: " +
                        "\n| Directory: " +
                        "\n| " + ROOT_DIR +
                        "\n| Port: " +
                        "\n| " + PORT +
                        "\n| Usage: <directory> <port>" +
                        "\n| ctrl-c to exit" +
                        "\n==============================================");
            /* The server is running */
                while (isRunning) {

                    try {

                /* Accept connection by client */
                        Socket socket = serverSocket.accept();

                /* Each connected client gets a new thread */
                        new Thread(new RequestHandler(socket, ROOT_DIR)).start();

                    } catch (IOException e) {
                        System.err.println(e.getMessage());
                    }
                }
            } catch (IOException e) {
                System.err.println("Address already in use!" +
                        "\nClose running connection or choose other port");
            }
        } else
            usageMsg();
        System.exit(1);
    }

    public static boolean isDirectory(String path){
        File filePath = null;
        try{

            filePath = new File(path);
            /* False if file is not a directory */
            if (!filePath.isDirectory())
                return false;

        }
        catch (Exception e){
            System.err.println(e.getMessage());
        }
        /* Seems to be a file path */
        return true;
    }

    public static boolean isCorrect(String[] args){

        if (args.length != 2){
            usageMsg();
            return false;
        }

        try{
            ROOT_DIR = args[0].toString();
            PORT = Integer.parseInt(args[1]);
        }
        catch (NumberFormatException n){
            System.err.println(n.getMessage());
        }

        if (!isDirectory(ROOT_DIR)){
            usageMsg();
            return false;
        }

        return true;
    }

    public static void usageMsg(){
        System.err.println("Invalid arguments"+
                "\nUsage: java -jar Webserver.jar <directory> <port>");
    }
}

RequestHandler:

import java.io.*;
import java.net.Socket;
import java.util.StringTokenizer;

/**
 * Web Server Request Handler.
 * Created on 2015-02-16.
 */

public class RequestHandler implements Runnable {

    /*
    TODO ( ) Problem 1
    TODO ( ) Problem 2
    TODO ( ) Problem 3
    TODO (X) Index page for first page.
    TODO (X) Read and download images & other files
    TODO ( ) Fix header responses
    TODO ( ) Error responses
     */

    private String
            OK = "HTTP/1.0 200 OK",
            NOT_FOUND = "HTTP/1.0 404 Not Found",
            BAD_REQUEST = "HTTP/1.0 400 Bad Request",
            FORBIDDEN = "HTTP/1.0 403 Forbidden",
            SERVER_ERROR = "HTTP/1.0 500 Internal Server Error";

    private String ROOT_DIR;
    private Socket client;
    private PrintStream send;
    private DataInputStream fromClient;
    private DataOutputStream out;

    RequestHandler(Socket client, String ROOT_DIR) {
        this.client = client;
        this.ROOT_DIR = ROOT_DIR;

        try {
            send = new PrintStream(client.getOutputStream());
            fromClient = new DataInputStream(client.getInputStream());
            out = new DataOutputStream(new BufferedOutputStream(client.getOutputStream()));


        } catch (IOException e) {
            System.err.println(e.getMessage());
        }
    }

    /* Reads the HTTP request and responds */
    public void run() {
        String request = null;
        String fileName = null;
        StringTokenizer tok = null;

        try {

            /* Read HTTP request from client */
            while ((request = fromClient.readLine()) != null) {

                System.out.println(request);
                tok = new StringTokenizer(request);

            /* Extracts the file path from the GET command */
                if (tok.hasMoreElements() && tok.nextToken().equals("GET")
                        && tok.hasMoreElements()) {

                    fileName = tok.nextToken();
                } else
                    throw new FileNotFoundException();

                /* */
                if (fileName.endsWith("/"))
                    fileName += "index.html";

                /* Illegal characters, prevent access to super directories */
                if (fileName.indexOf("..") >= 0 || fileName.indexOf('|') >= 0
                        || fileName.indexOf(':') >= 0 || fileName.indexOf('~') >= 0) {

                    error(FORBIDDEN, "Forbidden Access", fileName);
                }
                else

                if (new File(fileName).isDirectory()) {
                    fileName = fileName.replace('\\', '/');
                    send.close();
                    return;
                }

            /* File name is ROOT_DIR + file name */
                fileName = ROOT_DIR + fileName;

            /* Create file */
                File file = new File(fileName);

                if (file.isDirectory()) {
                    fileName = fileName + "index.html";
                }
            /* File does not exist */
                if (file.exists()){
            /* Determines the MIME type of the file */
                    String mimeType = getMimeType(file);

            /* Sends the file */
                    sendFile(file, mimeType, fileName);
                    client.close();
                }
                else
                    error(NOT_FOUND, "404 File Not Found", fileName);
            }

        }
        catch (FileNotFoundException e) {
            System.err.println(e.getMessage());
        }
        catch (IOException e){
            System.err.println(e.getMessage());
        }
    }

    /* Sends the requested file to the client */
    public void sendFile(File file, String fileType, String fileName) {
        try {
            // Buffer must not be to low, => fragments
            int length = (int) file.length();
            FileInputStream fileIn = new FileInputStream(fileName);
            byte[] bytes = new byte[length];

            /* Write until bytes is empty */
            while ((length = fileIn.read(bytes)) != -1 ){
                out.write(bytes, 0, length);
                out.flush();
                out.writeBytes(OK);
                out.writeBytes("Server: Jakobs Web Server v1.0");
                out.writeBytes("Content-Type: " + fileType + "\r\n");
                out.writeBytes("Content-Length: " + length + "\r\n");
                out.writeBytes("");
            }
            send.close();
        } catch (IOException e) {
            System.err.println(e.getMessage());
        }
    }

    /* Sends the header response to the client */
    public void sendHeaderResponse(String code, String fileType){
        try {
            out.writeBytes(code);
            out.writeBytes("Server: Jakobs Web Server v1.0");
            out.writeBytes("Content-Type: " + fileType + "\r\n");
            out.writeBytes("");
        }
        catch (IOException e){
            System.err.println(e.getMessage());
        }
    }

    /* Sends error response to the client */
    public void error(String code, String error, String fileName){
        System.err.println(error +
                "\nFile Requested: " + fileName);

        /* Sends the error code header */
        sendHeaderResponse(code, fileName);

        /* Sends the error message and cause to client */
        send.print("<html><head><title>" + error + "</title></head><body>");
        send.print("<h1>" + error + "</h1>\r\n");
        send.println("Location: /" + fileName + "/\r\n");
        send.println("Exception Cause: " + error + "\r\n");
        send.print("<a href=\"index.html\">Start Page</a>");
        send.print("</body>\"</html>");
        send.flush();
        send.close();
    }

    /* Finds out the MIME type of the requested file */
    public String getMimeType(File f) {
        String file = f.toString();
        String type = "";
        if (file.endsWith(".txt")) {
            type = "text/txt";
        } else if (file.endsWith(".html") || file.endsWith("htm")) {
            type = "text/html";
        } else if (file.endsWith(".jpg")) {
            type = "image/jpg";
        } else if (file.endsWith(".png")) {
            type = "image/png";
        } else if (file.endsWith(".jpeg")) {
            type = "image/jpeg";
        } else if (file.endsWith(".gif")) {
            type = "image/gif";
        } else if (file.endsWith(".pdf")) {
            type = "application/pdf";
        } else if (file.endsWith(".mp3")) {
            type = "audio/mpeg";
        } else if (file.endsWith(".class")){
            type = "application/octet-stream";
        } else if (file.endsWith(".mp4")){
            type = "video/mp4";
        }
        return type;
    }
}

Answer:

Make sure that you write e.g. HTTP/1.1 404 Not Found to the client, not just the 400.

Actually no, your problem is that you don't end the response properly. The browser keeps receiving data and shows no response code received. Let me see how this can be fixed in your code.

Also, you use two wrapper streams around client.getOutputStream() to send data to the client (send and out). Not sure why you do this. This looks weird. You should use just one wrapper stream. And you never close out, probably that's your problem, that's why the browser thinks the response is not yet fully received. Try to use one stream and handle it properly.

OK, here is your code fixed.

import java.io.*;
import java.net.Socket;
import java.util.StringTokenizer;

/**
 * Web Server Request Handler.
 * Created on 2015-02-16.
 */

public class RequestHandler implements Runnable {

    /*
    TODO ( ) Problem 1
    TODO ( ) Problem 2
    TODO ( ) Problem 3
    TODO (X) Index page for first page.
    TODO (X) Read and download images & other files
    TODO ( ) Fix header responses
    TODO ( ) Error responses
     */

    private String
            OK = "HTTP/1.0 200 OK",
            NOT_FOUND = "HTTP/1.0 404 Not Found",
            BAD_REQUEST = "HTTP/1.0 400 Bad Request",
            FORBIDDEN = "HTTP/1.0 403 Forbidden",
            SERVER_ERROR = "HTTP/1.0 500 Internal Server Error";

    private String ROOT_DIR;
    private Socket client;
    private PrintStream send;
    private DataInputStream fromClient;
    // private DataOutputStream out;

    RequestHandler(Socket client, String ROOT_DIR) {
        this.client = client;
        this.ROOT_DIR = ROOT_DIR;

        try {
            send = new PrintStream(client.getOutputStream());
            fromClient = new DataInputStream(client.getInputStream());
            // out = new DataOutputStream(new BufferedOutputStream(client.getOutputStream()));


        } catch (IOException e) {
            System.err.println(e.getMessage());
        }
    }

    /* Reads the HTTP request and responds */
    public void run() {
        String request = null;
        String fileName = null;
        StringTokenizer tok = null;

        try {

            /* Read HTTP request from client */
            while ((request = fromClient.readLine()) != null) {

                System.out.println(request);
                tok = new StringTokenizer(request);

            /* Extracts the file path from the GET command */
                if (tok.hasMoreElements() && tok.nextToken().equals("GET")
                        && tok.hasMoreElements()) {

                    fileName = tok.nextToken();
                } else
                    throw new FileNotFoundException();

                /* */
                if (fileName.endsWith("/"))
                    fileName += "index.html";

                /* Illegal characters, prevent access to super directories */
                if (fileName.indexOf("..") >= 0 || fileName.indexOf('|') >= 0
                        || fileName.indexOf(':') >= 0 || fileName.indexOf('~') >= 0) {

                    error(FORBIDDEN, "Forbidden Access", fileName);
                }
                else

                if (new File(fileName).isDirectory()) {
                    fileName = fileName.replace('\\', '/');
                    send.close();
                    return;
                }

            /* File name is ROOT_DIR + file name */
                fileName = ROOT_DIR + fileName;

            /* Create file */
                File file = new File(fileName);

                if (file.isDirectory()) {
                    fileName = fileName + "index.html";
                }
            /* File does not exist */
                if (file.exists()){
            /* Determines the MIME type of the file */
                    String mimeType = getMimeType(file);

            /* Sends the file */
                    sendFile(file, mimeType, fileName);
                    client.close();
                }
                else
                    error(NOT_FOUND, "404 File Not Found", fileName);
            }

        }
        catch (FileNotFoundException e) {
            System.err.println(e.getMessage());
        }
        catch (IOException e){
            System.err.println(e.getMessage());
        }
    }

    /* Sends the requested file to the client */
    public void sendFile(File file, String fileType, String fileName) {
        try {
            // Buffer must not be to low, => fragments
            int length = 0; // (int) file.length();
            FileInputStream fileIn = new FileInputStream(fileName);
            byte[] bytes = new byte[1024];
            ByteArrayOutputStream bos = new ByteArrayOutputStream();

            /* Write until bytes is empty */
            while ((length = fileIn.read(bytes)) != -1 ){
                bos.write(bytes, 0, length);
                // send.write(bytes, 0, length);
                // send.flush();
            }
            bos.flush();
            bos.close();
            byte[] data1 = bos.toByteArray();

            System.out.print(new String(data1));
            send.print(OK);
            send.print("");
            send.print("Server: Jakobs Web Server v1.0");
            send.print("Content-Type: " + fileType + "\r\n");
            send.print("Content-Length: " + data1.length + "\r\n");
            send.println("");
            send.write(data1, 0, data1.length);
            send.println("");

            send.flush();
            send.close();

            fileIn.close();

        } catch (IOException e) {
            System.err.println(e.getMessage());
        }
    }

    /* Sends the header response to the client */
    public void sendHeaderResponse(String code, String fileType){
        try {
            send.print(code);
            send.print("Server: Jakobs Web Server v1.0");
            send.print("Content-Type: " + fileType + "\r\n");
            send.print("");
            send.println();
        }
        catch (Exception e){
            System.err.println(e.getMessage());
        }
    }

    /* Sends error response to the client */
    public void error(String code, String error, String fileName){
        System.err.println(error +
                "\nFile Requested: " + fileName);

        /* Sends the error code header */
        sendHeaderResponse(code, fileName);

        // send.println("ERROR");

        /* Sends the error message and cause to client */

        send.print("<html><head><title>" + error + "</title></head><body>");
        send.print("<h1>" + error + "</h1>\r\n");
        send.println("Location: /" + fileName + "/\r\n");
        send.println("Exception Cause: " + error + "\r\n");
        send.print("<a href=\"index.html\">Start Page</a>");
        send.print("</body></html>");

        send.flush();
        send.close();
    }

    /* Finds out the MIME type of the requested file */
    public String getMimeType(File f) {
        String file = f.toString();
        String type = "";
        if (file.endsWith(".txt")) {
            type = "text/txt";
        } else if (file.endsWith(".html") || file.endsWith("htm")) {
            type = "text/html";
        } else if (file.endsWith(".jpg")) {
            type = "image/jpg";
        } else if (file.endsWith(".png")) {
            type = "image/png";
        } else if (file.endsWith(".jpeg")) {
            type = "image/jpeg";
        } else if (file.endsWith(".gif")) {
            type = "image/gif";
        } else if (file.endsWith(".pdf")) {
            type = "application/pdf";
        } else if (file.endsWith(".mp3")) {
            type = "audio/mpeg";
        } else if (file.endsWith(".class")){
            type = "application/octet-stream";
        } else if (file.endsWith(".mp4")){
            type = "video/mp4";
        }
        return type;
    }
}

Question:

I have an IOT device which is sending data continuously to a configured server using TCP protocol.

Can i receive the data using HTTP protocol and any java http services application like spring-boot application?

Device details : https://teltonika.lt/product/fmb920/


Answer:

As the device was returning the data in TCP protocol so I could not get any solution to get data in http protocol.

I achieved that by establishing the TCP connection using sockets.