Hot questions for Using Transmission Control Protocol in nginx

Question:

I have five tomcat instances behind nginx.

Sometimes the nginx upstream_response_time is very big, more than 1 second, while the tomcat local access log shows the process time is only 50ms(I use %D to log process time).

What is the possible reason and how to fix it? It does not seems the network is slow since other applications run fast.

Update:

Seems the nginx upstream_response_time = %D + 1 sec.


Answer:

My hypothesis for your observation are packet losses. This seems the most probable problem to me since you are saying that it happens when they are many requests. To test this, you can monitor the traffic e.g. with tcpdump or iftop. If you are under Ubuntu, you can install and start the latter with

sudo apt-get install iftop
sudo iftop

There are many other network monitoring solutions in Linux, the amazing Wireshark works for all operating systems.

One reason for package losses can be collisions, you can check that with a ifconfig [interface] if you are under Linux:

me@mymahine:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddrf f:41:8d:ef:41:8d  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 <-------------------------- check here ---
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Are the Tomcats and the Nginx on the same physical (or virtual) machine?

Further reading

Question:

  • Rationale :

I'm having the case when the server side (backing service) can not provide authorisation/authentication layer.

  • Approach:

I'm trying to hide behing the nginx and thus to achieve a client-server mutual authorisation (from java client to nginx) java client <-(ssl,certificates)->nginx:8102<-(open tcp)->backing service:18102 For simplification nginx and backing service reside on one VM.

To achieve that I followed the tutorial : http://nategood.com/client-side-certificate-authentication-in-ngi I have generated ca,client,server keys,certificates as follows (the server certificate is self signed)

I also have created some nginx set up:

worker_processes  1;
error_log .../error.log debug;
events {
  worker_connections  1024;
}
stream {
  upstream stream_backend {
    server localhost:18102;
  }
  server {
    listen                8102 ssl;
    proxy_pass            stream_backend;
    ssl_certificate        .../server.crt;
    ssl_certificate_key    .../server.key;
    ssl_trusted_certificate .../ca.crt;
    ssl_client_certificate .../client.crt;
    ssl_verify_depth 1;
    ssl_verify_client on;
    #ssl_protocols          TLSv1.1 TLSv1.2;
    #ssl_ciphers           HIGH:!aNULL:!MD5;
    #ssl_ciphers "AES256-SHA:SHA512withECDSA";
    #ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    #ssl_session_cache     shared:SSL:20m;
    ssl_session_timeout   4h;
    ssl_handshake_timeout 30s;
  }
}

Then i was able to autorise using open ssl

openssl s_client -cert client.crt -key client.key -debug -connect <nginxhost>:8102

So far so good.

  • The issue:

Then i followed to integrate it with JVM client

I generated the password protected PKCS12 file for client.crt

openssl pkcs12 -export -in client.crt -inkey client.key  -out client.p12

and also was trying to generate the pkcs12 file for all combination of certificates (using the trick with trust store)

keytool -import -file ca.crt -alias ca -keystore trust-store
keytool -import -file server.crt -alias server -keystore trust-store;
keytool -import -file client.crt -alias client -keystore trust-store;
keytool -importkeystore -srckeystore trust-store -destkeystore client.p12  -srcstoretype JKS -deststoretype PKCS12 -deststorepass <anygiven passwd>

, but was not able to properly autorize I run a plain Jedis client

Jedis jedis = new Jedis(uri);//, sslSocketFactory, sslParameters, null);
jedis.set("foo", "bar");
System.out.println(">>>" + jedis.get("foo"));
jedis.disconnect();
jedis.close();

with options

-Djavax.net.debug=ssl:record  -Djavax.net.debug=ssl:handshake -Djavax.net.debug=ssl -Djavax.net.ssl.keyStore=client.p12 -Djavax.net.ssl.keyStorePassword=<given>

What i get is :

*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1498167495 bytes = { 191, 197, 46, 4, 183, 146, 82, 209, 69, 135, 151, 205, 65, 55, 24, 150, 123, 60, 122, 226, 192, 23, 211, 254, 14, 59, 34, 244 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
***
main, WRITE: TLSv1.2 Handshake, length = 167
main, READ: TLSv1.2 Handshake, length = 89
*** ServerHello, TLSv1.2
RandomCookie:  GMT: -1537147415 bytes = { 45, 32, 242, 160, 72, 168, 116, 190, 65, 91, 130, 252, 92, 249, 244, 109, 101, 74, 162, 188, 255, 196, 148, 149, 17, 213, 198, 57 }
Session ID:  {57, 214, 187, 172, 64, 105, 42, 71, 106, 76, 50, 178, 51, 65, 243, 193, 201, 181, 197, 12, 180, 23, 43, 213, 228, 121, 240, 227, 68, 115, 112, 147}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
Extension ec_point_formats, formats: [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
***
%% Initialized:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
** TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
main, READ: TLSv1.2 Handshake, length = 904
*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 148561064520149382265326610552347817946745543429726527079571023440869096298585266785042725149311452500913565526887156223039098472927158834345207963498945320440101735487342373095629935479052530558787124675414307741499494705584955553763604061463652853202297289153979489934006401951701144761964805588493413561041
  public exponent: 65537
  Validity: [From: Thu Jun 22 10:25:22 CEST 2017,
               To: Fri Jun 22 10:25:22 CEST 2018]
  Issuer: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
  SerialNumber: [    01]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 78 C2 25 5D EE 44 50 1D   E1 77 5A EB 3F D7 A6 E6  x.%].DP..wZ.?...
0010: 55 08 C8 C2 E2 24 44 B7   D8 AE 35 51 CA 93 2E CE  U....$D...5Q....
..
01E0: B5 89 B4 A6 9E 2F B4 C6   37 5A CE 0A D3 39 B7 D4  ...../..7Z...9..
01F0: CC 6A 9A B5 46 05 B8 17   89 29 90 DB DC 9E B5 B9  .j..F....)......

]
***
Found trusted certificate:
[
[
  Version: V1
  Subject: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 148561064520149382265326610552347817946745543429726527079571023440869096298585266785042725149311452500913565526887156223039098472927158834345207963498945320440101735487342373095629935479052530558787124675414307741499494705584955553763604061463652853202297289153979489934006401951701144761964805588493413561041
  public exponent: 65537
  Validity: [From: Thu Jun 22 10:25:22 CEST 2017,
               To: Fri Jun 22 10:25:22 CEST 2018]
  Issuer: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
  SerialNumber: [    01]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 78 C2 25 5D EE 44 50 1D   E1 77 5A EB 3F D7 A6 E6  x.%].DP..wZ.?...
0010: 55 08 C8 C2 E2 24 44 B7   D8 AE 35 51 CA 93 2E CE  U....$D...5Q....
..
01E0: B5 89 B4 A6 9E 2F B4 C6   37 5A CE 0A D3 39 B7 D4  ...../..7Z...9..
01F0: CC 6A 9A B5 46 05 B8 17   89 29 90 DB DC 9E B5 B9  .j..F....)......
]
main, READ: TLSv1.2 Handshake, length = 205
*** ECDH ServerKeyExchange
Signature Algorithm SHA512withRSA
Server key: Sun EC public key, 256 bits
  public x coord: 99646631278318376795489495527453257751498300731056420595286629015741962063348
  public y coord: 4942359814115858826459478922742110932389912636636308073122086926577854097688
  parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
main, READ: TLSv1.2 Handshake, length = 119
*** CertificateRequest
Cert Types: RSA, DSS, ECDSA
Supported Signature Algorithms: SHA512withRSA, Unknown (hash:0x6, signature:0x2), SHA512withECDSA, SHA384withRSA, Unknown (hash:0x5, signature:0x2), SHA384withECDSA, SHA256withRSA, SHA256withDSA, SHA256withECDSA, SHA224withRSA, SHA224withDSA, SHA224withECDSA, SHA1withRSA, SHA1withDSA, SHA1withECDSA
Cert Authorities:
<O=Internet Widgits Pty Ltd, ST=Some-State, C=AU>
*** ServerHelloDone
Warning: no suitable certificate found - continuing without client authentication
*** Certificate chain
<Empty>
***
*** ECDHClientKeyExchange
ECDH Public value:  { 4, 146, 247, 118, 74, 204, 87, 229, 88, 182, 122, 231, 203, 29, 89, 160, 251, 221, 92, 20, 67, 20, 70, 198, 21, 216, 245, 102, 33, 84, 141, 27, 247, 73, 120, 165, 200, 203, 98, 156, 155, 30, 127, 40, 106, 229, 240, 129, 59, 146, 14, 168, 195, 233, 43, 25, 55, 206, 24, 255, 17, 172, 230, 42, 212 }
main, WRITE: TLSv1.2 Handshake, length = 77
SESSION KEYGEN:
PreMaster Secret:
0000: A7 B2 BA B4 E7 4B 06 7E   B5 DB E9 8C C1 C6 57 AF  .....K........W.
0010: C2 45 88 D5 AF 27 25 BF   DB 25 07 9C 53 FE 8A 06  .E...'%..%..S...
CONNECTION KEYGEN:
Client Nonce:
0000: 59 4C 39 C7 BF C5 2E 04   B7 92 52 D1 45 87 97 CD  YL9.......R.E...
0010: 41 37 18 96 7B 3C 7A E2   C0 17 D3 FE 0E 3B 22 F4  A7...<z......;".
Server Nonce:
0000: A4 61 FE E9 2D 20 F2 A0   48 A8 74 BE 41 5B 82 FC  .a..- ..H.t.A[..
0010: 5C F9 F4 6D 65 4A A2 BC   FF C4 94 95 11 D5 C6 39  \..meJ.........9
Master Secret:
0000: F7 CF EF 52 4B 1A 5C DB   FB 10 21 CE 08 CA 53 DF  ...RK.\...!...S.
0010: 4B FD 21 27 74 D3 48 ED   9B BF 69 FC F0 D9 C1 81  K.!'t.H...i.....
0020: C5 A1 8A 8E 97 F3 00 F0   40 2A CB F4 5D AC 8E 0F  ........@*..]...
Client MAC write Secret:
0000: 4B FC 7A 6D 42 5F 3C 17   EC 7E 96 49 16 FE 48 FD  K.zmB_<....I..H.
0010: A5 E9 12 A3 D8 9C A1 7C   0A 83 3A CF 38 40 11 1D  ..........:.8@..
Server MAC write Secret:
0000: 19 22 CE 9A 44 28 25 DF   7B 7D EF C7 8E CC 02 8D  ."..D(%.........
0010: DB 65 62 70 B6 32 C2 05   96 A5 9E 2A DA A0 3F 2D  .ebp.2.....*..?-
Client write key:
0000: 3F AB B7 73 AD 71 BF 69   62 C6 5E D0 5B 41 BA 59  ?..s.q.ib.^.[A.Y
Server write key:
0000: 83 FF 63 92 F0 55 C0 E1   52 61 62 4C 52 08 C0 22  ..c..U..RabLR.."
... no IV derived for this protocol
main, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data:  { 83, 201, 118, 195, 61, 142, 225, 100, 14, 250, 36, 62 }
***
main, WRITE: TLSv1.2 Handshake, length = 80
main, READ: TLSv1.2 Change Cipher Spec, length = 1
main, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data:  { 16, 123, 83, 171, 99, 243, 7, 153, 78, 213, 172, 252 }
***
%% Cached client session: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
main, WRITE: TLSv1.2 Application Data, length = 80
main, READ: TLSv1.2 Alert, length = 64
main, RECV TLSv1.2 ALERT:  warning, close_notify
main, called closeInternal(false)
main, SEND TLSv1.2 ALERT:  warning, description = close_notify
main, WRITE: TLSv1.2 Alert, length = 64
main, Exception sending alert: java.net.SocketException: Broken pipe (Write failed)
main, called closeSocket(false)

```

If I'm not mistaken the server can't find matching certificate (among its known client certificate) for given client certificate ?


Answer:

The issue was on the client side.

When I creates the PKCS#12 file for client key,client cert

openssl pkcs12 -export -out client.p12 -inkey client.key -in client.crt

and then imported it into trust store

keytool -importkeystore -destkeystore ts-stage.jks -srckeystore client.p12 -srcstoretype PKCS12 -deststorepass <the same as above, truststore strong passwd>

together with server certificate

keytool -import -alias server -file server.crt -keystore ts-stage.jks -storepass <truststore strong passwd>

I was able to connect via JAVA client using SSLSocketFactory

private static SSLSocketFactory trustStoreWithClientPCKSAndServerCert() throws Exception {

    KeyStore trustStore = KeyStore.getInstance("jceks");
    InputStream inputStream = null;
    try {
        inputStream = new FileInputStream("ts-stage.jks");
        trustStore.load(inputStream, TRUST_STORE_PASSWD
                .toCharArray());
    } finally {
        inputStream.close();
    }

    final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
    keyManagerFactory.init(trustStore, PKCS_PASSWD.toCharArray());

    TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    trustManagerFactory.init(trustStore);
    TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();

    SSLContext sslContext = SSLContext.getInstance("TLS");
    sslContext.init(keyManagerFactory.getKeyManagers(), trustManagers, new SecureRandom());
    return sslContext.getSocketFactory();
}

Question:

I have a Java project using the java.net library to create a TCP server, and I wish to run it on AWS Elastic Beanstalk.

I understand that nginx can't host native TCP sockets (prior to v1.9, which EB doesn't support yet), and EB uses NGINX v1.8 (even though it says it uses v1.6.2 here).

Is there any way to get my server running on Elastic Beanstalk? Should I look into switching from Java SE to Java EE?

Thanks, Nathan


Answer:

@Zoyt You can deploy TCP server on the AWS Elastic Beanstalk (EB). EB is primary focused on the typical web application. However, there is ways to configure EB as you need. What I would recommended:

  1. Add to your application health check web page - eb is focused on the web, so we need to give to the eb web page;
  2. With help of the eb configuration, we can modify ELB settings and specify that we want to listen TCP port and set other required parameters to us. Available environment configuration options here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html
  3. In order to modify software and OS configuration you can use: .ebextensions configurations file: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html - this configs give ability to install packages, create file and run some scripts, required for the bootstrapping of the your instances. In general the idea is a same like: Cloud-init (https://cloudinit.readthedocs.org/en/latest/). Also if customisation is to complex, you may build own AMI and use it. Approach with the custom AMI also speed up start of the ec2 instances, because complicated bootstrap scripts can run slowly or fails.

As summary you have all the tools that allows to deploy your server on the EB.