Hot questions for Using Azure in docker

Question:

I have a dc/os cluster deployed to azure. I have deployed to the cluster a container with my java application. But I can't access it via jmx.

Let's take the example of deploying a standart tomcat image:

1) I opened a port 8081 according the next instruction: https://docs.microsoft.com/en-us/azure/container-service/container-service-enable-public-access#open-a-port-portal.

2) I deployed service using the next json:

{
  "id": "/tomcat",
  "instances": 1,
  "cpus": 1,
  "mem": 512,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "tomcat:8.0",
      "network": "BRIDGE",
      "portMappings": [
        { "protocol": "tcp", "hostPort": 8080   , "containerPort": 8080 },
        { "protocol": "tcp", "hostPort": 8081   , "containerPort": 8081 }
      ]
    }
  },
  "requirePorts": true,
  "acceptedResourceRoles": [
    "slave_public"
  ],
  "env": {      
    "JAVA_OPTS": "-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=10.0.0.4 -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.rmi.port=8081 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 120,
      "intervalSeconds": 30,
      "maxConsecutiveFailures": 3,
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "timeoutSeconds": 5
    }
  ]
}

To connect I use Oracle Java Mission Control. I fill fields 'Host' and 'Port' as 'prefixagents.westeurope.cloudapp.azure.com' and '8081'. But I can't connect and I get a message: 'Unable to connect'.

But for all that I can succesfully connect to this port using telnet client:

telnet prefixagents.westeurope.cloudapp.azure.com 8081

Also I can connect to port 8080 and I can open tomcat web console at the following URL: http://agents.westeurope.cloudapp.azure.com:8080

I installed one more jmx command line client - http://wiki.cyclopsgroup.org/jmxterm/ and tried to connect to the service:

java -jar jmxterm-1.0-alpha-4-uber.jar  --url service:jmx:rmi:///jndi/rmi://<prefix>agents.westeurope.cloudapp.azure.com:8081/jmxrmi

I got the next exception: "java.rmi.ConnectException: Connection refused to host: 10.0.0.4". And 10.0.0.4 is hostname of my public agents node.

I connected to my dc/os cluster ( master node ) using https://docs.microsoft.com/en-us/azure/container-service/container-service-connect. I also installed there jmxterm and tried to connect to the service via jmx:

java -jar jmxterm.jar --url service:jmx:rmi:///jndi/rmi://10.0.0.4:8081/jmxrmi

And I connected succesfully.

Does anybody have any ideas why I can connect to the service via jmx from my master node but I can't from my local machine? Port 8081 is opened.


Answer:

I changed a value of property -Djava.rmi.server.hostname in JAVA_OPTS string: -Djava.rmi.server.hostname="public agent ip". And it works for me. Working configuration for a standart tomcat container in case that port 8081 is open:

{
  "id": "/tomcat",
  "instances": 1,
  "cpus": 1,
  "mem": 512,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "tomcat:8.0",
      "network": "BRIDGE",
      "portMappings": [
        { "protocol": "tcp", "hostPort": 8080   , "containerPort": 8080 },
        { "protocol": "tcp", "hostPort": 8081   , "containerPort": 8081 }
      ]
    }
  },
  "requirePorts": true,
  "acceptedResourceRoles": [
    "slave_public"
  ],
  "env": {      
    "JAVA_OPTS": "-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=<public agent ip> -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.rmi.port=8081 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 120,
      "intervalSeconds": 30,
      "maxConsecutiveFailures": 3,
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "timeoutSeconds": 5
    }
  ]
}

And I can connect to tomcat via jmx using from local machine: host="public agent ip" and port=8081

public agent ip = "prefix"agents.westeurope.cloudapp.azure.com

Question:

I'm developing a SpringBoot app dockerized. The docker images are microservices and one of these communicate with Azure Event Hub.

Some of my properties:

spring-boot -> 2.0.7.RELEASE

spring-cloud.version -> Finchley.SR2

I've created a topic in Azure(with Kafka enabled).

I've follow some simple guide to set up my microservice and everything works fine.

@EnableBinding({Processor.class})
public class EventService {
    ...
    @Autowired private Processor ehProcessor;
    ...
    public void send(String event) {

        Message<String> message = MessageBuilder
                .withPayload(event)
                .setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
                .build();

        boolean send = ehProcessor.output().send(message, 5000L);

        if (!send) {

            log.error("Event NOT sent", event);
        }
    }

    ...

    @StreamListener(target = Processor.INPUT)
    public void receive(String event) {

        handle(event);
    }
}

For an entire month everything goes fine but in the last two days the microservice stucks because a continuous stacktrace is filling all my disc (a solution was to set up a docker log rotation).

java.lang.IllegalStateException: Unexpected error code 13 while fetching data
        at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:891) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:528) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1154) ~[kafka-clients-1.0.1.jar!/:na]
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1111) ~[kafka-clients-1.0.1.jar!/:na]
        at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:699) ~[spring-kafka-2.1.7.RELEASE.jar!/:2.1.7.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

I'm talking about 8/9 log messsage in 1 ms.

Looking inside the org.apache.kafka.common.protocol.Errors class the error is related to:

NETWORK_EXCEPTION(13, "The server disconnected before a response was received."

I'm not able to reproduce programmatically this error. I don't understand why when the first error is arised the log will start and no stop in infinite loop. I need to stop the docker container and sometimes the container will not stop. The only solution is to remove the container and recreate again.

UPDATE

I've open an issue on github here. I've already received a response and they are starting to investigate on it.

UPDATE

The problem has been fixed.

When they changed an UnknownServerException to a NetworkException, Spring Boot started getting stuck in the retry loop.


Answer:

It has confirm in this link that,

There was a recent change where an instance of UnknownServerException was changed to a NetworkException.

Issue details are here - https://github.com/Azure/azure-event-hubs-for-kafka/issues - with your namespace info. Thanks!

Question:

I'm new to docker and dc/os. I have deployed dc/os cluster in the microsoft azure. I need to set access via a jmx to my java applications but I can't. Let's take the example of deploying a standart tomcat image. I have docker installed on my local machine. To run a tomcat container with jmx access I use this command:

docker run -e 
     JAVA_OPTS= "-Dcom.sun.management.jmxremote 
                -Djava.rmi.server.hostname=127.0.0.1 
                -Dcom.sun.management.jmxremote.port=8081 
                -Dcom.sun.management.jmxremote.rmi.port=8081 
                -Dcom.sun.management.jmxremote.local.only=false 
                -Dcom.sun.management.jmxremote.authenticate=false 
                -Dcom.sun.management.jmxremote.ssl=false" 
     -p 8080:8080 -p 8081:8081 tomcat:8.0   

And I can connect to Tomcat via port 8081.

I try to do the same in dc/os cluster. I use the below json configuration to deploy:

{
  "id": "/tomcat",
  "instances": 1,
  "cpus": 1,
  "mem": 512,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "tomcat:8.0",
      "network": "BRIDGE",
      "portMappings": [
        { "protocol": "tcp", "hostPort": 8080, "containerPort": 8080 },
        { "protocol": "tcp", "hostPort": 8081, "containerPort": 8081 }
      ]
    }
  },
  "requirePorts": true,
  "acceptedResourceRoles": [
    "slave_public"
  ],
  "env": {      
    "JAVA_OPTS": "-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=10.0.0.4 -Dcom.sun.management.jmxremote.port=8081 -Dcom.sun.management.jmxremote.rmi.port=8081 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 120,
      "intervalSeconds": 30,
      "maxConsecutiveFailures": 3,
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "timeoutSeconds": 5
    }
  ]
}

After that I have access to tomcat webconsole but I can't connnect via jmx. I tried variouse values for "-Djava.rmi.server.hostname": 127.0.0.1, 10.0.0.4( agent ip ), agents.westeurope.cloudapp.azure.com. Please help me understand what I do wrong.

UPDATE: Thank Walter - MSFT who pointed out a fact which ports are opened by default on azure. I really forgot about it. But an issue with connecting via jmx is still actual for me. I opened new discussion where I give more details. DC/OS JMX Access


Answer:

You could read Azure official article:

Any DC/OS container in the ACS public agent pool is automatically exposed to the internet. By default, ports 80, 443, 8080 are opened, and any (public) container listening on those ports are accessible.

According to your description, it seems that port 8081 is not open. You could open port 8081 on Azure Portal.

More information about please refer to this link: Enable public access to an Azure Container Service application.

Update:

I test in my lab with your json file, it works for me, you don't change it. You should open port on Azure NSG and Load Balance.

NSG:

LoadBalncer

I test in my lab, I could open 8080 Web UI. When I test port 8081, I notice that the port is listening, I could access the port with Public IP.

azureuser@dcos-master-01234567-0:~$ netcat -z -v 13.84.176.235 8081
Connection to 13.84.176.235 8081 port [tcp/tproxy] succeeded!

You also could use curl to test, I get the following result.

azureuser@dcos-master-01234567-0:~$ curl 13.84.176.235:8081
curl: (52) Empty reply from server

If you could not access 8081 Web UI, I suggest you had better check docker container.