Study how can we execute applications from Java

[xni]

I don't know if starting Dockers directly from an application master is a good idea. If it is, okay, if no, we will start a Java application in Container, an it will start a Docker.

[xni]

At first, you are to add yarn user to the sudoers file by appending a line yarn ALL=NOPASSWD:ALL to the /etc/sudoers file.

[xni]

Then, install Docker: https://docs.docker.com/installation/ubuntulinux/#ubuntu-precise-1204-lts-64-bit

[xni]

Create a test container with Python abroad:

$ sudo docker run -d ubuntu sudo apt-get install -y python
0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
$ sudo docker logs 0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
...
Setting up python (2.7.5-5ubuntu3) ...
$ sudo docker commit 0630b6f59a20ce82e9d07ce9af9ef348a2837fbbac487b31c96bfda4fecf8dcc
ee55efa7a5fff1c223db815b867d37400a53c69759dd205d958bc7579d876978
$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
<none>              <none>              ee55efa7a5ff        28 seconds ago      299.9 MB
ubuntu              latest              e54ca5efa2e9        4 weeks ago         276.5 MB
$ sudo docker run ee55efa7a5ff python -c "print('Hello from dockers guts.')"
Hello from dockers guts.

So, I will use ee55efa7a5ff in my code.

[xni]

Some modifications in nodes code:

private void run() throws IOException {
    LOG.info("Running Container on {}", this.hostname);

    // http://stackoverflow.com/questions/617414/create-a-temporary-directory-in-java
    final File temp;
    final File parent_dir = new File("/opt/docker-output");
    temp = File.createTempFile("docker-", null, parent_dir);
    temp.delete();
    temp.mkdir();

    LOG.info("Starting docker");
    ProcessBuilder pb = new ProcessBuilder(
            "sudo",
            "docker", "run",
            "-i",
            "-v", temp.toString() + ":/opt/data",
            "ee55efa7a5ff",
            "python", "-c", "open('/opt/data/test.txt', 'w').write('Docker: Ok')");
    Process docker_process = pb.start();
    LOG.info("Docker is up. Waiting...");
    try {
        docker_process.waitFor();
        LOG.info("Application is finished. Exit status: {}", docker_process.exitValue());
        String stdout = slurp(docker_process.getInputStream(), 1024);
        String stderr = slurp(docker_process.getErrorStream(), 1024);
        LOG.info("Stdout: {}", stdout);
        LOG.info("Stderr: {}", stderr);

    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    try {
        Thread.sleep(60000);

    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

// http://stackoverflow.com/questions/309424/read-convert-an-inputstream-to-a-string
public static String slurp(final InputStream is, final int bufferSize)
{
    final char[] buffer = new char[bufferSize];
    final StringBuilder out = new StringBuilder();
    try {
        final Reader in = new InputStreamReader(is, "UTF-8");
        try {
            for (;;) {
                int rsz = in.read(buffer, 0, buffer.length);
                if (rsz < 0)
                    break;
                out.append(buffer, 0, rsz);
            }
        }
        finally {
            in.close();
        }
    }
    catch (UnsupportedEncodingException ex) {
    }
    catch (IOException ex) {
    }
    return out.toString();
}

Files are created,

$ head /opt/docker-output/*/test.txt
==> /opt/docker-output/docker-1435383504020414450.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-2130967975752402865.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-2887434490859823761.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-5126749122784000365.tmp/test.txt <==
Docker: Ok
==> /opt/docker-output/docker-5945479457457188145.tmp/test.txt <==
Docker: Ok

But Java freezes on docker_process.waitFor(); for forever. So I'll perform some extra investigations.

[xni]

I've changed -i to -d, and container start was successful:

14/07/20 09:30:41 INFO yarntest.MyContainer: Container just started on master/192.168.7.10
14/07/20 09:30:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/20 09:30:43 INFO yarntest.MyContainer: Running Container on master/192.168.7.10
14/07/20 09:30:43 INFO yarntest.MyContainer: Starting docker
14/07/20 09:30:43 INFO yarntest.MyContainer: Docker is up. Waiting...
14/07/20 09:30:44 INFO yarntest.MyContainer: Application is finished. Exit status: 0
14/07/20 09:30:44 INFO yarntest.MyContainer: Stdout: 3830f2f2236a9df06a4b516c12d5737345804b39578802c2109824e23bc82c1b
14/07/20 09:30:44 INFO yarntest.MyContainer: Stderr: 


14/07/20 09:33:10 INFO yarntest.MyContainer: Container just started on master/192.168.7.10
14/07/20 09:33:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/20 09:33:14 INFO yarntest.MyContainer: Running Container on master/192.168.7.10
14/07/20 09:33:14 INFO yarntest.MyContainer: Starting docker
14/07/20 09:33:14 INFO yarntest.MyContainer: Docker is up. Waiting...
14/07/20 09:33:14 INFO yarntest.MyContainer: Application is finished. Exit status: 0
14/07/20 09:33:14 INFO yarntest.MyContainer: Stdout: 1fe97bd025b5229fc4ba52d1d40f28e684af77e94505bbed3088d96b6b4e036d
14/07/20 09:33:14 INFO yarntest.MyContainer: Stderr:

So, parsing this container ids and parsing docker ps may be useful.