Quick Start

This page gets you up and running the DtCraft system in minutes.

System Requirements

DtCraft is quite self-contained and requires only a few packages to install. You only need:

64-bit Linux machines (Ubuntu recommended)
GNU C++17 compiler G++ v7.2 or newer
Linux kernel version 3.8 or higher
GNU Autotool (autoconf, automake, and libtool)

You also need the privilege (sudo) for DtCraft to launch Linux containers and manage cluster resources.

Download

Download the latest release from here.

The source of DtCraft is placed in folders include, src, and webui, respectively. The folder bin contains the core kernel binaries. Kernel execution scripts and configuration files are stored in folders conf and sbin.

# Directory tree
├── bin        # Kernel binaries
├── conf       # Configuration file
├── example    # Tutorial examples
├── include    # Header files
│   └── dtc
│       ├── archive
│       ├── concurrent
├──     ...
├── lib        # Library to link
├── main       # Kernel source
├── sbin       # Scripts
├── src        # CPP source
│   ├── event
│   ├── ipc
│   ├── kernel
│   ...
├── unittest   # Unit test source
└── webui      # Web UI HTML source

Build

We use GNU Autotool to manage the DtCraft package.

# Change the working directory
$ cd DtCraft 

# Configure and build
$ ./configure
$ make

You can also run make -j <core count> to speed up the build process.

Regression

After the build process is completed, you can run the regression to test the code base on your machine.

# Unit testing
$ make check
PASS: unittest/statgrab.sh
PASS: unittest/utility.sh
...
============================================================================
Testsuite summary for DtCraft
============================================================================
...

DtCraft Local Mode

Running a DtCraft application on a local machine is as easy as running a binary at your terminal.

# Run the hello-world example in local mode
$ ./example/hello_world 
Sent 'hello world from B' to stream 5
Sent 'hello world from A' to stream 4
Received: hello world from A
Received: hello world from B

Notice that DtCraft is by default parallel and asynchronous. Different runs might give you different results in terms of output order. In practice, the local mode is particularly useful for diagnostics purpose. You can get a quick run on your stream graph and check whether the result is desirable.

DtCraft Distributed Mode (Single Host)

Distributing your DtCraft application to a computer cluster is fairly simple through our scripts. To avoid steep learning curve, we start from a single host. On your local host (127.0.0.1), open three terminals. We will use one for master, one for agent, and one for executor in the foreground. In one terminal, start the master.

# Terminal 1: Launch the master
$ sudo ./bin/master
I 65280 2017-11-27 11:57:22 master.cpp:178] Master @127.0.0.1 [agent:9909|graph:9910|webui:9912]

Switch to another terminal and start the agent.

# Terminal 2: Launch the agent
$ sudo ./bin/agent
I 42880 2018-05-15 19:22:31 agent.cpp:135] Agent @127.0.0.1 [frontier:9913]
I 42880 2018-05-15 19:22:31 agent.cpp:138] cg-subsys.memory "/sys/fs/cgroup/memory/dtc" [limit:24993161216]
I 42880 2018-05-15 19:22:31 agent.cpp:139] cg-subsys.cpuset "/sys/fs/cgroup/cpuset/dtc" [cpus:4]
I 42880 2018-05-15 19:22:31 agent.cpp:140] cg-subsys.blkio "/sys/fs/cgroup/blkio/dtc" [weight:500]

Now submit your DtCraft application through our submission script sbin/submit.sh.

# Terminal 3: Run the hello-world example in distributed mode
$ ./sbin/submit.sh --master=127.0.0.1 /home/dtcraft/DtCraft/example/hello_world
I 26496 2018-05-15 19:24:58 executor.cpp:159] Executor @csl-408-08.csl.illinois.edu [stdout:45137|stderr:45741]
I 26496 2018-05-15 19:24:58 executor.cpp:161] Submit graph to master @127.0.0.1:9910
Sent 'hello world from A' to stream 3
Received: hello world from B
Sent 'hello world from B' to stream 4
Received: hello world from A
I 38656 2018-05-15 19:24:58 executor.cpp:173] Solution received
[Graph 0]
+----+---------+---------+-----------+-------------------+
|Task|    Agent|   Status|Elapsed (s)|Memory (peak/limit)|
+----+---------+---------+-----------+-------------------+
| 0-5|127.0.0.1|exited ok|   0.067994| 1921024/1073741824|
+----+---------+---------+-----------+-------------------+
| 0-2|127.0.0.1|exited ok|   0.068002| 2166784/1073741824|
+----+---------+---------+-----------+-------------------+
Graph finished with 0 error(s)

DtCraft Distributed Mode (Cluster)

The steps to distribute a DtCraft application to multiple machines is by and large the same as Distributed Mode (Single Host), except we provide scripts to simplify the process. Here, we demonstrate an example of submitting a program to a cluster of two machines, A and B. A will be the master and both A and B have an agent. To avoid error by password prompt, enable automatic ssh login without password from A to B, and password-less privilege for each login user.

	Hostname (DNS resolvable)	User login	DtCraft home
Machine A	cluster-host-A	dtcraft	/home/dtcraft/DtCraft
Machine B	cluster-host-B	dtcraft	/home/dtcraft/DtCraft

Modify the cluster host files conf/master and conf/agents on machine A.

# user host DtCraft-home
dtcraft cluster-host-A /home/dtcraft/DtCraft

# user host DtCraft-home
dtcraft cluster-host-A /home/dtcraft/DtCraft
dtcraft cluster-host-B /home/dtcraft/DtCraft

Start the DtCraft kernel through the kernel script sbin/kernel.sh. This will daemonize the master on machine A, and agents on machines A and B. You can inspect the kernel log file in the folder workspace/log of respective machines.

# Machine A: Start the kernel
$ ./sbin/kernel.sh --action=start
start dtc-master @cluster-host-A
start dtc-agent @cluster-host-A
start dtc-agent @cluster-host-B

# Machine A: Ping the kernel
$ ./sbin/kernel.sh --action=ping
dtc-master is running @cluster-host-A
dtc-agent is running @cluster-host-A
dtc-agent is running @cluster-host-B

# Machine A: workspace/log/
workspace/
└── log
    ├── dtc-agent@cluster-host-A
    └── dtc-master@cluster-host-B

# Machine B: workspace/log/
workspace/
└── log
    └── dtc-agent@cluster-host-B

Now submit your DtCraft application through our submission script sbin/submit.sh. The path to your binary must be visible to all cluster nodes. In a production environment, this is typically handled by a uniform resource identifier (URI) or a network file system (NFS).

# Machine A: Run the hello-world example in distributed mode
$ ./sbin/submit.sh --master=cluster-host-A /home/dtcraft/DtCraft/example/hello_world

# Machine A: Stop the kernel
$ ./sbin/kernel.sh --action=stop
stop dtc-master @cluster-host-A
stop dtc-agent @cluster-host-A
stop dtc-agent @cluster-host-B

Web UI

You can inspect the cluster status through our web-based interface at http://localhost:9912/.

Troubleshooting

Kernel script failed to start

If you are getting error on password prompt or permission denied when launching kernel.sh, it is very likely you forget to enable automatic ssh login and password-less privilege. The script is just a wrapper that executes the commands described in Distributed Mode (Single Host) at each remote machine. Make sure you successfully launch a local DtCraft cluster before going distributed.

Execution error on no such file

The executable to submit to a DtCraft cluster must be visible to all machines. This includes the file path, runtime libraries, and other input files required by the executable. In a production cloud, engineers typically install a shared storage such as Network File System (NFS).