idml.github.io.off/basic-usage.html at master · IDML/idml.github.io.off · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
<html><head><title>IDML: Basic Usage</title><meta charset="utf-8" /><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" /><meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="author" content="DataSift" /><meta name="description" content="Ingestion Data Mapping Language" /><meta name="og:image" content="/img/poster.png" /><meta name="image" property="og:image" content="/img/poster.png" /><meta name="og:title" content="IDML: Basic Usage" /><meta name="title" property="og:title" content="IDML: Basic Usage" /><meta name="og:site_name" content="IDML" /><meta name="og:url" content="https://idml.io/" /><meta name="og:type" content="website" /><meta name="og:description" content="Ingestion Data Mapping Language" /><link rel="icon" type="image/png" href="/img/favicon.png" /><meta name="twitter:title" content="IDML: Basic Usage" /><meta name="twitter:image" content="/img/poster.png" /><meta name="twitter:description" content="Ingestion Data Mapping Language" /><meta name="twitter:card" content="summary_large_image" /><link rel="icon" type="image/png" sizes="16x16" href="/img/favicon16x16.png" /><link rel="icon" type="image/png" sizes="24x24" href="/img/favicon24x24.png" /><link rel="icon" type="image/png" sizes="32x32" href="/img/favicon32x32.png" /><link rel="icon" type="image/png" sizes="48x48" href="/img/favicon48x48.png" /><link rel="icon" type="image/png" sizes="57x57" href="/img/favicon57x57.png" /><link rel="icon" type="image/png" sizes="60x60" href="/img/favicon60x60.png" /><link rel="icon" type="image/png" sizes="64x64" href="/img/favicon64x64.png" /><link rel="icon" type="image/png" sizes="70x70" href="/img/favicon70x70.png" /><link rel="icon" type="image/png" sizes="72x72" href="/img/favicon72x72.png" /><link rel="icon" type="image/png" sizes="76x76" href="/img/favicon76x76.png" /><link rel="icon" type="image/png" sizes="96x96" href="/img/favicon96x96.png" /><link rel="icon" type="image/png" sizes="114x114" href="/img/favicon114x114.png" /><link rel="icon" type="image/png" sizes="120x120" href="/img/favicon120x120.png" /><link rel="icon" type="image/png" sizes="128x128" href="/img/favicon128x128.png" /><link rel="icon" type="image/png" sizes="144x144" href="/img/favicon144x144.png" /><link rel="icon" type="image/png" sizes="150x150" href="/img/favicon150x150.png" /><link rel="icon" type="image/png" sizes="152x152" href="/img/favicon152x152.png" /><link rel="icon" type="image/png" sizes="196x196" href="/img/favicon196x196.png" /><link rel="icon" type="image/png" sizes="310x310" href="/img/favicon310x310.png" /><link rel="icon" type="image/png" sizes="310x150" href="/img/favicon310x150.png" /><link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" /><link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" /><link rel="stylesheet" href="/highlight/styles/solarized-light.css" /><link rel="stylesheet" href="/css/style.css" /><link rel="stylesheet" href="/css/palette.css" /><link rel="stylesheet" href="/css/codemirror.css" /><link rel="stylesheet" href="/css/override.css" /><link rel="stylesheet" href="/css/toc.css" /></head><body class="docs"><div id="wrapper"><div id="sidebar-wrapper"><ul id="sidebar" class="sidebar-nav"><li class="sidebar-brand"><a href="/" class="brand"><div class="brand-wrapper"><span>IDML</span></div></a></li> <li><a href="/dependencies.html" class="">Dependencies</a></li> <li><a href="/basic-usage.html" class=" active ">Basic Usage</a></li>     </ul></div><div id="page-content-wrapper"><div class="nav"><div class="container-fluid"><div class="row"><div class="col-lg-12"><div class="action-menu pull-left clearfix"><a href="#menu-toggle" id="menu-toggle"><i class="fa fa-bars" aria-hidden="true"></i></a></div><ul class="pull-right"><li id="gh-eyes-item" class="hidden-xs"><a href="https://github.com/idml/idml.github.io"><i class="fa fa-eye"></i><span>WATCH<span id="eyes" class="label label-default">--</span></span></a></li><li id="gh-stars-item" class="hidden-xs"><a href="https://github.com/idml/idml.github.io"><i class="fa fa-star-o"></i><span>STARS<span id="stars" class="label label-default">--</span></span></a></li></ul></div></div></div></div><div id="content" data-github-owner="idml" data-github-repo="idml.github.io"><div class="content-wrapper"><section><h1>Basic Usage</h1>

<nav role="navigation" id="toc"></nav>

<p>If you want to embed the library in your service, here’s a short guide on how to use its internal API.</p>

<h2 id="hello-world">Hello world</h2>

<p>Here’s a simple Java “hello world”:</p>

<pre><code>import com.datasift.ptolemy.*;

public class JavaExample {
    public static void main(String[] args) {

        // Initialize the runtime. You only need one of these per application
        Ptolemy ptolemy = new Ptolemy();

        // Parse a simple mapping that turns
        PtolemyMapping mapping = ptolemy.fromString("x = a.b.int()");

        // Parse some json data taken from a string
        PtolemyValue input = PtolemyJson.parse("{\"a\": {\"b\": \"123\"}}");

        // Run a mapping
        PtolemyValue output = mapping.run(input);

        // Pretty-print the output as json
        System.out.println(PtolemyJson.pretty(output));
    }
}
</code></pre>

<h2 id="initialization">Initialization</h2>

<p>The Ptolemy object is the main execution context of the system. It acts as a factory for all other objects and can safely be shared between threads.</p>

<pre><code>Ptolemy ptolemy = new Ptolemy();
</code></pre>

<h2 id="working-with-data">Working with data</h2>

<p>The PtolemyVaue class wraps some queryable data. Right now it supports JSON using Jackson and XML using JSoup.</p>

<pre><code>PtolemyValue input = PtolemyJson.parse("{\"a\": {\"b\": \"123\"}}");
</code></pre>

<p>Basic transformations can be performed on the object itself, but usually this isn’t necessary</p>

<pre><code>PtolemyValue result = input.get("a").get("b").bool();
</code></pre>

<h2 id="mappings">Mappings</h2>

<p>You can create a mapping from a string or use one of the built-in resolvers which will allow you to load from an embedded JAR resource or from the local filesystem.</p>

<pre><code>PtolemyMapping a = ptolemy.fromString("x = b");
PtolemyMapping b = ptolemy.loadResource("my-mapping.ini");
PtolemyMapping c = ptolemy.loadFile("my-mapping.ini");
PtolemyValue output = mapping.run(input);
</code></pre>

<h2 id="chains">Chains</h2>

<p>A chain is a way of composing mappings so that the results of one mapping are transformed into the next, allowing you to split your mapping up into smaller pieces.</p>

<p>This use case evolved from the interation data model at DataSift where we have raw data, for example, Twitter, a sanitized version of that data, and then generalized:</p>

<pre><code>PtolemyChain facebookMapping = ptolemy.newChain(
    // Normalization: The text body of Facebook posts varies depending on whether it's a story, video, etc.
    ptolemy.fromString("facebook.content = this.(text | story | message).string()"),

    // Generalization: interaction.content refers to the body in all types.. Facebook, Twitter, Sina, etc.
    ptolemy.fromString("interaction.content = facebook.content.stripTags()"),

    // Validation: This schema validation is shared by all mappings to make sure that we follow uniform rules
    ptolemy.fromString("interaction.content : string().default(\"Aw damn, there's no tex on this one!\")")
);
// A simplified version of data received from the Facebook Graph API
PtolemyValue facebookPost = PtolemyJson.parse("{\"message\": \"Some message\"}");
// Chains can be run just like any other mapping
PtolemyValue output = facebookMapping.run(facebookPost);
</code></pre>

<p><img src="./diagrams/chain.png" alt="chain diagram" /></p>

<h2 id="mapping-merges">Mapping Merges</h2>

<p>As an alternative to chains, merges allow you to run multiple mappings against the same input data, then merge the output in order.</p>

<p>This allows you to write parallel mappings rather than mappings which are applied end to end.</p>

<pre><code>// Initialize the runtime. You only need one of these per application
Ptolemy ptolemy = new Ptolemy();

// Merge some mappings together into one mapping
Mapping merged = PtolemyMapping.fromMultipleMappings(ptolemy, Arrays.asList(
        ptolemy.fromString("interaction.content = text"),
        ptolemy.fromString("interaction.author = author"),
        ptolemy.fromString("interaction.content = real_text")
));

PtolemyValue input = PtolemyJson.parse(
        "{\"text\": \"an example body\", \"author\": \"bob\", \"real_text\": \"an alternate body\"}"
);

System.out.println(PtolemyJson.pretty(merged.run(input)));
</code></pre>

<p><img src="./diagrams/merge.png" alt="chain diagram" /></p>


<link rel="stylesheet" type="text/css" href="/css/toc.css"></link>
<script type="text/javascript" src="/js/toc.js"></script>
</section></div></div></div></div><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/js/bootstrap.min.js"></script><script src="/highlight/highlight.pack.js"></script><script>hljs.configure({languages:['scala','java','bash']});
hljs.initHighlighting();
              </script><script src="/js/idml-hilight.js"></script><script src="/js/toc.js"></script><script src="/js/main.js"></script></body></html>