amazon web services - Efficiently bulk upserting vertices and edges containing properties in AWS Neptune - Stack Overflow

admin管理员组

文章数量:1431721

Context

We're ingesting a combined total of between (500, 1500, 5000) (P25, P50, P75) vertices and edges per minute. Each vertex has roughly 5 "records" on it (id, label, 3 string properties), and each edge has roughly 3 "records" on it (fromId, toId, label).

When we POC'ed Neptune we didn't add the string properties to the vertices. Following a process of trial-and-error due to insert performance issues, we eventually settled on this solution:

void mergeVertices(final Collection<Vertex> vertices) {
    List<Map<Object, Object>> maps = new ArrayList<>();

    for (Vertex vertex : vertices) {
        Map<Object, Object> map = new HashMap<>();
        map.put(T.id, vertex.id());
        map.put(T.label, vertex.label());
        maps.add(map);
    }

    g.inject(maps).unfold().mergeV().iterate();
}

A similar approach was followed for upserting the edges. However, when we moved passed the POC phase, and we wanted to start adding properties to the vertices, we modified the code to look something like this:

void mergeVertices(final Collection<Vertex> vertices) {
    List<Map<Object, Object>> maps = new ArrayList<>();

    for (Vertex vertex : vertices) {
        Map<Object, Object> map = new HashMap<>();
        map.put(T.id, vertex.id());
        map.put(T.label, vertex.label());
        Map<String, Object> properties = vertex.properties();
        for (Map.Entry<String, Object> entry : properties.entrySet()) {
            map.put(entry.getKey(), entry.getValue());
        }
        maps.add(map);
    }

    g.inject(maps).unfold().mergeV().iterate();
}

We then started receiving exceptions, specifically:

{
"code":"ConstraintViolationException",
"requestId":"<elided>",
"detailedMessage":"Vertex with id already exists: ",
"message":"Vertex with id already exists: "
}

When we reverted those changes, the errors went away. We tested the code-with-property-upsert on a local deployment of TinkerPop Server, and we received no errors.

Question

How can we upsert properties efficiently, without encountering these errors?

The original reason why we did the g.inject(maps) approach, was because we were struggling to get decent insert performance in Neptune, due to our inexperience with the technology, i.e., 30 - 60 second to upsert approx. 200 "records".

Note to reader:

"records" is terminology borrowed from Neptune's documentation.

Context

We're ingesting a combined total of between (500, 1500, 5000) (P25, P50, P75) vertices and edges per minute. Each vertex has roughly 5 "records" on it (id, label, 3 string properties), and each edge has roughly 3 "records" on it (fromId, toId, label).

When we POC'ed Neptune we didn't add the string properties to the vertices. Following a process of trial-and-error due to insert performance issues, we eventually settled on this solution:

void mergeVertices(final Collection<Vertex> vertices) {
    List<Map<Object, Object>> maps = new ArrayList<>();

    for (Vertex vertex : vertices) {
        Map<Object, Object> map = new HashMap<>();
        map.put(T.id, vertex.id());
        map.put(T.label, vertex.label());
        maps.add(map);
    }

    g.inject(maps).unfold().mergeV().iterate();
}

A similar approach was followed for upserting the edges. However, when we moved passed the POC phase, and we wanted to start adding properties to the vertices, we modified the code to look something like this:

void mergeVertices(final Collection<Vertex> vertices) {
    List<Map<Object, Object>> maps = new ArrayList<>();

    for (Vertex vertex : vertices) {
        Map<Object, Object> map = new HashMap<>();
        map.put(T.id, vertex.id());
        map.put(T.label, vertex.label());
        Map<String, Object> properties = vertex.properties();
        for (Map.Entry<String, Object> entry : properties.entrySet()) {
            map.put(entry.getKey(), entry.getValue());
        }
        maps.add(map);
    }

    g.inject(maps).unfold().mergeV().iterate();
}

We then started receiving exceptions, specifically:

{
"code":"ConstraintViolationException",
"requestId":"<elided>",
"detailedMessage":"Vertex with id already exists: ",
"message":"Vertex with id already exists: "
}

When we reverted those changes, the errors went away. We tested the code-with-property-upsert on a local deployment of TinkerPop Server, and we received no errors.

Question

How can we upsert properties efficiently, without encountering these errors?

The original reason why we did the g.inject(maps) approach, was because we were struggling to get decent insert performance in Neptune, due to our inexperience with the technology, i.e., 30 - 60 second to upsert approx. 200 "records".

Note to reader:

"records" is terminology borrowed from Neptune's documentation.

• amazon-web-services
• gremlin
• amazon-neptune
• tinkerpop3

Share Improve this question Follow asked Nov 19, 2024 at 12:34 MarleyMarley 39111 gold badge55 silver badges99 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

When you use mergeV() solely on it's own (without any parameters) it is looking for a 100% complete match (including on properties). When it finds another vertex with the same ID, this creates a constraint violation. One of the few constraints in Neptune is that every ID for a vertex or edge must be unique.

What you need is a merge with a match on the vertex ID. That means you need to include the ID within the mergeV().

One example of doing this is in the TinkerPop docs, where you include the ID as a separate list in the map. Then you unfold the first list in the map inside of the mergeV():

gremlin> g.inject([[(T.id):400],[(T.label):'Dog',name:'Pixel',age:1],[updated:'2022-02-1']]).
           mergeV(limit(local,1)). 
           option(Merge.onCreate,range(local,1,2)). 
           option(Merge.onMatch,tail(local))
==>v[400]

Once you find the match, then you can decide what to do on the create or match. You'll need to include both options, as not including either would be a no-op for a match on the id.

https://tinkerpop.apache./docs/current/reference/#mergevertex-step

本文标签：