************ Introduction ************ This walkthrough adopts a practical approach that we hope will help our readers to get familiar with NGSI-LD in general and the Scorpio Broker in particular - and have some fun in the process :). The walkthrough is based on the NGSI-LD Specification, that can be found in here [https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.02.02_60/gs_CIM009v010202p.pdf]. --> will become gs_CIM009v010301p.pdf soon ... You should also have a look at the NGSI-LD implementation notes. --> once they are available To get familiar with NGSI-LD, you may also have a look at the NGSI-LD Primer [https://www.etsi.org/deliver/etsi_gr/CIM/001_099/008/01.01.01_60/gr_CIM008v010101p.pdf] that is targeted at developers. The main section is about context management. It describes the basic context broker functionality for context management (information about entities, such as the temperature of a car). Context source management (information not about the entities themselves, but about the sources that can provide the information in a distributed system setup) is also described as part of this document. It is recommended to get familiar with the theoretical concepts on which the NGSI-LD model is based before starting. E.g. entities, properties, relationships etc. Have a look at the FIWARE documentation about this, e.g. this public presentation. [... find suitable presentation] Starting the Scorpio Broker for the tutorials ############################################# In order to start the broker we recommend to use docker-compose. Get the docker-compose file from the github repo of Scorpio. :: curl https://raw.githubusercontent.com/ScorpioBroker/ScorpioBroker/development/docker-compose-aaio.yml and start the container with :: sudo docker-compose -f docker-compose-aaio.yml up You can also start the broker without docker. For further instructions please refer to the readme https://github.com/ScorpioBroker/ScorpioBroker/blob/development/README.md Issuing commands to the broker ############################## To issue requests to the broker, you can use the curl command line tool. curl is chosen because it is almost ubiquitous in any GNU/Linux system and simplifies including examples in this document that can easily be copied and pasted. Of course, it is not mandatory to use it, you can use any REST client tool instead (e.g. RESTClient). Indeed, in a real case, you will probably interact with the Scorpio Broker using a programming language library implementing the REST client part of your application. The basic patterns for all the curl examples in this document are the following: For POST: curl localhost:9090/ngsi-ld/v1/ -s -S [headers]' -d @- < -s -S [headers] -X PUT -d @- < -s -S [headers] -X PATCH -d @- < -s -S [headers] For DELETE: curl localhost:9090/ngsi-ld/v1/ -s -S [headers] -X DELETE Regarding [headers] you have to include the following ones: Accept header to specify the payload format in which you want to receive the response. You should explicitly specify JSON or JSON-LD. curl ... -H 'Accept: application/json' ... or curl ... -H 'Accept: application/ld-json' depending on whether you want to receive the JSON-LD @context in a link header or in the body of the response (JSON-LD and the use of @context is described in the following section). If using payload in the request (i.e. POST, PUT or PATCH), you have to supply the Context-Type HTTP header to specify the format (JSON or JSON-LD). curl ... -H 'Content-Type: application/json' ... or -H 'Content-Type: application/ld+json' In case the JSON-LD @context is not provided as part of the request body, it has to be provided as a link header, e.g. curl ... -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json" where the @context has to be retrievable from the first URI, i.e. in this example: https://uri.etsi.org/ngsi-ld/primer/store-context.jsonld Some additional remarks: Most of the time we are using multi-line shell commands to provide the input to curl, using EOF to mark the beginning and the end of the multi-line block (here-documents). In some cases (GET and DELETE) we omit -d @- as no payload is used. In the examples, it is assumed that the broker is listening on port 9090. Adjust this in the curl command line if you are using a different port. In order to pretty-print JSON in responses, you can use Python with msjon.tool (examples along with tutorial are using this style): (curl ... | python -mjson.tool) <; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json" Context Management ################## To show the use of @context, most examples in this tutorial will be done as application/ld+json having the @context entries in the body of the payload. At the end of this section, you will have the basic knowledge to create applications (both context producers and consumers) using the Scorpio Broker with context management operations. *************** Entity creation *************** Assuming a fresh start we have an empty Scorpio Broker. First, we are going to create house2:smartrooms:room1. Let's assume that at entity creation time, temperature is 23 ?C and it is part of smartcity:houses:house2. :: curl localhost:9090/ngsi-ld/v1/entities -s -S -H 'Content-Type: application/ld+json' -d @- <; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' The reply now looks like this. :: { "id": "smartcity:houses:house2", "type": "House", "hasRoom": [{ "type": "Relationship", "object": "house2:smartrooms:room1", "datasetId": "somethingunique1" }, { "type": "Relationship", "object": "house2:smartrooms:room2", "datasetId": "somethingunique2" }], "location": { "type": "GeoProperty", "value": { "type": "Polygon", "coordinates": [[[-8.5, 41.2], [-8.5000001, 41.2], [-8.5000001, 41.2000001], [-8.5, 41.2000001], [-8.5, 41.2]]] } }, "entrance": { "type": "GeoProperty", "value": { "type": "Point", "coordinates": [-8.50000005, 41.2] } }, "@context": [ "https://pastebin.com/raw/Mgxv2ykn" ] } Since we provide the core context in our own @context it is not added to the result. From here on we will use the custom @context so we can use the short names in all of our requests. You can also request an entity with a single specified attribute, using the attrs parameter. For example, to get only the location: :: curl localhost:9090/ngsi-ld/v1/entities/smartcity%3Ahouses%3Ahouse2/?attrs=location -s -S -H 'Accept: application/ld+json' -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' Response: :: { "id": "smartcity:houses:house2", "type": "House", "location": { "type": "GeoProperty", "value": { "type": "Polygon", "coordinates": [[[-8.5, 41.2], [-8.5000001, 41.2], [-8.5000001, 41.2000001], [-8.5, 41.2000001], [-8.5, 41.2]]] } }, "@context": [ "https://pastebin.com/raw/Mgxv2ykn" ] } Query ##### The second way to retrieve information is the NGSI-LD query. For this example we first add a new Room which belongs to another house. :: curl localhost:9090/ngsi-ld/v1/entities -s -S -H 'Content-Type: application/ld+json' -d @- <; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' Note that this request has the accept header application/json, i.e. the link to the @context is returned in a link header. The result is :: [ { "id": "house2:smartrooms:room1", "type": "Room", "temperature": { "value": 23, "unitCode": "CEL", "type": "Property", "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor0815" } }, "isPartOf": { "type": "Relationship", "object": "smartcity:houses:house2" } }, { "id": "house2:smartrooms:room2", "type": "Room", "temperature": { "value": 21, "unitCode": "CEL", "type": "Property" "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor4711" } }, "isPartOf": { "type": "Relationship", "object": "smartcity:houses:house2" } }, { "id": "house99:smartrooms:room42", "type": "Room", "temperature": { "value": 21, "unitCode": "CEL", "type": "Property", "providedBy": { "type": "Relationship", "object": "smartbuilding:house99:sensor36" } }, "isPartOf": { "type": "Relationship", "object": "smartcity:houses:house99" } } ] Filtering ######### NGSI-LD provides a lot of ways to filter Entities from query results (and subscription notifications respectively). Since we are only interested in our smartcity:houses:house2, we are using the 'q' filter on the Relatioship isPartOf. (URL encoding "smartcity:houses:house2" becomes %22smartcity%3Ahouses%3Ahouse2%22) :: curl localhost:9090/ngsi-ld/v1/entities/?type=Room\&q=isPartOf==%22smartcity%3Ahouses%3Ahouse2%22 -s -S -H 'Accept: application/json' -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' The results now looks like this. :: [ { "id": "house2:smartrooms:room1", "type": "Room", "temperature": { "value": 23, "unitCode": "CEL", "type": "Property", "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor0815" } }, "isPartOf": { "type": "Relationship", "object": "smartcity:houses:house2" } }, { "id": "house2:smartrooms:room2", "type": "Room", "temperature": { "value": 21, "unitCode": "CEL", "type": "Property" "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor4711" } }, "isPartOf": { "type": "Relationship", "object": "smartcity:houses:house2" } } ] Now an alternative way to get the same result would be using the idPattern parameter, which allows you to use regular expressions. This is possible in this case since we structured our IDs for the rooms. :: curl localhost:9090/ngsi-ld/v1/entities/?type=Room\&idPattern=house2%3Asmartrooms%3Aroom.%2A -s -S -H 'Accept: application/json' -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' (house2%3Asmartrooms%3Aroom.%2A == house2:smartrooms:room.*) Limit the attributes #################### Additionally we now want to limit the result to only give us the temperature. This is done by using the attrs parameter. Attrs takes a comma seperated list. In our case since it's only one entry it looks like this. :: curl localhost:9090/ngsi-ld/v1/entities/?type=Room&q=isPartOf==%22smartcity%3Ahouses%3Ahouse2%22\&attrs=temperature -s -S -H 'Accept: application/json' -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' :: [ { "id": "house2:smartrooms:room1", "type": "Room", "temperature": { "value": 23, "unitCode": "CEL", "type": "Property", "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor0815" } } }, { "id": "house2:smartrooms:room2", "type": "Room", "temperature": { "value": 21, "unitCode": "CEL", "type": "Property" "providedBy": { "type": "Relationship", "object": "smartbuilding:house2:sensor4711" } } } ] KeyValues results ################# Now assuming we want to limit the payload of the request even more since we are really only interested in the value of temperature and don't care about any meta information. This can be done using the keyValues option. KeyValues will return a condenced version of the Entity providing only top level attribute and their respective value or object. :: curl localhost:9090/ngsi-ld/v1/entities/?type=Room\&q=isPartOf==%22smartcity%3Ahouses%3Ahouse2%22\&attrs=temperature\&options=keyValues -s -S -H 'Accept: application/json' -H 'Link: ; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' Response: :: [ { "id": "house2:smartrooms:room1", "type": "Room", "temperature": 23 }, { "id": "house2:smartrooms:room2", "type": "Room", "temperature": 21 } ] ******************************************* Updating an entity & appending to an entity ******************************************* NGSI-LD allows you to update entities (overwrite the current entry) but also to just append new attributes. Additonally you can of course just update a specific attribute. Taking the role of the Context Producer for the temperature for house2:smartrooms:room1 we will cover 5 scenarios. 1. Updating the entire entity to push new values. 2. Appending a new Property providing the humidity from the room. 3. Partially updating the value of the temperature. 4. Appending a new multi value entry to temperature providing the info in degree Kelvin 5. Updating the specific multi value entries for temperature and Fahrenheit. Update Entity ############# You can basically update every part of an entity with two exceptions. The type and the id are immutable. An update in NGSI-LD overwrites the existing entry. This means if you update an entity with a payload which does not contain a currently existing attribute it will be removed. To update our room1 we will do an HTTP POST like this. :: curl localhost:9090/ngsi-ld/v1/entities/house2%3Asmartrooms%3Aroom1 -s -S -H 'Content-Type: application/json' -H 'Link: https://pastebin.com/raw/Mgxv2ykn' -d @- </attrs/ In order to update the temperature we do a POST like this :: curl localhost:9090/ngsi-ld/v1/entities/house2%3Asmartrooms%3Aroom1/attrs/temperature -s -S -H 'Content-Type: application/json' -H 'Link: https://pastebin.com/raw/Mgxv2ykn' -d @- </attrs/ with the new attribute as payload. Append in NGSI-LD by default will overwrite an existing entry. If this is not desired you can add the option parameter with noOverwrite to the URL like this /entities//attrs?options=noOverwrite. Now if we want to add an additional entry for the humidity in room1 we do an HTTP PATCH like this. :: curl localhost:9090/ngsi-ld/v1/entities/house2%3Asmartrooms%3Aroom1/attrs -s -S -X PATCH -H 'Content-Type: application/json' -H 'Link: https://pastebin.com/raw/Mgxv2ykn' -d @- <:@]:[]/[[/]...] So a subscription would generally look like this. :: curl localhost:9090/ngsi-ld/v1/subscriptions -s -S -H 'Content-Type: application/ld+json' -d @- <; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' we will get back our original registration and everything that has been registered with the type Room. Context Registry usage for normal queries & subscriptions ######################################################### A context registry entry can have multiple entries which are taken into consideration when normal queries or subscriptions arrive in Scorpio. As you can see there is an entities entry similar to the one in the subscriptions. This is the first thing to be taken into consideration. If you register a type, Scorpio will only forward a request which is matching that type. Similarly the location is used to decide if a query with geo query part should be forwarded. While you shouldn't overdo it, the more details you provide in a registration the more efficiently your system will be able to determine to which context source a request should be forwarded to. Below you see an example with more properties set. :: { "id": "urn:ngsi-ld:ContextSourceRegistration:csr1a3459", "type": "ContextSourceRegistration", "name": "NameExample", "description": "DescriptionExample", "information": [ { "entities": [ { "type": "Vehicle" } ], "properties": [ "brandName", "speed" ], "relationships": [ "isParked" ] }, { "entities": [ { "idPattern": ".*downtown$", "type": "OffStreetParking" } ] } ], "endpoint": "http://my.csource.org:1026", "location": "{ \"type\": \"Polygon\", \"coordinates\": [[[8.686752319335938,49.359122687528746],[8.742027282714844,49.3642654834877],[8.767433166503904,49.398462568451485],[8.768119812011719,49.42750021620163],[8.74305725097656,49.44781634951542],[8.669242858886719,49.43754770762113],[8.63525390625,49.41968407776289],[8.637657165527344,49.3995797187007],[8.663749694824219,49.36851347448498],[8.686752319335938,49.359122687528746]]] }" } There are two entries in the information part. In the first you can see there are two additional entries describing the two properties and one relationship provided by that source. That means any query which asks for type Vehicle, without an attribute filter, will be forwarded to this source and if there is an attribute filter it will only be forwarded if the registered properties or relationships match. The second entry means that this source can provide Entities of type OffStreetParking, which have an Entity ID ending with "downtown".