Open Content Swagger REST-API documentation

The link above assumes that you are running Open Content locally. The api docs can be found at http://localhost:8080/opencontent/apidocs/

REST API for content There is a Swagger REST API available for adding, modifying and deleting content as well as performing various kind queries. The query syntax is following the standard Solr syntax, but is also adding a set of extra comfort functions , like related content.

REST API for admin There is also REST API available for all kind of administration issues, like index, properties, extraction and storage management.

Event log API The events for the last 30 days are recorded and stored in the event log, accessible using the event log api.

Read more about the Open Content Rest API and you could also even try it yourself.

Onboarding We offer onboarding, on location or remote, for Open Content developers to get the most out of the available tools and solutions.

A module to create an event driven workflow. It enables you to listen to changes for specific queries and get notified when the answer for that query is changed.

Notifier can be used to release cache, or send notifications to Live Content Cloud. Possible to use Notifier for any http POST receiving server. Register the url towards which the notification details should be sent.

Documentation about the Open Content Notifier can be found here:

https://naviga-hub.atlassian.net/wiki/spaces/NP/pages/6558221758/Open+Content+Notifier+English

In parallel, we are working on a modern, easy-to-consume JSON based version of the document format. That format will successively replace the NewsML XML format. More info about the new Naviga JSON Document format is found https://app.gitbook.com/@infomaker/s/document-format-v2/.

This exercise shows how to delete objects in Open Content.

• ./delete.sh [uuid] will delete object with the specified uuid

• ./delete-mine.sh will delete objects with source set to lab-$(whoami)

  • the script in this exercise sets the source to lab-$(whoami)

This exercise shows :

• Upload of an article with correct filename, script will insert the uuid and filename for the 3 images

The script ./upload-concepts.sh will upload 6 concepts to Open Content using a curl multipart POST request.

For more details how this is done take a look at the script:

cd ~/oc-lab/lab-newsitem/1-concept
less upload-concepts.sh

​

This exercise shows :

• Calculate the filename when image is used by writer

  • Using openssl

• Create preview and thumb to be used by Open Content

• Create xml metadata file

• Upload of Image with preview, thumb and metadata file

For more details see the ./upload-image.sh file.

When uploading images for Digital Writer the image file needs to be upload to an internal S3 bucket and also be copied to an external S3 bucket with a calculated filename.

What is Open Content?

If you are thinking about creating your own headless CMS, Open Content will fit right in, and will solve several tedious parts of your journey ahead – storage, API’s, scalability, authentication and indexing just to name a few.

Open Content is a handy toolbox: we use it in our own solutions, for example as content backend for our Digital Writer and Newsroom apps as well as powering the Naviga web presentation layer. We also use it in our XLibris archive solution. Our customers uses it to power in-house built presentation solutions.

Together with the Naviga Creation and Presentation tools, Open Content delivers a standardised easy-to-maintain setup. You can also use Open Content as a content agnostic storage and search engine for digital content.

Live Content Cloud

Any digital material can be stored in Open Content using the Open Content REST-API. Open Content configuration makes it possible to group content into Content Type (typically : Article, Image, Page, Concept, Planning, Lists, Packages). Content from different systems can normalised into the same Content Type.

A Content object (item) consists of a primary file and a metadata file describing the the primary file.

Normally xml-metadata files are used to describe the content uploaded and properties are extracted from the meta data files using expressions.

Open Content is configured with a browser based UI or by YAML-files.

Typical use cases for Open Content are:

• Long-time archive with the XLibris search client

A small tool for Open Content API test can be used for easier learning of Open Content search API. The UI can be accessed from the url http://localhost:8800 when Open Content docker-compose is started. T

The configuration in these exercises uses nested properties and assumes that you have the content from lab 1-3 uploaded.

• Property extraction based on relations between content types has been removed. In OC 3.0 version you need to supply all metadata needed for property extraction within the content item itself.

• Query time evaluations of XPath expressions has been removed from the Search API. Previously, if the value of a configured property was not indexed, OC would fetch the document and evaluate the XPath for the missing properties before returning the search result to make sure they were always included. In OC 3.0 only what is indexed will be returned in a search result. If you change the properties config, a reindex of the content is needed.

All info about every Open Content version, release notes, upgrade info, admin guides etc.

This book is intended for anyone managing or integrating with Open Content. If you're new to Open Content, we recommend starting with the . If you're a developer, feel free to jump straight to the .

We urge you to reach out to us at if you have any questions. Certain sections are still incomplete, and in other sections we have yet to define well documented best practices.

The XLibris search client lets you easily search for anything you store. Predefined search modules for content types helps the end user find the right material. There is a facetting feature, to help narrow down the result if it is too big.

Searching in XLibris is fast, even in an archive with over 25 million objects. You can use the query syntax in a really simple manner, just like a Google search. But there is also a powerful query language available behind the scenes if you are interested in power searches.

The developer friendly availability platform A well documented and flexible platform that makes all content available, all the time.

The backend for your headless CMS A headless CMS without a content repository is like an electric car without batteries. Instead of building batteries build your chassis.

Built for Amazon AWS Run Open Content in AWS, then we can handle upgrades and changes with zero downtime with unlimited storage and backup possibilities.

Integrated to Naviga Content solutions Works out-of-the-box with solutions such as Newspilot, Digital Writer, Dashboard and Naviga web.

API’s for everything Use our user interfaces for admin and search, or use the OC REST API:s. Regardless of approach, it’s all open for integration.

Reliable backend Spend less time on server issues and let us manage the hosting. Open Content supports a range of different setups, from a small single-node setup to large, clustered, high availability setups.

OC Concepts is a metadata structure, built around the IPTC NewsMLG2 standard. One of the most important parts of that is of course how to use it. For the editor, the developer as well as the end user. All concepts are stored and made available in Open Content.

In our view, metadata like categories and tags are not just text strings. Instead, each metadata is an object – each with a unique id, name and its own set of metadata and links.

Like an author. It could be just a name, But when you think of it as an object with a unique id, first name, last name, email, phone, description, avatar image, high res image and links things get really powerful.

These can be shown in your frontend if you want to, for example when showing articles for a specific category on a search page could then show the long description or image for that category.

Examples of Concepts:

• Author

• Category

• Persons

• Organisations

• Topics

• Places (poI:s or geo areas)

• Story

• Functional tags

The concepts are administered using our Dashboard application, your journalists use Digital Writer to choose the right concepts, and Everyware and the App Platform will show and let the user follow selected topics or geo areas.

The exercises can be downloaded from

# In terminal do 
cd ~/oc-lab
mkdir lab-newsitem
cd lab-newsitem
curl -s https://s3-eu-west-1.amazonaws.com/open-content-artifacts/lab-newsitem.zip --output lab-newsitem.zip

# this will download the lab-newsitem.zip, unzip it 
unzip lab-newsitem.zip

Structure of lab-newsitem dir

lab-newsitem/
├── 0-config
│   ├── configure.sh
│   └── lab-newsml-config.yml
├── 1-concept
│   ├── 29889da3-e930-4846-a12b-096508e1054d
│   ├── 8c7437ce-a7ca-414d-8bfc-7bf2d1054fc3
│   ├── 9197a3ea-9624-404a-aef5-4d80eaadc99f
│   ├── b7399f0c-fb3d-4a4f-b849-9935a77d9512
│   ├── db09e859-43d4-42f8-a6ca-c810b653ec6a
│   ├── fb5911fa-b97f-436e-83f7-de7f7a203ea9
│   ├── upload-concepts.sh
│   └── uuids
├── 2-upload-image
│   ├── image-template.xml
│   ├── one.jpg
│   ├── one.jpg.uuid
│   ├── three.jpg
│   ├── three.jpg.uuid
│   ├── two.jpg
│   ├── two.jpg.uuid
│   └── upload-image.sh
├── 3-upload-article
│   ├── article.xml
│   └── upload-article.sh
├── 4-search
│   └── readme.md
├── 5-delete
│   ├── delete-mine.sh
│   └── delete.sh
├── 6-event-sourcing
│   └── listen.sh
├── build.sh
├── lab-newsitem.zip
├── readme.md
└── settings

Settings file

The settings file holds information about the host, user and password for the Open Content to be used with the script in exercises directories. Update the settings file to the Open Content you will use.

This lab configures Open Content using editorial standard config

The script ./configure.sh in ~/oc-lab/opencontent-configs will configure Open Content:

cd ~/oc-lab/opencontent-configs/scripts

./configure.sh \
http://admin:admin@localhost:8080/opencontent \
editorial

Verify the configuration in Open Content admin UI (http://localhost/admin)

Activate the config either using the admin UI or curl below:

curl -u admin:admin \
-X POST "http://localhost:8080/opencontent/admin/configuration/activate" \
-H "accept: */*" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "reason=configured from script&name=$(whoami)"

Examine the + Configuration menu and more in Open Content admin UI

• History

• Compare (remove something)

• Import/Export

• Undo configuration

Open Content Replicator is a module that allows the Open Content to replicate content to an Open Content Satellite. The OC Satellite works as a "read only" and can store anything from all of the information in the Master to a part of it.

The Replicator can be run automatically in near real time and/or be triggered manually in batches.

For example, a satellite can consist of content with a specific setup of meta data (products, categories etc.) and another satellite can have a different content

Open Content Replicator is in this environment:

http://localhost:8180/replication

The replicator can be used to replicate objects from one Open Content to another Open Content.

Different types of replication exists:

• Full replication; replicate objects using a query

• Incremental replication; replicate object on incremental re-indexing event. Uses RabbitMQ Indexer needs to be configured for this.

  • used for replication between editorial and public Open Content.

• Batch replication; replicate objects on batch re-indexing event. Uses RabbitMQ Indexer needs to be configured for this.

  • almost never used

• Partial replication; updates target Open Content for filter changes

  • not used

• Event-log replication; polls event-log or content-log

The Docker images for Open Content are primarily for development purposes, not production. So if you are a developer looking for how to start Open Content locally for integration testing or trying things out, then this is for you.

# create a directory where to work, in home oc-lab 
cd 
mkdir oc-lab
cd oc-lab

Start Open Content

Wait until all containers are downloaded and started. Now there is an empty Open Content without configuration or content.

Logging

Open Content UI

Configuration is done using the admin UI or the admin API. The UI can be found here .

Below is the menu for the Open Content admin UI.

Configure Open Content

The first thing that has to be configured is storage. This can either be done in the UI at or with this curl command:

Open Content configuration in this setup is done using a local copy of our Bitbucket repository for configuration. Use the Open Content admin UI to inspect the detailed settings for the different configuration options.

Go to the opencontent-configuration directory where the configure.sh script is

Configure Open Content for public use

Configure Open Content for public and app use

Configure Open Content for editorial use

Activation of the configuration

The Naviga content could act as a standard end-to-end solution. You use our standard authoring setup in combo with our solutions for presentation on the web and in mobile apps. In that case, we are managing everything from setup, configuration, hosting, support etc. You are still able to interact with the backend, but we recommend to use our more high-level API:s for content creation (like ingestion of content) instead of using the more low level OC API.

You can also use Naviga content solutions as a headless CMS, and build your own presentation layer. In that case, we recommend to use our content distribution API to power your own presentation solution. You may also use the more low level OC REST API to power your presentation layer. The distribution API also offers a cache solution. If you use the OC REST API, you need to add your own cache mechanism between OC and your presentation engine. It's possible to just scale up the read capacity of Open Content, but that will be a quite expensive solution in most cases.

Both solutions uses a separate Open Content for production, and one to power presentation layers. When a content item, like an article, is ready to be published (useable) it's copied to the public content repo by the Replicator service.

Scaling Open Content doesn’t mean the same thing for all our customers. Some need a massive index, others have need for a smaller index but lots of traffic or API calls. Regardless of your needs, we are confident we will solve them.

The Open Content stack consists of several parts, all running in the Amazon cloud.

• Load Balancer. The OC stack uses the standard Amazon application load balancers.

• OC API. Is the REST API for queries, read and write, as well as the OC Admin API. Runs in ECS and scales horizontally.

• S3 is the storage where all content items are stored.

We always recommend a multi-AZ setup for all parts of the stack. That means the Open Content stack is running on multiple datacenters in parallel, enabling high availability.

For Open Content pre 3.0, you'll need to use the master-satellite mechanism (see below) to reach multi-AZ redundancy.

When using Open Content as a creation backend, we always use a Satellite Open Content for the presentation layers. The production and presentation is totally separated each of them can be configured and scaled in the appropriate way.

We recommend to use the Naviga standard configuration for Creation and Presentation. They are both versioned and maintained by Naviga, and are updated when needed to be in sync with the Naviga Creation and Presentation tools.

Master - satellite In complex environments setting up multiple Open Content Satellites might be a suitable way to scale. All content is stored in an Open Content Master setup, and predefined replication rules make sure the correct content is available in each Satellite. This does not require additional storage, they are setup as read-only OC’s, reading the content from the same S3 bucket, saving both time and money. As content can differ each Satellite maintains its own index.

This section will show how an upload to Open Content is performed

• To be able to do the exercises you may to prepare your system. You need an Open Content Server to perform upload request towards.

• You need a bash terminal for execution of the scripts

  • The prepare Windows section explains how to enable a bash terminal for Windows

• Need to have following installed;

  • aws cli

  • imagemagick

  • unzip

The examples will use show how to upload all objects types referenced from an Article.

• Concepts

• Images

• Article

The article has relations to 6 different concepts and 3 images. Certain conventions must be known before uploading these to an Open Content.

UI and REST-API documentation for Open Content

• Search client (XLibris)

• Admin client,

• Rest-Api Swagger documentation,

The 3.0 version of Open Content is a major upcoming release. A lot of effort, on all levels, have been put into the areas of increased performance, scalability and availability. Many pieces have been optimised, rewritten or redesigned. The APIs are still the same, except for a few functions that has been deprecated.

SolrCloud support Running one single instance of Solr means that you have one single index running on one Solr node. Even if we have quick restore processes, that’s not a redundant solution. With the 3.0 version we have standardized a multi node SolrCloud setup as an option to the standard setup.

The SolrCloud setup runs in a Kubernetes () cluster, starting with 3 Solr nodes plus the necessary orchestration mechanisms. The Solr version used in the 3.0 version is 8.x.x.

In this exercise you will start a script which will poll the Open Content event log every 5th second. If any events are found the script will print information of the event.

The script will persist the last event to a file (lastevent) which holds the last event id processed by the script. When started next time and lastevent file exists the script will start processing events with id larger than the lastevent processed . This means that even if the listener is off. It will continue from last event next time the listener is started. This way no event is missed.

This is how to get the /eventlog/ endpoint response:

Response

Get the last event id use event=-1

The event log tells you what has happened after a last known event. Depending on your use-case you can either process the eventlog from the beginning (it keeps a history of one month), or start at the last event. It's useful to process all retained events if you want to prepopulate a cache, but if you just need it for invalidation of a cache that you start cold and build ad-hoc it makes more sense to start with the last event.

A request to the eventlog looks like this: GEThttps://oc.tryout.infomaker.io:8443/opencontent/eventlog If called without any query parameters you get events from the start of the log:

If you pass in a negative value, like so GET https://oc.tryout.infomaker.io:8443/opencontent/eventlog?event=-2, you get the last -N events in the log.

The id attribute in the events can be used to paginate though the eventlog. So if we have processed events up until 406374 we would ask the eventlog for all events after it, like so GET https://oc.tryout.infomaker.io:8443/opencontent/eventlog?event=406374:

To fetch the updated object the normal objects endpoint is used GET

To get a log that covers all content in OC you must use the contentlog instead (). It comes with some other trade-offs though as it only contains the last event for every object and it doesn't publish any delete events. That means that it is useful for bootstrapping f.ex. a cache with a full data set, but not very useful for invalidation.

The contentlog event id is not consistent when migrating to a new version/install of OC either (it gets "compacted"), so depending on it as state for long-running tasks is not recommended.

Wildfly is deployed as a container in Amazon ECS

In 2.3 the Wildfly service is available as a container image. When installing Open Content in Amazon AWS the Wildfly Service is deployed on Amazon ECS. The Wildfly instances belong to a private subnet and will not get public IPs. Access to the API has to be done through the public load balancer.

XLibris and OC Admin are also available as container images. They are using the latest versions of PHP and Apache httpd. When using Amazon ECS XLibris and OC Admin are deployed on their own EC2 instances without public IPs. The public loadbalancer is the only way to access the services.

API Cache

When Open Content is deployed on Amazon AWS an API cache can be put in front of Wildfly. It is running on port 9999 while Wildfly is running on 8080. When a client wants to use the cache it must use port 9999. Because both ports are open it's always possible to directly talk to Wildfly even if the cache is enabled.

Validation of properties on add and update

Until now there has been no validation of properties when adding content. For example if an article is uploaded with a property "TextLength" of type Integer but contains a string instead, then the upload still succeeds.

In 2.3 properties are validated according to their type. Validation is performed on adds and updates of content and if it fails it results in HTTP 400 Bad Request.

New property type called WKT (Well Known Text)

Until now latitude, longitude and spatial geometries had to be of property type "String". There has been no validation of the string until it has been indexed into Solr. When Solr refuses to accept the invalid WKT an entry has been added to the indexer error log. But the indexing happens in the background and the user who uploaded the content may never discover that something was wrong with the content that was uploaded.

In 2.3 there is a new property type called "WKT". This property is validated when content is either added or updated. If the property contains an invalid WKT string then the add or update will return HTTP 400 Bad Request.

Failing extractors are no longer suppressed

An XPath can be valid but throw an error anyway depending on the text it's applied on. Until now Open Content has silently suppressed these errors. A result of this is that content can be indexed while one or two properties are missing from the index.

In 2.3 no extractors are suppressed. If an XPath fails at content upload time the upload will respond with a HTTP 400 Bad Request. If an XPath fails at indexing time the content will not get indexed at all.

Fall back to a default dynamic path / if none is configured

Until now it has been mandatory to configure dynamic path for the storage.

In 2.3 the absence of a dynamic path will lead to content being placed in the root path. So for example if S3 is used for storage and no dynamic path is configured, then the uuid will be be in the root of the S3 bucket with no prefix added.

Upgrade to Solr 7.7

Open Content has used version 5.5 of Solr for a long time. In 2.3 it has been upgraded to . This has the implication that all content has to be reindexed.

A backward incompatible change is that it is not possible to do a query like Pubdate:* to get all content that has a Pubdate. That query needs to be changed to Pubdate:[* TO *]. The reason is that Solr has deprecated TrieDateField for dates so Open Content is using DatePointField instead. This is true for index fields of type date, int, long, float and double.

Open Content does not deliver a clustered Solr out of the box yet, but it has been prepared by using SolrCloud and allowing multiple configured ZooKeeper.

Update Swagger apidocs to OpenAPI 3.0

The Swagger support has been reimplemented from scratch and has a couple of improvements.

The specification is updated from Swagger 1.2 to OpenAPI 3.0.

OpenAPI specification for Open Content is now generated at release time and is a static file so loading Swagger is much faster.

Swagger UI is updated to the latest version which is why the look and colours have changed.

Many REST API documentation errors have been fixed.

Upgrade to Java 11

Until now Wildfly, Indexer, Notifier and Replicator have been using Oracle's distribution of OpenJDK 8.

In 2.3 all Java based services use 's distribution of OpenJDK 11.

Upgrade to Wildfly 15

Has support for Java 11.

Upgrade Saxon and get support for XPath 3.1

In the effort to keep as many 3rd party libraries as up-to-date as possible Saxon has also been updated. This means that XPath and XSLT extractors now do support XPath 3.1 and XSLT 3.0.

Nested properties API call may result in too long Solr GET request

When using nested properties Wildfly makes HTTP GET requests to Solr. In some circumstances the URL length hits a limit and Solr refuses the requests. In order to get around this Wildfly now sends the requests to Solr using HTTP POST instead.

Sortable flag on index fields is removed

Sorting in Solr is memory intensive. The sortable flag has been a not-perfect guard against out of memory errors in Solr. By not allowing to sort on just any arbitrary number of fields there has been some sort of protection.

In 2.3 docValues have been enabled for many index field types. This means memory consumption is lower when sorting on these index fields. Therefore the not-perfect guard (sortable flag) has been removed.

Open Content will ignore the sortable flag when it reads the old configuration and the next time the configuration is activated the sortable flag will not be in oc.yaml anymore.

Support for HTTPS in replicator

Until now the replicator has only been able to replicate using HTTP. Now in Open Content 2.3 the replicator can replicate content using HTTPS.

Deprecated and will be removed in 3.0

In a microservice world we want to split out the search and suggest functionality to its own service. The Wildfly service will not use Solr anymore. When using the Wildfly API you will know that you get the source of truth. When using the Search API you know that only Solr will be asked for data. Because Wildfly will not ask an eventually consistent search engine for data anymore some small parts of Wildfly will be backwards incompatible.

Upload is a multipart request towards Open Content /opencontent/objectupload endpoint.

Curl

Below is how a curl request for upload can look like:

Open Content is used to power the XLibris Archive. When importing content to an Open Content based archive you have two main options:

• Convert and migrate all your content items to the Content NewsML format used by all Naviga Creation and Presentation tools. Of course, binary artefacts are more or less just copied to the OC Storage, but all meta data files, articles, meta data etc needs to be migrated. Depending of the quality and format used in your old content that can be a really massive work, or not. Contact us to discuss the scope of your migration. The benefit of doing so is that your content is more future proof, streamlined to one, well known format. Content items can more easily be reused. Standard configuration can be used.

• The other option is more like a "copy" of the content to the OC Storage, and then create a configuration that adapts to your content. You still need your articles and meta data side car files in XML format. If you don't have that you need to migrate your content anyway. The advantage with this model is lower migration costs. On the other side, you'll have a more complex, customised configuration. Your content is still in the original format. You'll not be able to reuse content items in the same easy way as form migrated content.

Consult us to discuss what's best in your specific case.