This guide walks you through the steps needed to set up a data stream, role, and user, then gather the configuration needed to create an OpenSearch Sink integration.
Before you begin
Log in to the OpenSearch dashboard with a user assigned enough permissions to perform all of the steps listed here. The most straightforward method is to use an admin user. This user is not used for the integration, only for set up.
Create a datastream, user, and role
1. Create a Data Stream
To ingest data into OpenSearch, it is highly encouraged to use a data stream. This will split events into multiple indexes, rolling them automatically as the configured thresholds are reached, ensuring no one index becomes too large. This provides much greater control over the data lifecycle within OpenSearch.
Note: These steps could also be performed via REST API, but this guide provides instruction for using the Dashboard UI.
Open the Dashboard menu and navigate to Management > Dev Tools. Remove all text in the Console and paste in the following request. Replace the :template-name url token with the name you would like to use for your index template. If desired you can also replace the index pattern from logs-synqly-* with a custom one, but make sure it ends with a star. Any time the index template is referred to in this document, update the name with your custom one.
PUT /_index_template/:template-name
{
"index_patterns": [
"logs-synqly-*"
],
"data_stream": {},
"priority": 100,
"template": {
"settings": {},
"mappings": {
"dynamic": "false",
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
],
"date_detection": false,
"properties": {
"@timestamp": {
"type": "date"
},
"agent": {
"properties": {
"build": {
"properties": {
"original": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"ephemeral_id": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
},
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"cloud": {
"properties": {
"account": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"availability_zone": {
"type": "keyword",
"ignore_above": 1024
},
"image": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"instance": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"machine": {
"properties": {
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"project": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"provider": {
"type": "keyword",
"ignore_above": 1024
},
"region": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"component": {
"properties": {
"binary": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "wildcard"
},
"old_state": {
"type": "keyword",
"ignore_above": 1024
},
"state": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"container": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
},
"image": {
"properties": {
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"labels": {
"type": "object"
},
"name": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"data_stream": {
"properties": {
"dataset": {
"type": "keyword"
},
"namespace": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"ecs": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"elastic_agent": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
},
"process": {
"type": "keyword",
"ignore_above": 1024
},
"snapshot": {
"type": "boolean"
},
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"event": {
"properties": {
"dataset": {
"type": "keyword"
}
}
},
"host": {
"properties": {
"architecture": {
"type": "keyword",
"ignore_above": 1024
},
"containerized": {
"type": "boolean"
},
"domain": {
"type": "keyword",
"ignore_above": 1024
},
"hostname": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"ip": {
"type": "ip"
},
"mac": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"os": {
"properties": {
"build": {
"type": "keyword",
"ignore_above": 1024
},
"codename": {
"type": "keyword",
"ignore_above": 1024
},
"family": {
"type": "keyword",
"ignore_above": 1024
},
"kernel": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"text": {
"type": "text"
}
}
},
"platform": {
"type": "keyword",
"ignore_above": 1024
},
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"log": {
"properties": {
"level": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"message": {
"type": "text"
},
"unit": {
"properties": {
"id": {
"type": "wildcard"
},
"old_state": {
"type": "keyword",
"ignore_above": 1024
},
"state": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
},
"aliases": {}
}
}In the console a small 'Play' button will show next to the request. Click this button. If successful, the result pane will show the template has been created:
{
"acknowledged": true
}Update the Console with a new request to create the data stream based on the index template created. Replace the :stream-name token with a name of your choosing. This should match the index pattern used when you created the index template. For example, if using the default "logs-synqly-*" index pattern, you could create a 'default' stream with the stream name "logs-synqly-default".
PUT /_data_stream/:stream-nameClick the "Play" button to make the request. You should see a successful acknowledgement by OpenSearch.
{
"acknowledged": true
}Record your stream name to a safe location.
2. Create a User
Open the menu and navigate to Management > Security. From the security menu on the left click "Internal users". At the top of the list of users on the right hand side, click the "Create internal user" button. Fill out the form supplying a username and password and then click "Create".
Note the username and password for this user in a safe location. This is the user that will authenticate your integration.
3. Create a limited permissions role
If not already in the Security screen, open the menu and navigate to Management > Security. From the security menu on the left click "Roles". Click the 'Create role' button on the right above the list of roles.
Fill out the form giving your role a name. Add the following "Index permissions" to the role:
Index: logs-synqly-*
- Index permissions:
- write
Click 'Create' to create the new role.
4. Assign Roles
From the list of available roles, find the role you created in step 3. Click on the role name to open it.
At the top of the role page, find the tab named 'Mapped users' and click on it. On the right click the 'Manage mapping' button. In the 'Users' section find the user you created in step 2 and select it.
Click the 'Map' button in the bottom right to assign this role to your user.
Configure the Integration
Create your integration by supplying the following configuration values.
URL: This is your base OpenSearch REST API URL without any path segments. For example, "https://search.myopensearch.com:9200".
Create Index: This is data stream name created in step 1.
Credential Username This is the username of the user created in step 2.
Credential Secret This is the password of the user created in step 2.