Seamless Integration of Amazon API Gateway with AWS IoT Core

Overview

In the evolving landscape of IoT applications, integrating Amazon API Gateway with AWS IoT Core offers a powerful way to securely publish device data over HTTP while leveraging Amazon API Gateway’s rich features.

However, handling MQTT topics with hierarchical names (like ‘gtu/ret’) poses unique challenges, especially around URL encoding and request routing.

In this blog, we dive deep into how to correctly configure Amazon API Gateway to publish messages to AWS IoT Core topics contain slashes, explore common pitfalls, and share practical configuration steps with screenshots from a real-world setup.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Understanding the Problem

AWS IoT Core uses MQTT topics with a hierarchical structure, separated by slashes (/). For example, ‘gtu/ret’ represents a topic named ‘ret’ under the parent ‘gtu’. When publishing via HTTPS to AWS IoT Core’s data-plane endpoint, the topic must be URL-encoded in the request path. This means slashes are encoded as ‘%2F’, so publishing to ‘gtu/ret’ requires the path to include ‘gtu%2Fret’.

However, when Amazon API Gateway acts as the frontend, it tends to decode these encoded slashes before forwarding the request, breaking the expected format and causing 403 Forbidden errors.

Why Does This Happen?

One of the subtle yet critical behaviors of Amazon API Gateway is its handling of URL-encoded path variables. When a client sends an HTTP request to API Gateway with an encoded topic path, such as:

POST /register/gtu%2Fret

1	POST /register/gtu%2Fret

The encoded string %2F (representing a forward slash /) would be expected to be preserved and passed as-is to the backend integration, such as AWS IoT Core. However, this is not what happens in practice.

Amazon API Gateway automatically decodes all percent-encoded characters in path parameters before forwarding the request to the integration target. As a result, the encoded path /register/gtu%2Fret is interpreted as /register/gtu/ret. The %2F is converted back to a literal slash, effectively splitting what was meant to be a single string (gtu/ret) into multiple path segments (gtu and ret).

This behavior creates a serious issue when integrating with AWS IoT Core, which treats MQTT topics as flat string identifiers that may contain slashes. When publishing to the data-plane endpoint (e.g., https://<endpoint>.iot.<region>.amazonaws.com/topics/gtu%2Fret), AWS IoT Core expects the entire topic name to remain encoded in the request path. The premature decoding by Amazon API Gateway leads to an invalid topic structure, or worse, causes the topic to be interpreted incorrectly by IoT Core.

As a result of this mismatch, the request often fails authorization checks. The IAM policy associated with the API Gateway integration (typically using a wildcard like arn:aws:iot:<region>:<account-id>:topic/*) no longer matches the malformed topic path. This leads to a 403 Forbidden error, a common symptom of the decoding issue.

In essence:

Amazon API Gateway decodes encoded slashes (%2F) into literal slashes (/)
This alters the topic structure expected by AWS IoT Core
AWS IAM authorization fails because the decoded path no longer matches the policy’s expectations

Solution

Step 1: Define Amazon API Gateway Resource with Greedy Path Variable ‘{topic+}’

To correctly forward hierarchical MQTT topics that contain slashes (e.g., gtu/ret) from Amazon API Gateway to AWS IoT Core, it is essential to use a greedy path variable in the API resource definition. A greedy path variable captures multiple path segments, including slashes, into a single variable. This is critical for maintaining the full structure of MQTT topics.

In Amazon API Gateway, a greedy path variable is defined using a + suffix, like {topic+}. Unlike a standard path variable (e.g., {topic}), which captures a single segment, the {topic+} variable captures everything that follows in the path, even if it includes slashes (/).

Create a resource path like ‘/register/{topic+}’, where ‘{topic+}’ captures the entire path segment, including slashes.

step1

Step 2: Configure Integration Request with Path Override

Use the AWS Service Integration for IoT Data, choose the ATS endpoint obtained from IoT core as the AWS Subdomain, and set the Path Override as: ‘topics/{topic}’. Map the integration request path parameter ‘topic’ from the method request path parameter ‘topic’.

step2

step2b

step2c

Create an integration mapping template.

step2d

Step 3: Ensure the AWS IAM Role Has Proper Permissions

Make sure the AWS IAM role Amazon API Gateway uses has a policy allowing publishing to all topics, including hierarchical ones:

{
  "Effect": "Allow",
  "Action": "iot:Publish",
  "Resource": "arn:aws:iot:ap-south-1:<account-id>:topic/*"
}

{

"Effect": "Allow",

"Action": "iot:Publish",

"Resource": "arn:aws:iot:ap-south-1:<account-id>:topic/*"

}

step3

Step 4: Deploy and Test

Send a POST request to your API like: ‘POST https://<api-id>.execute-api.ap-south-1.amazonaws.com/dev/register/gtu/ret’ with JSON body ‘{“humidity”: 80}’. The MQTT client subscribed to appropriate topics will receive the message.

step4

Viewing the Exact IoT Core Response

To see the raw response from IoT Core in your Amazon API Gateway response, add an Integration Response Mapping Template like:

{
  "message": "published",
  "timestamp": "$context.requestTime",
  "topic": "$input.params('topic')",
  "iot_response": $input.body
}

{

"message": "published",

"timestamp": "$context.requestTime",

"topic": "$input.params('topic')",

"iot_response": $input.body

}

step5

This way, you get detailed feedback from IoT in your HTTP response.

Conclusion

Hierarchical MQTT topics with slashes must be handled carefully by the Amazon API Gateway.
Use greedy path variables ‘{topic+}’ and map integration path parameters properly.
Allow broad or scoped IoT publish permissions via AWS IAM.
Let Amazon API Gateway handle URL encoding, and clients should send natural slashes.
Verify integration response mapping to see detailed IoT feedback.

With this setup, you can efficiently integrate Amazon API Gateway with AWS IoT Core for scalable IoT solutions.

Drop a query if you have any questions regarding Amazon API Gateway and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

FAQs

1. Why do I get a 403 Forbidden error when publishing IoT topics with slashes via Amazon API Gateway?

ANS: – Because Amazon API Gateway decodes URL-encoded slashes, causing AWS IoT Core to reject the malformed topic path.

2. How can I publish messages to hierarchical topics like gtu/ret using Amazon API Gateway?

ANS: – Use a greedy path variable {topic+} and map it correctly in the integration request UR.

WRITTEN BY Rishi Raj Saikia

Rishi Raj Saikia is working as Sr. Research Associate - Data & AI IoT team at CloudThat. He is a seasoned Electronics & Instrumentation engineer with a history of working in Telecom and the petroleum industry. He also possesses a deep knowledge of electronics, control theory/controller designing, and embedded systems, with PCB designing skills for relevant domains. He is keen on learning new advancements in IoT devices, IIoT technologies, and cloud-based technologies.