v3.1: application/x-www-form-urlencoded potential inconsistency between examples and referenced RFC

[mnahkies]

tldr;
I think that for application/x-www-form-urlencoded content types, all special characters including : should always be percent-encoded, excepting spaces should always be the special case of encoded as a +

Though I do appreciate it is probably easy to find examples of languages/libraries where this isn't followed 😅 and so it maybe difficult stipulate as a MUST...

Describe the error in the specification
With respect to the examples given here:

• 4.8.15.2.1 Example: URL Encoded Form with JSON Values
• 4.8.12.4 Style Examples

RFC 1866 is referenced, and from my skimming of the RFC the only section that seems relevant in this context to me is 8.2 Form Submission which defines serialization of `application/x-www-form-urlencoded', specifically:

The form field names and values are escaped: space characters are replaced by +, and then reserved characters are escaped as per [URL]; that is, non-alphanumeric characters are replaced by %HH, a percent sign and two hexadecimal digits representing the ASCII code of the character. Line breaks, as in multi-line text field values, are represented as CR LF pairs, i.e. %0D%0A.

The example searilization given in the OAI specification under 4.8.15.2.1 is then:

id=f81d4fae-7dec-11d0-a765-00a0c91e6bf6&address=%7B%22streetAddress%22:%22123+Example+Dr.%22,%22city%22:%22Somewhere%22,%22state%22:%22CA%22,%22zip%22:%2299999%2B1234%22%7D

Given that in this example, we have a field name address with a value that happens to be (ie: form data isn't aware of this, its just an opaque string AFAIK) a JSON serialized string, I'd expect all special characters to be percent-encoded, giving a result of:

id=f81d4fae-7dec-11d0-a765-00a0c91e6bf6&address=%7B%22streetAddress%22%3A%22123+Example+Dr.%22%2C%22city%22%3A%22Somewhere%22%2C%22state%22%3A%22CA%22%2C%22zip%22%3A%2299999%2B1234%22%7D

This matches the behavior of URLSearchParams on web/nodejs, eg:

const params = new URLSearchParams();
params.append("id", "f81d4fae-7dec-11d0-a765-00a0c91e6bf6");
params.append("address", JSON.stringify({streetAddress: "123 Example Dr.", city: "Somewhere", state: "CA", zip: "99999+1234"}));
params.toString();

Yields the above, and it will also round-trip back, eg: using JSON.parse(new URLSearchParams(params.toString()).get("address"))

The behavior here is very likely to differ across platforms / implementations, but as a quick sense-check I observe the same behavior in java/kotlin using both java.net.URLEncoder (playground example) and org.apache.http.client.utils.URLEncodedUtils which increases my confidence in my interpretation of the RFC.

org.apache.httpcomponents:httpclient:4.5.13

// depends on `implementation("org.apache.httpcomponents:httpclient:4.5.13")`
import org.apache.http.client.utils.URLEncodedUtils
import org.apache.http.message.BasicNameValuePair
import java.nio.charset.StandardCharsets

fun main() {
    val params = listOf(
        BasicNameValuePair("id", "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"),
        BasicNameValuePair("address", """{"streetAddress":"123 Example Dr.","city":"Somewhere","state":"CA","zip":"99999+1234"}""")
    )
    val encoded = URLEncodedUtils.format(params, StandardCharsets.UTF_8)
    println(encoded)
}

There are several other examples under section 4.8.12.4 that exhibit similar discrepancy:

• style: form, explode: false: , not escaped for array and object
• style: spaceDelimited, explode: false: replaced with %20 instead of with +

Later in appendix E.5, reference is made to RFC 3986

I haven't fully read and understood that RFC, but my current interpretation of statements such as:

URI producing applications should percent-encode data octets that correspond to characters in the reserved set unless these characters are specifically allowed by the URI scheme to represent data in that component.

Mean that gen-delims including : etc indicates that a : falling within the query, Should still be percent encoded.

query       = *( pchar / "/" / "?" )

(pchar includes :)

as they are not "representing data" in the context of the URI scheme / semantic meaning of the query portion - though I can't find where "representing data" is actually explicitly defined, so my reading of this could be wrong (language used is a bit ambiguous IMO)

Additional context
Slack conversation: https://open-api.slack.com/archives/C1137F8HF/p1753276004758329

[mnahkies]

Additional C# example using WebUtility.UrlEncode https://dotnetfiddle.net/r8B5Cn has the same behavior as the examples I gave above.

C# / WebUtility.UrlEncode

using System;
using System.Collections.Generic;
using System.Net;
using System.Text;

public class Program
{
	public static string EncodeFormData(Dictionary<string, string> parameters)
	{
		var builder = new StringBuilder();
		foreach (var kv in parameters)
		{
			if (builder.Length > 0)
				builder.Append('&');
			builder.Append(WebUtility.UrlEncode(kv.Key));
			builder.Append('=');
			builder.Append(WebUtility.UrlEncode(kv.Value));
		}

		return builder.ToString();
	}

	public static void Main()
	{
		var formParams = new Dictionary<string, string>
		{
			{
				"id",
				"f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
			},
			{
				"address",
				"{\"streetAddress\":\"123 Example Dr.\",\"city\":\"Somewhere\",\"state\":\"CA\",\"zip\":\"99999+1234\"}"
			}
		};
		string encoded = EncodeFormData(formParams);
		Console.WriteLine(encoded);
	}
}

[handrews]

OK I think I have mostly worked this out. There is a larger problem which I will elaborate, and then there is the question of the exact behavior of something that attempts to comply with form-urlencoded which will have to wait until I'm feeling better.

The big problem is that when using the style/explode/allowReserved in either the Encoding Object or Parameter Object, RFC6570 (which is based on RFC3986) MUST be followed. While certain configurations of those fields produce results that could, with very carefully selective additional pre-(allowReserved: true) or post-(allowReserved: false)template expansion percent-encoding, be made compliant with whichever form-urlencoded spec one chooses to follow, this is not possible in the general case. For example, RFC6570 reserves , as a sub-delimiter, which is 100% complilant with RFC3986 but not with form-urlencoded. RFC6570's Security Considerations section even notes that URI Templates are like forms, but are not identical to them.

For contentEncoding in the Encoding Object, it makes sense to require form-urlencoded behavior as that requirement is clearly expressed for the Encoding Object despite promptly being violated by the above-noted fields.

For content in the Parameter Object, it is not entirely clear which specification governs; I would guess RFC3986 as that is our general-purpose URI specification.

The upshot here is that it is possible to create a form-urlencoded serialization that is compliant with however one reads the percent-encoding requirements for form-urlencoded, but only if you avoid certain OAS features and do some really fiddly additional percent-encoding that will not be supported by any standard or 3rd-party libraries.

However, this is not a reasonable restriction. So we need to figure out what is really required here. And then once we know where form-urlencoded behavior is really required, we can argue what that behavior is, as the three or four specs involved (depending on how you count) all contradict each other at least somewhat.

[handrews]

I think ultimately this all has to do with form-urlencoded as being both one of the most aggressive applications of Postel's law (RFC760 §3.2), and one of the most prominent examples of the harm that the common interpretation of that law can cause (see RFC9413).

If you want to produce technically correct (according to some combination of RFC1738/RFC1866/HTML4.01/WHATWG URL that I will sort out later)form-urlencoded, then:

• In a Media Type Object with Encoding Objects, use contentType (the default approach) and do your own percent-encoding accordingly
• In the Parameter Object, use in: querystring with content and Encoding Objects (which reduces this case to the previous one), or use in: query with content and do your own percent-encoding accordingly

However, note that all form-urlencoded specifications can successfully decode any valid URI query string (including %20 for space) as long as &, =, and + are only used for their delimiting/escape behavior, and percent-encoded otherwise.

• This means that you can have a serialization, including those produced in accordance with RFC6570/RFC3986, that is not quite in compliance with the requirements for producing form-urlencoded, but is fully compliant in terms of consuming form-urlencoded.

Therefore it's going to be up to the user to decide how compliant they wish to be. We should, of course, pick something for examples, and I'll come back to unraveling that mess later. But we can't force exactly-correct-production of form-urlencoded in all cases without removing the functionality provided by RFC6570 that we also support.

[handrews]

@mnahkies OK, time to attempt to explain the percent-encoding part.

The root of the problem is that RFC1866 (and W3C HTML 4.01) say to "reserved characters are escaped as per RFC]; that is, non-alphanumeric characters are replaced by %HH, a percent sign and two hexadecimal digits representing the ASCII code of the character." (HTML 4.01 clarifies that RFC1738 §2.2 is meant).

However this is not what RFC1738 says! While it explicitly allows percent-encoding anything, including alphanumerics, that is not being used as a delimiter, it clearly just a "btw over-encoding is safe" as the inclusion of alphanumerics implies. RFC1738 defines -._*+!'(),$ as the set of safe (not reserved) non-alphanumeric characters that can be used without percent-encoding (note that + is included!)

The paraphrase in RFC1866 and HTML 4.01 is also, AFAICT, not implemented by anyone anywhere, not even the examples you linked to. Both of your examples leave . un-percent-encoded even though it is a non-alphanumeric character. It is, however, in the "safe" (RFC1738) or "unreserved" (RFC3986) or "not in any percent-encode" (WHATWG URL) sets.

RFC1738's safe set is clearly problematic as it includes + as well as other characters now considered "sub-delims" by RFC3986. But RFC1738 and RFC1866 have both been superseded multiple times. The current version of RFC1738 is RFC3986, and the current version of RFC1866 is WHATWG, specifically their URL spec (which is purely browser-oriented and has refused to address non-browser URI use cases).

• RFC3986 globally safe: -_.~
• WHATWG URL form-urlencoded safe: -_.*

🤦

So there is literally no way to make every specification happy here.

Since AFAICT no one implements percent-encoding all non-alphanumerics, we're not going to require that either, no matter what RFC1866 says. RFC1738's safe set is problematic, so we can't recommend following that strictly. And the most restrictive approaches in RFC3986 and WHATWG conflict with each other, although not with the more relaxed rules specific to query components (although those also differ).

Next big comment (probably some smaller ones first): Why it is not sufficient to just pick a percent-encoding set, and why we also need to consider the algorithm of how and when percent-encoding is applied.

[handrews]

@mnahkies

as they are not "representing data" in the context of the URI scheme / semantic meaning of the query portion - though I can't find where "representing data" is actually explicitly defined,

It's in the paragraph after that ABNF line:

 query       = *( pchar / "/" / "?" )

The characters slash ("/") and question mark ("?") may represent data
within the query component.

This means that anything in pchar, plus / and ? which were commented on explicitly because they are special cases for the query, are allowed to represent data un-percent-encoded, unless (as discussed in the last paragraph under "Path") they are also reserved characters and something else defines a non-data use for them. For example, form-urlencoded defining uses for &=@, and RFC6570 defining a delimiter use for ,. Which brings us to:

There are several other examples under section 4.8.12.4 that exhibit similar discrepancy:

• style: form, explode: false: , not escaped for array and object

Because these are RFC6570 rules, and RFC6570 is reserving the , as a delimiter. Likewise:

• style: spaceDelimited, explode: false: replaced with %20 instead of with +

This is also the correct RFC6570 rule, although this one gets exceptionally difficult as we'll see in the future comment I promised about why not just the character sets but the algorithm is significant.