Percent encoding `|` in paths

[nathaniel-daniel]

• Initially raised as discussion Percent encoding `|` in paths #3479

I got no response, so I'm opening this issue for more visibility.

OS: Windows 11
python --version: Python 3.12.8
httpx version: 0.28.1

I believe the | should be percent encoded in paths, which is not currently the case. If I'm understanding RFC3986 correctly, path characters are pchar, which can be unreserved, pct-encoded, sub-delims, ":", or "@". unreserved can be composed of ALPHA, DIGIT, "-", ".", "_", or "~". pct-encoded is the percent encoding sequences. sub-delims can be "!", "$", "&", "'", "(", ")", "*", "+", ",", ";", or "=". Nowhere in this set is the | character present, meaning it has to be percent-encoded.

Simplifying my problem, httpx seems to call its internal urlparse function to process urls. So, here's an example using that function. This function normally percent-encodes characters as needed, like spaces:

httpx._urlparse.urlparse('http://example.com/ ')

will return

ParseResult(scheme='http', userinfo='', host='example.com', port=None, path='/%20', query=None, fragment=None)

However, this does not happen for |:

httpx._urlparse.urlparse('http://example.com/|')

will return

ParseResult(scheme='http', userinfo='', host='example.com', port=None, path='/|', query=None, fragment=None)

In Firefox and Google Chrome, | is percent-encoded:

encodeURI('http://example.com/|') 

will return

"http://example.com/%7C"

In the requests library, | is also percent-encoded:

requests.utils.requote_uri('http://example.com/|')

will return

'http://example.com/%7C'

The rfc3986 library also percent encodes |:

rfc3986.urlparse('http://example.com/|')

will return

ParseResult(scheme='http', userinfo=None, host='example.com', port=None, path='/%7C', query=None, fragment=None)

Using urllib itself, | also seems to be percent-encoded for path components:

urllib.parse.quote('/|')

will return

'/%7C'

I'm fairly certain that I've interpreted this RFC right, and I think that | should be excluded from the PATH_SAFE set here. Here is its current value: "!$%&'()*+,-./0123456789:;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_abcdefghijklmnopqrstuvwxyz|~".

Potential Fix: nathaniel-daniel@a2f327f

[lovelydinosaur]

The source code indicates our point of reference here. (whatwg)

A good prompt for a convo on how...

• Differing specs/implementations exist wrt. URL encoding sets.
• Some amount of variance/discordance here probably isn't a terrible thing.
• There's edge elements here that shouldn't matter / it's not worth anyone's time / non-broken servers don't care.

We could just work to the same set as Pythons stdlib, rather than whatwg. Being more closely in line with dominant browsers is a path of less resistance.

[nathaniel-daniel]

The source code indicates our point of reference here. (whatwg)

I see, I was thrown off by the documentation in that module. Perhaps it should be updated.

Also, I think the whatwg standard also says that a | cannot be a part of a url path segment. I believe path segments must be comprised of url units, which are composed of percent encoded bytes or url code points, which seem to not include that character.

However, this seems self-conflicting as the path percent encode set also doesn't seem to include that character. I think I'll give the standard another look-through before I open an issue on the whatwg url tracker.

• Some amount of variance/discordance here probably isn't a terrible thing.
• There's edge elements here that shouldn't matter / it's not worth anyone's time / non-broken servers don't care.

I opened this issue because I was changing the http library in some code from requests to httpx. I sent a | as part of a path segment, and the server issued a 301 to the same url with the | percent encoded, which ended up throwing an error since this library doesn't follow redirects by default. Looking into the url handling in this library, I assumed it followed RFC3986 and got the impression that | should be percent encoded from reading it. I'm not sure if the server's behavior is wrong here.

We could just work to the same set as Pythons stdlib, rather than whatwg. Being more closely in line with dominant browsers is a path of less resistance.

Looking into it more, I think Python's stdlib follows RFC 1808, which RFC3986 claims to obsolete. In turn, the whatwg url standard also seems to claim to obsolete RFC3986. I think following the whatwg url standard is the right decision, since it will align this library closer with browsers.

Edit:
Alright, just found whatwg/url#852. Looks like whatwg url encoders are allowed to create invalid whatwg urls, and whatwg url parsers are expected to throw a non-fatal error in such scenarios. I'm not sure what the best course of action is here.

As for browsers, encodeURI follows RFC3986, but URL follows whatwg. However, when sending urls to the server, Chrome will percent-encode |, but Firefox will send the character without encoding it.

Feel free to close I guess since this seems like it's technically the correct behavior.

[nathaniel-daniel]

Actually, I read whatwg/url#852 (comment) as meaning including bare | chars may not be recommended due to potential compatibility issues. Would it be possible to add some kind of option or control to change how this character gets encoded?

[lovelydinosaur]

We wouldn't add user-facing controls here, tho could review what escape sets we use.

[Mr-Neutr0n]

Opened a PR for this: #3764