**Bug introduced in 11.0 or earlier and persists through 12.0**

(*confirmed by WRI Support but read summary below*)

---

Answers summary
===============

I think we need a quick summary here to justify keeping [bug](/questions/tagged/bug "show questions tagged 'bug'"). 

Original problem **is not a bug**. Here is why:

```
URLRead[#, "Body"]&:        <raw http response> -> json_String
   (*decoding, driven by content-type/charset header*)

ImportString[#,"RawJSON"]&: json_String -> wlJSON_(Association|List)
   (* decoding driven by assumption that JSONString was UTF8 encoded*)

```

So the problem occurs because of double decoding. Workaround? Use `BodyBytes` (see below) or `URLExecute`. It should be easier to understand this and documentation does not help so much. Even WRI Support was confused.

What it also implies is that if you are preparing json with `ExportString` for `HTTPRequest / HTTPResponse` then you need to include `CharacterEncoding->None` for them to avoid double encoded message.

**There is an issue** though, which I'm tempted to call a documentation bug. Handling *content-type*, *charset* and *content-encoding* by family of `URL*` functions is not documented and changes with versions (improves) but it is not clear what to expect from them. It is not documented what a *http response body* is and what to expect from it with respect to mentioned headers.

**There is a bug**, JSON by its standard is UTF8 encoded and *charset* should not play any role, it can be there or not. *content-type:application/json* header should be enough to expect http response body to be decoded. It is not when there is no *charset* spec. Then the problem does not occur and the confusion is doubled (was for me when I tried to prepare a minimal example with different source of http responses)

The bug can be confirmed with:

```
urls = CloudDeploy @ Delayed[
  HTTPResponse[
    ExportString[{"\[Dash]"}, "RawJSON", "Compact" -> True]
  , <|"ContentType" -> "application/json" <> #|>
  , CharacterEncoding -> None
  ]
] & /@ {"", "; charset=utf-8"};

bodies = URLRead[#, "Body"] & /@ urls
ImportString[#, "RawJSON"] & /@ bodies

```

> 
> 
> ```
> {{"\[Dash]"}, $Failed}
> 
> ```
> 
> 

This is **NOT** about `$Failed` it is about two different results caused by different `charset` spec. Which should not matter for `JSON`.

---

Original question
=================

Background
----------

V11.1.1 I'm using `URLRead` to fetch some data, 

can't show everything but headers contain:

```
"content-type->application/json; charset=utf-8"

```

and body (returned from URLRead) contains something you can create by:

```
body = "[\"" <> FromCharacterCode[8211] <> "\"]";

```

[![enter image description here](https://i.stack.imgur.com/ZBvbA.png)](https://i.stack.imgur.com/ZBvbA.png)

---

Problem
-------

Since header is correct I'd expect the `body` to be ready for `ImportString`, but it is not:

```
ImportString[body, "JSON"]

(*$Failed*)

ImportString[body, "RawJSON"]

(*$Failed and 
  General::jsonoutofrangeunicode :  Out of range unicode code point encountered.
*)

```

What works though is:

```
ImportString[ ToString[body, OutputForm, CharacterEncoding -> "UTF8"], "RawJSON"]

```

> 
> 
> ```
> {"\[Dash]"}
> 
> ```
> 
> 

Who is to blame, me, mathematica or the server for malformed response? Or maybe no one but then my method seems ugly for something that should be a standard procedure.

I tried to dig in encoding, responses, headers etc but I got lost in what should happen when.

Would appreciate clarification.

---

Update
------

I tried to mimic a round trip:

```
jsonBytes = ByteArray[Join[
    ToCharacterCode["[\"", "UTF-8"],
    ToCharacterCode[FromCharacterCode[8211], "UTF-8"],
    ToCharacterCode["\"]", "UTF-8"]
    ]];

co = CloudDeploy @ Delayed[
    HTTPResponse[
     jsonBytes,
     <|"ContentType" -> "application/json", 
      "CharacterEncoding" -> "UTF8"|>
     ]
    ];

URLRead[co, {"Headers", "Body"}]

```

> 
> 
> ```
> <|"Headers" -> { ...
>       ,  "content-type" -> "application/json"
>       , "vary" -> "Accept-Encoding"
>       ,  "transfer-encoding" -> "chunked"}
>     , "Body" -> "[\"â\"]"
>  |>
> 
> ```
> 
> 

Notice the body!, it looks 'ok' now:

```
URLRead[co, {"Headers", "Body"}]["Body"] //  ImportString[#, "RawJSON"] &

```

> 
> 
> ```
> {"\[Dash]"}
> 
> ```
> 
> 

But the encoding information is missing in headers. I just said `"CharacterEncoding" -> "UTF8"`, didn't I?

If I force the encoding in content type field:

```
... <|"ContentType" -> "application/json; charset=utf-8"|> ...

```

then it is preserved 

```
URLRead[co, {"Headers", "Body"}]

```

but body is incorrect:

```
<|"Headers" -> {..., 
  "content-type" -> "application/json;charset=utf-8", 
  "vary" -> "Accept-Encoding", 
  "transfer-encoding" -> "chunked"}
,"Body" -> "[\"\[Dash]\"]"
|>

```

And import string fails.

Summing up:

* ignored character encoding, should not happen in my opinion
* misinterpreted encoding when encoding is provided correctly
* my interpretation is that `URLRead` and friends handle communcation from MMA to WPC if you don't care so much but it looks like some things are assumed instead of read from e.g. headers so communication with external services is flawed.

What is the story? I don't have time for that...

---

Related:

* [Importing malformed XML: Import[... "XMLObject"] vs ImportString[..., "XML"]](https://mathematica.stackexchange.com/q/50761/5478)
* <https://mathematica.stackexchange.com/a/145771/5478>
* [URL functions overload](https://mathematica.stackexchange.com/q/127874/5478)