filepath-1.4.100.0: Library for manipulating FilePaths in a cross platform way.

System.OsString.Internal

Synopsis

Documentation

encodeUtf :: MonadThrow m => String -> m OsString Source #

Partial unicode friendly encoding.

On windows this encodes as UTF16-LE (strictly), which is a pretty good guess. On unix this encodes as UTF8 (strictly), which is a good guess.

Throws a EncodingException if encoding fails.

Arguments

 :: TextEncoding unix text encoding -> TextEncoding windows text encoding -> String -> Either EncodingException OsString

Encode an OsString given the platform specific encodings.

Like encodeUtf, except this mimics the behavior of the base library when doing filesystem operations, which is:

1. on unix, uses shady PEP 383 style encoding (based on the current locale, but PEP 383 only works properly on UTF-8 encodings, so good luck)
2. on windows does permissive UTF-16 encoding, where coding errors generate Chars in the surrogate range

Looking up the locale requires IO. If you're not worried about calls to setFileSystemEncoding, then unsafePerformIO may be feasible (make sure to deeply evaluate the result to catch exceptions).

decodeUtf :: MonadThrow m => OsString -> m String Source #

Partial unicode friendly decoding.

On windows this decodes as UTF16-LE (strictly), which is a pretty good guess. On unix this decodes as UTF8 (strictly), which is a good guess. Note that filenames on unix are encoding agnostic char arrays.

Throws a EncodingException if decoding fails.

Arguments

 :: TextEncoding unix text encoding -> TextEncoding windows text encoding -> OsString -> Either EncodingException String

Decode an OsString with the specified encoding.

The String is forced into memory to catch all exceptions.

Like decodeUtf, except this mimics the behavior of the base library when doing filesystem operations, which is:

1. on unix, uses shady PEP 383 style encoding (based on the current locale, but PEP 383 only works properly on UTF-8 encodings, so good luck)
2. on windows does permissive UTF-16 encoding, where coding errors generate Chars in the surrogate range

Looking up the locale requires IO. If you're not worried about calls to setFileSystemEncoding, then unsafePerformIO may be feasible (make sure to deeply evaluate the result to catch exceptions).

Constructs an OsString from a ByteString.

On windows, this ensures valid UCS-2LE, on unix it is passed unchanged/unchecked.

Throws EncodingException on invalid UCS-2LE on windows (although unlikely).

QuasiQuote an OsString. This accepts Unicode characters and encodes as UTF-8 on unix and UTF-16 on windows.

unpack :: OsString -> [OsChar] Source #

Unpack an OsString to a list of OsChar.

pack :: [OsChar] -> OsString Source #

Pack a list of OsChar to an OsString

Note that using this in conjunction with unsafeFromChar to convert from [Char] to OsString is probably not what you want, because it will truncate unicode code points.

Truncates on unix to 1 and on Windows to 2 octets.

Converts back to a unicode codepoint (total).
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.4304545223712921, "perplexity": 20945.990113316828}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2023-06/segments/1674764500094.26/warc/CC-MAIN-20230204044030-20230204074030-00505.warc.gz"}