When opening text files, most people only look at the extension of the file to determine the file type extension (e.g. txt, pdf, docx). File type extensions in Linux can be changed as easily as changing the file name. When opening a text file, there are scores of byte encodings that must be decoded and displayed to the user. In this project, you will write a program to read input bytes from the stdin and count and record the occurrence of each character in the file. At the end of the file or when the file is closed, you will write to stdout the list of characters sorted by the number of times the characters appear in the input in order of greatest to least. note: some characters may be invisible but they shuold still be printed.
After doing this project you will have a deep understanding of byte and bit manipulation in C, data structures in C, Unicode, and UTF-8. Each of these skills are important for practical computer science.
The code solution to this project must be done individually. There are many Unicode parsers available, do not use the internet to copy solutions to this problem from the network and do not look at or copy solutions from others. However, you may use the Internet for inspiration and you may discuss your general approach with others. To avoid academic dishonesty violations, add all your sources of inspiration to your COLLABORATORS file.
The initial Unix/Linux implementations assume a small set of common characters. In 1963, ASCII was first as a seven-bit code that points to all usable characters. Because seven bits are used, ASCII can only include 2^7 or 128 different characters. This limits the number of characters to a subset of English Roman-based alphabets. Because many programming languages and applications have assumed everyone is using ASCII, serious file recognition issues have emerged.
Around 2008, Unicode and UTF-8 over took Unicode as the most popular web file type encoding. The web is world wide and this trend has only increased. Below we introduce Unicode and UTF-8.
The Unicode standard was developed as a way to encode all of the characters in every language. Unicode is a collection of agreed upon mapping characters (⌛️,⏰️) to a numerical representation called and code points (U+231B, U+23F0). Currently, there is room for 1.1 million characters in the Unicode standard but only ~137K are currently being used. The standard is regularly assigning new characters to codepoints.
UTF-8 is the Unicode Transformation Format, it is a variable-sized encoding for Unicode. It is used to encode text of any language. UTF-8 maps the code point defined in Unicode to a binary representation. Browsers, terminal, and editors, if they know the encoding of the text file (e.g. UTF-8), they can translate the binary representation to the Unicode code point and then the glyph.
In this project, your code will need to read the binary encodings of a UTF-8 file and count/print all of the characters in the document.
In this first example, let’s assume that we are looking at the thorn character in a file (Þ).
We can look up this character on the helpful unicode-table.
We can see that the Unicode code point is
If we add this character to thorn.txt and call hexdump we can see the byte representation of the thorn character.
What we are looking at is the UTF-8 encoded representation.
Some things to remember when looking at hexdumps: (1) each number in the is a nibble and two nibbles make a byte. (2) the representation is little endian, so the first byte is the least significant byte.
cgrant@instance-1 % hexdump -x thorn.txt
Translated to bytes, the UTF-8 encoded bytes is
We can see that it one character stored as two bytes.
Before explaining how the encoding works, lets take a look at one more example.
Lets assume we have a file with only one emoji character (⌛️).
We can look up this character on helpful unicode-table website.
The name of this character is Hourglass, the Unicode number is
If we copy and paste this emoji into a file on our *nix based systems (and remove any extra characters like trailing new lines) we can observe the hexadecimal and binary representation of the file. This emoji is three bytes long.
cgrant@instance-1 % hexdump -x hourglass.txt
0000000 8ce2 009b
Knowing this, and that the hex representation is in little endian form, the UTF-8 binary representation is:
11100010 10001100 10011011
The detection scheme for UTF-8 characters follow just a few rules:
In the first example above, the bit representation of the thorn character is
The two ones in the high order bit tell us that this character is two bytes long.
According to the rule, the second byte also has a 1 in the high order bit.
Here we we put boxes around the UTF-8 control bits:
If we just look at the bytes used to represent the code point we get
00011 011110 ->
000 1101 1110 ->
0 D E.
You will notice that this is the exact code point representation!
In the second example, we will progress through the bytes in the same way.
11100010 10001100 10011011 ->
0010 001100 011011 ->
0010 001100 011011 ->
0010 0011 0001 1011 ->
2 3 1 B
Voilà! Exactly the code point.
Now that you understand byte representations of Unicode code points using UTF-8 you need to use loop through the file and count the occurrences each character. Then, you should sort the collection from largest to smallest and write the characters along with their occurrences to the stdout file. If there is a tie, the character that appears first if input file should appear first in the output.
First make clean is called to clear out an old binaries.
The make all is called to compile the executable.
We assume that a file is in local directory called
We use cat to out put the contents of
| symbol reads
test1 and writes it to
stdin of the
project0 is run it reads from
stdin and writes to
stdout all of the characters in order of the number of times they appear.
The characters and the counts are separated by a dash and the greater than sign (
->), and a new line is on each line.
% make clean
% make all
% cat test1 | ./project0
We encourage you to create a private github repository with the name
liuyunl777 by going to
Settings > Manage Access. (Or
Settings > Collaborators in older version of GitHub).
Your repository should have, at least, the following files:
When you are ready to submit, open the assignment’s submission area on Gradescope. You may make a submission by either uploading directly from your computer or linking to a GitHub or BitBucket repository. Gradescope will allow you to make multiple submissions, and get rapid feedback on those submissions.
Please make sure that you upload all of the required files:
Include a makefile with the standard targets
make all command should create an executable
make clean target should remove executables and temporary files.
The README file should be all uppercase with either no extension or a
.txt or anyother extenstion.
You should (1) write your name in it, (2) a usage example of how to compile and run it, (3) any bugs that should be expected, and (4) a list of any web or external libraries that are used.
The README file should make it clear contain a list of any bugs or assumptions made while writing the program.
Note that you should not be copying code from any website not provided by the instructor.
Using any code from previous semesters or students will results in an academic dishonesty violation.
You should include directions on how to compile and use your code.
You should describe all functions method in the README file.
When describing any known bugs and sources be sure to include any assumptions you make for your solution.
This file should contain a pipe separated list describing who you worked with and a small text description describing the nature of the collaboration. This information should be listed in three fields as in the example is below:
Katherine Johnson | email@example.com | Helped me understand calculations
Dorothy Vaughan | firstname.lastname@example.org | Helped me with multiplexed time management
Stackoverflow | https://example | helped me with a compilation but
Your submission/release is due on Thursday, March 2st at the end of day. The late policy of the course will be followed.
Grades will be assessed according to the following distribution:
echo "a" | xxd
echo "a" | xxd -b
00000000: 01100001 00001010
echo "a" | hexdump -x
echo "😀" | hexdump -x
0000000 9ff0 8098 000a
Wakati wa kufungua faili za maandishi, watu wengi huangalia tu upanuzi wa faili ili kuamua upanuzi wa aina ya faili (kwa mfano, Txt, pdf, docx). Faili aina upanuzi katika Linux inaweza kubadilishwa kwa urahisi kama kubadilisha jina la faili. Wakati wa kufungua faili ya maandishi, kuna alama za encodings byte ambayo lazima decoded na kuonyeshwa kwa mtumiaji. Katika mradi huu, utaandika programu ya kusoma bytes pembejeo kutoka stdin na kuhesabu na kurekodi kutokea kwa kila tabia katika faili. Mwisho wa faili au wakati faili ni kufungwa, utaandika kwa stdout orodha ya wahusika waliopangwa kwa idadi ya mara wahusika kuonekana katika pembejeo kwa utaratibu wa kubwa kwa ndogo. Kumbuka: baadhi ya wahusika inaweza kuwa asiyeonekana lakini wao bado shuold kuchapishwa. Baada ya kufanya mradi huu utakuwa na uelewa wa kina wa byte na bit udhibiti katika C, miundo ya data katika C, Unicode, na UTF-8. Kila moja ya ujuzi huu ni muhimu kwa ajili ya sayansi ya kompyuta vitendo. Suluhisho la kificho kwa mradi huu lazima lifanywe kwa mtu binafsi. Kuna mengi ya Unicode parsers inapatikana, si kutumia mtandao kunakili ufumbuzi wa tatizo hili kutoka mtandao na si kuangalia au kunakili ufumbuzi kutoka kwa wengine. Hata hivyo, unaweza kutumia Intaneti kupata msukumo na unaweza kuzungumzia mbinu yako ya jumla na wengine. Ili kuepuka ukiukaji wa ukosefu wa uaminifu wa kitaaluma, ongeza vyanzo vyako vyote vya msukumo kwenye faili yako ya WANAFUANISHI. Utekelezaji wa awali wa Unix / Linux unadhani seti ndogo ya wahusika wa kawaida. ASCII ni kificho cha kwanza cha 7 bit ambacho kinaonyesha herufi zote zinazotumika. Kwa sababu ya kutumia bits saba, ASCII inaweza tu kujumuisha 2 × 7 au herufi 128 tofauti. Hii hupunguza idadi ya wahusika kwa subset ya Kiingereza Kirumi-msingi alfabeti. Kwa sababu lugha nyingi za programu na maombi wamefikiria kila mtu anatumia ASCII, masuala makubwa ya utambuzi wa faili yameibuka. Mwaka 2008, Unicode na UTF-8 ilichukua Unicode kama encoding maarufu zaidi ya aina ya faili ya wavuti. Mtandao ni wa ulimwengu wote na mwenendo huu umeongezeka tu. Hapa chini tunaelezea jinsi ya kuunda Unicode na UTF-8. Kiwango cha Unicode kilitengenezwa kama njia ya kuingiza herufi zote katika kila lugha. Unicode ni mkusanyiko wa alama zilizokubaliwa za ramani (<unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk>, <unk> <unk>, <unk> <unk Kwa sasa, kuna nafasi ya herufi milioni 1.1 katika kiwango cha Unicode lakini ~ 137K tu zinatumiwa. kiwango ni mara kwa mara kugawa wahusika wapya kwa codepoints. UTF-8 ni Unicode Transformation Format, ambayo ni encoding variable-size kwa Unicode. Inatumiwa kuingiza maandishi ya lugha yoyote. UTF-8 ramani code uhakika maalum katika Unicode kwa uwakilishi binary. Browsers, terminal, na wahariri, kama wanajua encoding ya faili ya maandishi (kwa mfano UTF-8: UTF-8 ni aina ya maandishi ya binary ambayo hufafanua alama ya Unicode na alama ya alama ya alama ya alama. Katika mradi huu, code yako itahitaji kusoma encodings binary ya faili UTF-8 na kuhesabu / kuchapisha wahusika wote katika hati. Katika mfano huu wa kwanza, hebu kudhani kwamba sisi ni kuangalia katika mti tabia katika faili (Þ). Tunaweza kuangalia juu ya tabia hii juu ya msaada unicode-meza. "Kama tunaona katika Unicode, ""thorn"" ni herufi ya kwanza katika thorn.txt, na ""hexdump"" ni herufi ya pili katika thorn." Kwa mfano, kwa mfano, UTF-8 encoded representation. Kwa mfano, hexadecimal ni aina ya data ambayo inahitajika kwa ajili ya kutengeneza hexadecimal: kila nambari katika hexadecimal ni nimble, na nimble mbili hufanya byte. Kwa mfano, kama ni kweli kwamba ni ndogo, basi byte ya kwanza ni ndogo. "Ctrl + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Kabla ya kueleza jinsi encoding kazi, hebu kuchukua kuangalia mfano mmoja zaidi. Hebu kudhani tuna faili na moja tu emoji tabia (️). Tunaweza kuangalia juu ya tabia hii juu ya kusaidia unicode-meza tovuti. "Jina la tabia hii ni ""Hourglass"" na nambari ya Unicode ni ""If we copy and paste this emoji into a file on our Unix-based systems (and remove any extra characters like trailing new lines) we can observe the hexadecimal and binary representation of the file.""" Emoji hii ina urefu wa byte tatu. "Ufunuo wa ""hexadecimal"" ni sawa na ""hexadecimal"" na ""hexadecimal"" ni sawa na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal"" ni sawa na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal"" ni sawa na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal"" ni sawa na ""hexadecimal"" na ""hexadecimal"" na ""hexadecimal""." Kwa mujibu wa sheria, byte ya pili pia ina 1 katika bit ya utaratibu wa juu. Kwa mfano, kama wewe ni kuangalia kwa byte ya code point, unaweza kuona kwamba hii ni moja kwa moja code point representation. Katika mfano wa pili, tutaendelea kupitia bytes kwa njia ile ile. "111011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 Hasa sehemu ya kificho. Sasa kwamba unaelewa byte uwakilishi wa Unicode code pointi kutumia UTF-8 unahitaji kutumia kitanzi kupitia faili na kuhesabu matukio ya kila tabia. Kisha, unapaswa kupanga mkusanyiko kutoka kubwa kwa ndogo na kuandika wahusika pamoja na matukio yao kwa faili stdout. Kama kuna tie, tabia ambayo inaonekana kwanza kama faili ya pembejeo inapaswa kuonekana kwanza katika pato. Kwanza kufanya safi inaitwa kuondoa binaries zamani. kufanya wote ni kuitwa kukusanya executable. "Kama faili ya ""cat"" imeandikwa katika orodha ya ndani, ""cat"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test"" na ""test"" ni alama ya ""test""." Wahusika na hesabu ni kutenganishwa na dash na kubwa kuliko ishara ( ->), na mstari mpya ni juu ya kila mstari. "Kama ""Kufanya yote"" ni sawa na ""Kufanya yote"" - ""Kufanya yote"" ni sawa na ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote"" - ""Kufanya yote.""" (Au Mipangilio > Washiriki katika toleo la zamani la GitHub). Hifadhi yako inapaswa kuwa, angalau, faili zifuatazo: Wakati wewe ni tayari kuwasilisha, kufungua kazi ya kuwasilisha eneo juu ya Gradescope. Unaweza kuwasilisha kwa ama kupakia moja kwa moja kutoka kwa kompyuta yako au kuunganisha kwenye hifadhi ya GitHub au BitBucket. Gradescope itakuruhusu kufanya mawasilisho mengi, na kupata maoni ya haraka juu ya mawasilisho hayo. Tafadhali hakikisha kwamba wewe kupakia wote wa faili required: Ni pamoja na makefile na malengo ya kawaida kufanya amri zote lazima kuunda executable kufanya safi lengo lazima kuondoa executables na faili ya muda. README faili inapaswa kuwa wote uppercase na ama hakuna upanuzi au.txt au upanuzi mwingine wowote. Unapaswa kuandika jina lako, mfano wa jinsi ya kuunganisha na kuendesha, makosa yoyote ambayo yanatarajiwa, na orodha ya maktaba yoyote ya nje ya mtandao. README faili lazima wazi kuwa na orodha ya makosa yoyote au dhana alifanya wakati wa kuandika programu. Kumbuka kwamba unapaswa kuwa nakala code kutoka tovuti yoyote si zinazotolewa na mwalimu. Kutumia nambari yoyote kutoka semesters zilizopita au wanafunzi itasababisha ukiukaji wa udanganyifu wa kitaaluma. Unapaswa kujumuisha maelekezo juu ya jinsi ya kukusanya na kutumia kificho chako. Unapaswa kuelezea njia zote kazi katika faili ya README. Wakati kuelezea yoyote inayojulikana bugs na vyanzo kuwa na uhakika wa ni pamoja na dhana yoyote ya kufanya kwa ajili ya ufumbuzi wako. Faili hii inapaswa kuwa na orodha iliyogawanywa ya bomba inayoelezea ni nani uliyofanya kazi naye na maelezo madogo ya maandishi yanayoelezea asili ya ushirikiano. "Kama ilivyoelezwa katika makala ya kwanza ya makala hii, ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine Johnson: ""Katherine: ""Katherine Johnson: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine: ""Katherine Sera ya kuchelewa ya kozi itafuatwa. "Kama ilivyoelezwa hapo juu, ""kijana"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" ni ""mwanamke"" na ""mwanamke"" na ""mwanamke"" ni ""wanamke"" na ""wanamke"""