Date: Tue, 27 Oct 2020 12:59:29 +0000 (UTC) Message-ID: <1350277030.16244.1603803569242@i-0014.tor1.incd.ca> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_16243_972461663.1603803569240" ------=_Part_16243_972461663.1603803569240 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html McCord Document Conversion

# McCord Document Conversion

This page summarises the results of a number of techniques for c= onverting the McCord XML data into a JSON form suitable to be stored in Cou= chDb. A sample of the current XML schema can be seen at https://source.fluidproject.org/svn/fl= uid/engage/trunk/src/testdata/artifacts. Note that this schema is not s= table, and is expected to change repeatedly over the course of the project.= Indeed, each museum will likely have their own variations on a schema, and= so our approach fundamentally embraces this variability.

=20

We have evaluated multiple approaches, in several languages. In particul= ar, "pure" benchmarks written in Python and Java, using the respective lang= uages' XML parsers, and Javascript benchmarks written in three styles - =20

=20
1. using the "fastXmlPull" parser which Fluid uses to parse HTML for its <= a href=3D"/display/Infusion13/Renderer">Renderer
2. =20
3. using John Resig's HTML parser (now delivered in env.js). Resig's parse= r was first announced on his blog at http://ej= ohn.org/blog/pure-javascript-html-parser/ and the freshest instance of = the source seems to be held in his github for env.js at http://github.com/jeresig/env-js/tree/master
4. =20
5. using the browser's native DOM, available as the return from an XmlHttp= Request
6. =20
7. Thomas Frank's "xml2json" parser, held at http://w= ww.thomasfrank.se/xml_to_json.html. Frank references a better implement= ation held at www.terracoder.com, however this site as of the time of writi= ng has been down for several weeks.
8. =20
=20

The Python code is currently attached to JIRA FLUID-2816, whereas the Java and Javascript tests are held in incubato= r SVN at https://source.fluid= project.org/svn/incubator/engage-sketches/trunk.

=20

## Javasc= ript measurements in-browser

=20

Firstly the in-browser measurements, performed in various browsers on di= fferent machines:

=20
=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20

Machine/Browser

fastXmlPull

Resig

DOM

Frank

Antranig/FF3

40 documents in 1057ms: 26.425ms per call

=

4 documents in 1604ms: 401ms per call

=20

40 documents in 1132ms: 28.3ms per call

<= /td>=20

4 documents in 3247ms: 811.75ms per call

<= /td>=20

Justin/FF3

16.8ms per call

467.75ms per call

17.175ms per call

Yura/FF3

40 documents in 770ms: 19.25ms per call

<= /td>=20

4 documents in 2399ms: 599.75ms per call

=

40 documents in 669ms: 16.725ms per call

<= /td>=20

Antranig/FF2

40 documents in 2187ms: 54.675ms per call

=

4 documents in 2422ms: 605.5ms per call

<= /td>=20

40 documents in 3406ms: 85.15ms per call

<= /td>=20

parsed 4 documents in 2328ms: 582ms per call<= /p>

Antranig/FF3.5b4

40 documents in 951ms: 23.775ms per call

<= /td>=20

4 documents in 1666ms: 416.5ms per call

=20

40 documents in 756ms: 18.9ms per call

=20

4 documents in 2071ms: 517.75ms per call

<= /td>=20

Justin/Chrome

8.2ms per call

41.75 ms per call

10.5 ms per call

=20

The first three lines establish basic normalisation between the three ma= chines. Basically Antranig:Yura:Justin is around in a ratio 1.5:1.2:1

= =20

## Observa= tions on Javascript results

=20

The Frank and Resig parsers are extremely slow. Whilst they make good us= e of RegExps to potentially accelerate performance in browsers, their overa= ll approach to parsing is very suboptimal, frequently duplicating the docum= ent text in memory several times over during the course of parsing. The Fra= nk parser in particular makes very frequent use of "eval" which will result= in poor performance on any runtime. These two parsers take roughly an orde= r of magnitude longer than the other approaches.

=20

NB - the Resig parser is somewhat specialised for HTML, at the expense o= f behaviour on XML documents. The base code has been tweaked in a few insta= nces to allow parsing to continue on general XML but the produced JSON is n= ot correct, as a result of an assumption in the code that any node type whi= ch is capable of containing CDATA is not also capable of containing child n= odes. However, the overall run time is probably still a good estimate of th= e speed of this parser.

=20

The "fastXmlPull" parser (the Fluid homegrown approach) is in general th= e fastest. Despite the name it is actually capable of parsing realistic bro= wser HTML as well as XML. Whilst on some platforms, the DOM approach appear= s marginally faster (on the very latest Firefox beta at 19ms against 25ms) = these tests give the DOM method an unfair advantage since they only measure= the cost of iteration over the DOM. The process of building the D= OM structure itself occurs inside the browser's XHR engine and is not possi= ble to profile separately from the overall fetch process for the files. Onc= e this is taken into account, fastXmlPull would probably have equal or grea= ter performance on all platforms.

=20

In terms of general strategy, fastXmlPull is optimised for "slow" browse= r environments. In "last-generation" Javascript engines, there is a conside= rable performance premium on applying native operations such as RegExps and= indexOf (see blog commentary at How Lo= ng, Oh Lord). However, the more modern VMs such as those in Firefox 3.5= and Chrome in Google will increasingly make this approach suboptimal as th= e speed of Javascript language primitives increases. A taste of this can be= seen in the form of the narrowing headroom between FastXmlPull (strongly r= egexp based) and the other approaches on the more modern VMs. For truly exc= ellent performance in these modern JS environments, we will require a groun= d-up rewrite of the parser, more along the lines of the "FSM character cree= ping" model seen in highly successful XML parsers in environments like Java= , such as Aleksander Slominski's XPP3 parser that we benchmark later on.=20

## Measurement= s in Python and Java

=20

Now, for measurements in other languages. Yura on his machine has run be= nchmarks on CPython and JPython, and on mine I ran an equivalent conversion= in Java:

=20
=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20

Machine/Language

Reps

Total time

Per doc

Yura/Jython

20000

jython:real 2m27.557s, user 2m26.105s, sys 0m= 1.316s

7ms/doc

Yura/CPython 2.6

20000

real 1m6.248s, user 1m4.452s, sys 0m0.480s

3.3ms/doc

Yura/CPython 3

20000

real 1m4.939s, user 1m3.556s, sys 0m0.360s

3.3ms/doc

Yura/CPython

400

real 0m1.408s, user 0m1.344s, sys 0m0.040s

3.5ms/doc

Antranig/Java

4000

3600ms

0.9ms/doc

=20

So, the above figures show that the best Javascript performance we have,= in Chrome, broadly equivalent to the worst Python performance, in Jython. = The pure Java implementation is at least 3x faster than the best Python per= formance. In Python we use the framework standard "xml.sax.xmlreader" packa= ge, whereas in Java we use the XPP3 pull parser implementation from Aleksander Slominski.

=20

This test set only contains 4 documents, read repeatedly from the filesy= stem. Therefore it neglects caching and memory image effects. These documen= ts are around 10k each - so the Java performance equates to a conversion sp= eed of around 10MB/sec, which would be roughly the expected sustained read = speed from a fairly good disk. If the files were kept compressed in a ZIP v= olume, the CPU cost would begin to dominate.

=20

## Persisti= ng the JSON Data in CouchDB

=20

In terms of writing the converted JSON documents to store, we can look a= t the following resource: http://aa= rtemenko.com/texts/couchdb-bulk-inserts-performance/. McCord indicates = that they have around 117,000 records of this form - corresponding to aroun= d 1.2Gb of raw data. The number of records puts us towards the left end of = the CouchDB insertion graphs - insertion rate is still high. The second gra= ph indicates that the very largest bulk sizes will lead to highest insertio= n rate - perhaps in excess of 1500 documents/second. This again would place= conversion speed as the bottleneck - on this hardware, CouchDb can perform= an insert in around 0.6ms. This would imply buffering data in units of aro= und 100Mb. If the fileset and DB were on the same machine, alternating read= ing and writing in these large units would also lead to better use of the m= achine's I/O capacity.

=20

## Summary

=20
=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20

Runtime

Time (ms/doc)

Throughput (Mb/sec)

Total McCord XML-JSON Conversion (minutes) <= /p>

Python C

3.5

2.86

6.99

Python JVM

7

1.43

13.99

JavaScript JVM

41

0.244

81.97

Java 6

0.7

14.23

1.40

=20

Some ballpark estimates for whole workflow=E2=80=94including conversion = and persistence=E2=80=94on the various platforms:

=20
=20
• If Java were used for the conversion, we would expect to be able to com= mit the set of 117,000 documents into CouchDb in around 3 minutes.
• =20
• In Python, this time would probably extend to around 8-9 minutes.
• = =20
• Perhaps a Javascript V8 (Chrome-like) solution might take more than 15 = minutes.
• =20
• A Rhino Javascript solution, performing somewhat Firefox 2-like, might = take 1-2 hours.
• =20
------=_Part_16243_972461663.1603803569240--
{"ft_lang_label":"__label__en","ft_lang_prob":0.85119945,"math_prob":0.80304134,"size":8198,"snap":"2020-45-2020-50","text_gpt3_token_len":2429,"char_repetition_ratio":0.13363437,"word_repetition_ratio":0.020911127,"special_character_ratio":0.3161747,"punctuation_ratio":0.12109863,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9939664,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-10-27T12:59:29Z\",\"WARC-Record-ID\":\"<urn:uuid:0f70962d-30c1-4029-8c8e-19cbd05d00fa>\",\"Content-Length\":\"20754\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:725c5292-beb4-4dc3-9819-09685d44f5b1>\",\"WARC-Concurrent-To\":\"<urn:uuid:04dfa0f0-d98b-46d7-98f2-0e9c1963a14c>\",\"WARC-IP-Address\":\"205.211.169.49\",\"WARC-Target-URI\":\"https://wiki.fluidproject.org/exportword?pageId=7930034\",\"WARC-Payload-Digest\":\"sha1:DF6LQLPJ26HBTCWOQIAR63R7T4RCQ7WG\",\"WARC-Block-Digest\":\"sha1:67OT42XYI4GDLVQNNKV3A7FVRDCY7GEP\",\"WARC-Identified-Payload-Type\":\"message/rfc822\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-45/CC-MAIN-2020-45_segments_1603107894175.55_warc_CC-MAIN-20201027111346-20201027141346-00420.warc.gz\"}"}