   [Lua] Runtime compiling from C/C++
  Recently I've been interested in the network applications of Lua for prototyping out ideas in a server/client environment. I was thinking something along the lines of custom logic generated on a server being sent to a client for execution. For simple security reasons, it'd be best if this logic were not sent in a text format that could easily be modified by anyone, but rather in a compiled form.In order to do that though, I needed the ability to generate compiled Lua bytecode at runtime (from C/C++) without having to use the Luac program. I didn't want to have to write the logic to a file and then read in the compiled results to use. I definitely didn't want it to be bound by disk i/o either.However, I've not really been able to find any existing code that actually does this or any topics of the potential uses of it. So, I spent a little time hacking away at Luac's code to come up with a solution. This is what I ended up with, it seems to work fine, but I've not gone through any intensive testing yet. It is really just a basic PoC project that could be extended in many ways.Source:extern "C"{#include "zlib/zlib.h"#include "lua/lua.h"#include "lua/lualib.h"#include "lua/lauxlib.h"#include "lua/ldo.h"#include "lua/lundump.h"}#include <string>#include <sstream>#include <stdlib.h>#include <memory.h>#include <string.h>// Formats a stream of data into a hex/ascii dumpstd::string DumpToString(void * data_, size_t count);typedef struct TStreamBuffer{const char * src;size_t sze;size_t ind;char buff[LUAL_BUFFERSIZE];} TStreamBuffer;typedef struct TCompiledBuffer{char * ptr;size_t sze;int result;} TCompiledBuffer;int MyCustomWriter(lua_State * L, const void * src, size_t sze, void * userdata){TCompiledBuffer * cb = (TCompiledBuffer *)userdata;UNUSED(L);cb->ptr = (char *)realloc(cb->ptr, cb->sze + sze);if(cb->ptr == NULL){return 1;}memcpy(cb->ptr + cb->sze, src, sze);cb->sze += sze;return 0;}const char * MyBufferStreamer(lua_State * L, void * userdata, size_t * sze){TStreamBuffer * sb = (TStreamBuffer *)userdata;*sze = sizeof(sb->buff);UNUSED(L);memset(sb->buff, 0, sizeof(sb->buff));if(sb->ind >= sb->sze){*sze = 0;return NULL;}if(sb->sze - sb->ind < *sze){*sze = sb->sze - sb->ind;}memcpy(sb->buff, sb->src + sb->ind, *sze);sb->ind += *sze;return sb->buff;}TCompiledBuffer CompileString(lua_State * L, const char * input){const Proto * proto = 0;TCompiledBuffer cb = {0};TStreamBuffer sb = {0};sb.sze = strlen(input);sb.src = input;cb.result = lua_load(L, MyBufferStreamer, &sb, "");if(cb.result != 0){return cb;}lua_lock(L);proto = clvalue(L->top+(-1))->l.p;cb.result = luaU_dump(L, proto, MyCustomWriter, &cb, 1);lua_unlock(L);if(cb.result != 0){cb.sze = 0;if(cb.ptr){free(cb.ptr);cb.ptr = 0;}}return cb;}int main(int argc, char * argv[]){int res;lua_State * L = 0;lua_State * M = 0;TCompiledBuffer cb1;TCompiledBuffer cb3;L = lua_open();luaL_openlibs(L);cb1 = CompileString(L, "print(\"Hello World!\")\n");cb3 = CompileString(L, "-- create private index""\n""local index = {}""\n""""\n""-- create metatable""\n""local mt =""\n""{""\n""__index = function (t,k)""\n""print(\"*access to element \\\"\" .. tostring(k) .. \"\\\" in table \\\"\" .. t[index][\"name\"] .. \"\\\"\")""\n""return t[index][k]   -- access the original table""\n""end,""\n""""\n""__newindex = function (t,k,v)""\n""print(\"*update of element \\\"\" .. tostring(k) .. \"\\\" to \\\"\" .. tostring(v) .. \"\\\" in table \\\"\" .. t[index][\"name\"] .. \"\\\"\")""\n""t[index][k] = v   -- update original table""\n""end""\n""}""\n""""\n""function track (t)""\n""  local proxy = {}""\n""  proxy[index] = t""\n""  setmetatable(proxy, mt)""\n""  return proxy""\n""end""\n""""\n""tbl1 = {}""\n""tbl1[\"name\"]=\"tbl1\"""\n""tbl1 = track(tbl1)""\n""""\n""tbl1[\"id\"] = 0""\n");lua_close(L);L = 0;M = lua_open();luaL_openlibs(M);if(cb1.result == 0){uLongf destLen;Bytef * dest;res = luaL_loadbuffer(M, cb1.ptr, cb1.sze, "");res = lua_pcall(M, 0, LUA_MULTRET, 0);destLen = cb1.sze * 2 < 1024 ? 1024 : cb1.sze * 2;dest = (Bytef *)malloc(destLen);memset(dest, 0, destLen);res = compress2(dest, &destLen, (Bytef *)cb1.ptr, cb1.sze, Z_BEST_COMPRESSION);if(res != Z_OK){printf("compress2 failed with error: %i\n", res);}printf("Original (%i):\n", cb1.sze);printf("%s\n", DumpToString(cb1.ptr, cb1.sze).c_str());printf("Compressed (%i):\n", destLen);printf("%s\n", DumpToString(dest, destLen).c_str());free(cb1.ptr);free(dest);}else{printf("There was an error compiling the input string.\n");}if(cb3.result == 0){uLongf destLen;Bytef * dest;res = luaL_loadbuffer(M, cb3.ptr, cb3.sze, "test");res = lua_pcall(M, 0, LUA_MULTRET, 0);destLen = cb3.sze * 2 < 1024 ? 1024 : cb3.sze * 2;dest = (Bytef *)malloc(destLen);memset(dest, 0, destLen);res = compress2(dest, &destLen, (Bytef *)cb3.ptr, cb3.sze, Z_BEST_COMPRESSION);if(res != Z_OK){printf("compress2 failed with error: %i\n", res);}printf("Original (%i):\n", cb3.sze);printf("%s\n", DumpToString(cb3.ptr, cb3.sze).c_str());printf("Compressed (%i):\n", destLen);printf("%s\n", DumpToString(dest, destLen).c_str());free(cb3.ptr);free(dest);}else{printf("There was an error compiling the input string.\n");}lua_close(M);M = 0;return 0;}// Formats a stream of data into a hex/ascii dumpstd::string DumpToString(void * data_, size_t count){char msg[256] = {0};std::stringstream smsg;int sze = count;int outputsize = sze;if(outputsize % 16 != 0) outputsize += (16 - outputsize % 16);if(outputsize == 0) outputsize = 16;int ctr = 0;char ch[17] = {0};int x1 = 0;unsigned char * data = reinterpret_cast<unsigned char *>(data_);for(int x = 0; x < outputsize; ++x){if(x < sze){unsigned char b = data[x];sprintf_s(msg + ctr, sizeof(msg) - ctr, "%.2X ", b);ch[x1] = (isprint(b) && !isspace(b) ? b : '.');}else{sprintf_s(msg + ctr, sizeof(msg) - ctr, "   ");ch[x1] = '.';}x1++;ctr += 3;if((x+1) % 16 == 0){x1 = 0;sprintf_s(msg + ctr, sizeof(msg) - ctr, "  %s", ch);ctr += 18;sprintf_s(msg + ctr, sizeof(msg) - ctr, "\n");try{smsg << msg;}catch (std::exception & e){printf("[%s] %s\n", __FUNCTION__, e.what());return "";}ctr = 0;memset(msg, 0, sizeof(msg));}}return smsg.str();}Example Output:Hello World!Original (97):1B 4C 75 61 51 00 01 04 04 04 08 00 00 00 00 00   .LuaQ...........00 00 00 00 00 00 00 00 00 00 02 02 04 00 00 00   ................05 00 00 00 41 40 00 00 1C 40 00 01 1E 00 80 00   ....A@...@......02 00 00 00 04 06 00 00 00 70 72 69 6E 74 00 04   .........print..0D 00 00 00 48 65 6C 6C 6F 20 57 6F 72 6C 64 21   ....Hello.World!00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................00   ................Compressed (71):78 DA 93 F6 29 4D 0C 64 60 64 61 61 E1 60 40 05   x...)M.d`daa.`@.4C 4C 2C 40 92 15 88 1D 1D 18 18 64 1C 18 18 E5   LL,@.......d....18 1A 18 98 80 7C 16 36 20 51 50 94 99 57 C2 C0   .....|.6.QP..W..C2 0B 64 7A A4 E6 E4 E4 2B 84 E7 17 E5 A4 28 32   ..dz....+.....(2A0 03 00 AC 3F 09 B4....?...........*update of element "id" to "0" in table "tbl1"Original (727):1B 4C 75 61 51 00 01 04 04 04 08 00 00 00 00 00   .LuaQ...........00 00 00 00 00 00 00 00 00 00 02 04 17 00 00 00   ................0A 00 00 00 4A 80 00 00 A4 00 00 00 00 00 00 00   ....J...........49 80 00 80 A4 40 00 00 00 00 00 00 49 80 80 80   I....@......I...A4 80 00 00 00 00 00 00 00 00 80 00 87 80 00 00   ................8A 00 00 00 87 C0 00 00 85 C0 00 00 89 C0 40 82   ..............@.85 80 00 00 C5 C0 00 00 9C 80 00 01 87 C0 00 00   ................85 C0 00 00 89 80 C1 82 1E 00 80 00 07 00 00 00   ................04 08 00 00 00 5F 5F 69 6E 64 65 78 00 04 0B 00   .....__index....00 00 5F 5F 6E 65 77 69 6E 64 65 78 00 04 06 00   ..__newindex....00 00 74 72 61 63 6B 00 04 05 00 00 00 74 62 6C   ..track......tbl31 00 04 05 00 00 00 6E 61 6D 65 00 04 03 00 00   1......name.....00 69 64 00 03 00 00 00 00 00 00 00 00 03 00 00   .id.............00 00 00 00 00 07 00 00 00 0A 00 00 00 01 02 00   ................08 11 00 00 00 85 00 00 00 C1 40 00 00 05 81 00   ..........@.....00 40 01 80 00 1C 81 00 01 41 C1 00 00 84 01 00   .@.......A......00 86 81 01 00 86 01 41 03 C1 41 01 00 D5 C0 81   .......A..A.....01 9C 40 00 01 84 00 00 00 86 80 00 00 86 40 00   ..@...........@.01 9E 00 00 01 1E 00 80 00 06 00 00 00 04 06 00   ................00 00 70 72 69 6E 74 00 04 15 00 00 00 2A 61 63   ..print......*ac63 65 73 73 20 74 6F 20 65 6C 65 6D 65 6E 74 20   cess.to.element.22 00 04 09 00 00 00 74 6F 73 74 72 69 6E 67 00   "......tostring.04 0D 00 00 00 22 20 69 6E 20 74 61 62 6C 65 20   .....".in.table.22 00 04 05 00 00 00 6E 61 6D 65 00 04 02 00 00   "......name.....00 22 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ."..............00 00 00 00 00 00 00 0C 00 00 00 0F 00 00 00 01   ................03 00 0B 14 00 00 00 C5 00 00 00 01 41 00 00 45   ............A..E81 00 00 80 01 80 00 5C 81 00 01 81 C1 00 00 C5   .......\........81 00 00 00 02 00 01 DC 81 00 01 01 02 01 00 44   ...............D02 00 00 46 42 02 00 46 42 C1 04 81 82 01 00 15   ...FB..FB.......81 02 02 DC 40 00 01 C4 00 00 00 C6 C0 00 00 C9   ....@...........80 80 00 1E 00 80 00 07 00 00 00 04 06 00 00 00   ................70 72 69 6E 74 00 04 15 00 00 00 2A 75 70 64 61   print......*upda74 65 20 6F 66 20 65 6C 65 6D 65 6E 74 20 22 00   te.of.element.".04 09 00 00 00 74 6F 73 74 72 69 6E 67 00 04 07   .....tostring...00 00 00 22 20 74 6F 20 22 00 04 0D 00 00 00 22   ...".to."......"20 69 6E 20 74 61 62 6C 65 20 22 00 04 05 00 00   .in.table.".....00 6E 61 6D 65 00 04 02 00 00 00 22 00 00 00 00   .name......"....00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................00 12 00 00 00 17 00 00 00 02 01 00 05 09 00 00   ................00 4A 00 00 00 84 00 00 00 49 00 00 01 85 00 00   .J.......I......00 C0 00 80 00 04 01 80 00 9C 40 80 01 5E 00 00   ..........@..^..01 1E 00 80 00 01 00 00 00 04 0D 00 00 00 73 65   ..............se74 6D 65 74 61 74 61 62 6C 65 00 00 00 00 00 00   tmetatable......00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................00 00 00 00 00 00 00................Compressed (395):78 DA 9D 52 41 4A 03 41 10 AC DE 4C 4C 34 8A 28   x..RAJ.A...LL4.(06 0F 8A 2C 39 7A F3 07 1B 51 41 F1 E2 5D 94 35   ...,9z...QA..].519 25 98 6C 82 19 D1 E3 6C 30 11 7D 43 F0 94 87   .%.l....l0.}C...64 C1 08 3E C0 9B 8F B1 66 4D 14 45 3D D8 50 3B   d..>....fM.E=.P;D5 33 CD 74 55 CF AE EC 5F 86 07 10 A5 54 1E 5F   .3.tU..._....T._C3 53 CB FC CE 10 7B 16 18 8C 77 77 2D EC 20 98   .S....{...ww-...70 6B 07 76 52 6F 71 4B 7E 4F 76 3B 04 BA C4 DD   pk.vRoqK~Ov;....30 E8 74 B9 37 22 EF 5B C8 C7 BE 4D 3A 6B AC CF   0.t.7".[...M:k..B1 36 6D 7B 7C 5C 8B AA FA 1A AA 90 26 91 BE 1A   .6m{|\......&...E7 53 CC CD 45 58 39 87 CA 3A 7A 52 DF 78 67 51   .S..EX9..:zR.xgQD8 D0 50 19 B2 5A 15 99 89 84 09 C9 8D 85 8B 87   ..P..Z..........FC 02 D7 2E 91 50 74 36 06 02 B1 58 8D 21 E5 04   .....Pt6...X.!..B8 11 A0 17 0B 7A 52 CE 24 65 C1 CB 30 96 7E 00   .....zR.$e..0.~.B9 61 7D 8F C2 7B E4 0F BC C6 89 75 52 52 3D AD   .a}..{.....uRR=.8B 5A 64 A0 8A A4 EB 61 A5 A2 DB 6D DF 34 7D 5D   .Zd....a...m.4}]D7 0D 1D 19 BF 04 35 ED 94 36 DB 86 75 67 50 73   ......5..6..ugPsCC 4A 7E 2D F2 4D 78 52 D7 EE FC 53 BF E7 CE F0   .J~-.MxR...S....53 CC 12 F3 CE 41 06 85 25 AE 23 C7 CB C0 36 1D   S....A..%.#...6.58 3A 38 A4 83 98 0E 46 B1 7B 29 C8 2B 73 F1 04   X:8....F.{).+s..5B BC 72 67 D3 23 12 15 77 04 C5 D8 F3 5E E9 E2   [.rg.#..w....^..91 65 4F 1C FD B3 B5 F8 18 FD 77 37 97 AD 6A 68   .eO.......w7..jhB4 DF 3C FD DD 4D 2E 75 43 BF A5 FF 3A 5B 24 DC   ..<..M.uC...:[$.8F 45 B1 59 77 F7 1E E1 E6 BD 4B 83 EE 9D 86 54   .E.Yw.....K....TA7 E8 B0 1F 58 39 1A CF 5E 9C 5A D7 AE AD 4D 43   ....X9..^.Z...MC9B 30 6D 88 BF E2 0D AC 13 75 FB   .0m......u......Press any key to continue . . .Download:luaRuntimeCompile (complete project)Basically, I want to explore sending logic to a client to execute rather than sticking to the traditional approach of sending data. I want to cut out the client's low level data processing and aim for something higher level now. I don't actually need the ability to compile Lua scripts at run time for that, since sending strings and calling DoString is good enough, but having it makes it all the more practical for a real demo.Just though I'd share in case anyone else had a need like this. Please feel free to leave any questions or comments!   
  Do you do any compression on those packets? looks like even simple RLE will get you some savings.[EDIT: i see from browsing the output arrays, that you do compress cool!]-ddn[Edited by - ddn3 on April 28, 2010 7:06:41 PM];Quote:Original post by Drew_BentonRecently I've been interested in the network applications of Lua for prototyping out ideas in a server/client environment. I was thinking something along the lines of custom logic generated on a server being sent to a client for execution. For simple security reasons, it'd be best if this logic were not sent in a text format that could easily be modified by anyone, but rather in a compiled form.Sending lua bytecode really isn't any more secure than sending source. Lua bytecode retains all variable names (including locals) and even line numbers, so decompilation is trivial. Mind you, I have not heard of any such decompiler, but I would guess that such a thing does exist, and if not, wouldn't take much to write.Even if you could obfuscate the code in such a way as to prevent decompilation, a properly inclined user could run arbitrary code anyways. I don't know much about security but I think you just have to assume that the user has full control over their machine, and work from there. Besides, it is rather degrading to assume or enforce that the user's machine is not soveriegn. I think the most robust strategy is to do validation and capability checking server-side, which theoretically can be secured.I hope this helps.EDIT: Sending bytecode might still be a good idea if you can compress it smaller. But again, don't rely on that for security.   