Title: WebHDFSClient Openfile cannot exceed 2Gb
Tags: c#;hadoop;hdfs;webhdfs
Question: I've encountered a problem with the WebHDFSClient in C#.
I try to access a file part by part in the Hadoop File System. It size is 5Gb.
However, the available methods to open a file are :
public Task<System.Net.Http.HttpResponseMessage&gt; OpenFile(string path);
public Task<System.Net.Http.HttpResponseMessage&gt; OpenFile(string path, int offset, int length);
As you can see, the two variables that I need, offset and length are integers. Therefore, it's impossible to retrieve a part beyond the first 2Gb.
Surprisingly, the WebHDFS REST API offers:
(https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#OPEN)
curl -i -L "http://<HOST&gt;:<PORT&gt;/webhdfs/v1/<PATH&gt;?op=OPEN
[&amp;offset=<LONG&gt;][&amp;length=<LONG&gt;][&amp;buffersize=<INT&gt;]"
Do you see a solution to access file exceeding 2Gb in C# ?
Thank you in advance.
Comment: Yes, that's finally the solution I've been using. Tank you !

Comment: It sounds like a problem of the C# API, not the webhdfs API. Since this is a webhdfs API, you could directly create http request following the webhdfs specification.



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:03/06/17', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<c#><hadoop><hdfs><webhdfs>', 'Title': 'WebHDFSClient Openfile cannot exceed 2Gb', 'ViewCount': 161, 'id': 42630034, 'user_AboutMe': '', 'user_AccountId': 8458885.0, 'user_DisplayName': 'Dragonsen', 'user_Reputation': 23.0, 'user_Views': 10.0, 'user_id': 6345278.0}}