Does 汉语书 (Hànyǔ shū) "Chinese book" only mean "a book for learning Chinese", or can it also mean "a book in Chinese"?
In China a _____ is a :

____ 语文书：Chinese book for Chinese kids to learn Chinese reading and writing
____ 数学书: maths book
____ 英语书：book for Chinese kids to learn English  

一本汉语书 = 一本中文书

'汉语书'更多用于学习汉语的书，
'Chinese book' is mainly used to refer to  a book for studying Chinese, (language) 

但是根据'汉语书'的字面意思，
but the literal meaning of 'Chinese book'

可以理解为'汉字书籍'，
can be understood as 'Chinese character book'

也就是“book written in Chinese”.
i.e. “book written in Chinese”.

中文书则更多用于“book written in Chinese”.

As you so rightly said, the context will reveal all!