Serial monitor returns "48" when I enter "0"
Serial.println(incomingByte, DEC);



You told it to print the incoming byte in decimal, so that is what it did. 48 is the decimal equivalent of ASCII '0'.

Change that line to:

Serial.println((char) incomingByte);


That casts the thing you are printing to a character, and then the appropriate version of Serial.println prints it as a character, not a decimal number.



10 is the decimal equivalent of a new line (newline) so that is why you are also seeing 10 printed.




  I need to convert the incoming stuff into integers so that it can see if im guessing right or not


Assuming you mean an integer other than 0 to 9 (like, 42 for example) then you need a bit more work. I have a page about that sort of thing:

http://www.gammon.com.au/serial

Example code:

/*
Example of processing incoming serial data without blocking.

Author:   Nick Gammon
Date:     13 November 2011. 
Modified: 31 August 2013.

Released for public use.
*/
// how much serial data we expect before a newline
const unsigned int MAX_INPUT = 50;

void setup ()
  {
  Serial.begin (115200);
  } // end of setup

// here to process incoming serial data after a terminator received
void process_data (const char * data)
  {
  // for now just display it
  // (but you could compare it to some value, convert to an integer, etc.)
  Serial.println (data);
  }  // end of process_data

void processIncomingByte (const byte inByte)
  {
  static char input_line [MAX_INPUT];
  static unsigned int input_pos = 0;

  switch (inByte)
    {

    case '\n':   // end of text
      input_line [input_pos] = 0;  // terminating null byte

      // terminator reached! process input_line here ...
      process_data (input_line);

      // reset buffer for next time
      input_pos = 0;  
      break;

    case '\r':   // discard carriage return
      break;

    default:
      // keep adding if not full ... allow for terminating null byte
      if (input_pos &lt; (MAX_INPUT - 1))
        input_line [input_pos++] = inByte;
      break;

    }  // end of switch

  } // end of processIncomingByte  

void loop()
  {
  // if serial data available, process it
  while (Serial.available () &gt; 0)
    processIncomingByte (Serial.read ());

  // do other stuff here like testing digital input (button presses) ...

  }  // end of loop


In this example, the function process_data is called when there is a complete number to be handled.