What kind of filler was used for this patio?
If your brick is level, and moves water away from your house, I wouldn't replace any of it. It takes a lot of time and expertise to get it to set just right, and probably isn't worth the trouble.

If you are really bent on making it look uniform, take a paint scraper, something simple you can get from your local hardware store like this.  The pointed end is great for clearing debris between bricks and the like.  Use the scraper to clean out as much of the dirt and junk from between the bricks.  You can then use a standard leveling sand (which you can also get from your local hardware store), and sweep it into the cracks.  

Normally, the purpose of the sand is to fill in the cracks between the bricks to keep them from moving.  However, if you get a sand which is lighter in color, then it would probably match well.