   Generating random data that looks like real-world data
  Hey guys,Does anyone know where/what I can look into to generate random data that looks models real-world data? The issue I'm having is that I need to test out algorithms that recommend items based on rankings users give to other items. These algorithms are based on similarity searching (if I rate Item 1 5/5 and Item 7 is really similar to Item 1 then the system implicitly rates Item 7 as well). The problems is the data, don't the type I need. I have one dataset that has a bunch of movies, and 70000 users and around 4 million rankings.So I was thinking that I could analyze that dataset somehow, and find an algorithm that can generate that dataset procedurally. Then parameterize it on features I need in the dataset. Does anyone have any idea where I can look to? I was trying to find mathematical models of network traffic data and see if I could compare it to how the data is spread out in the movie dataset. Couldn't find any though, I'm out of ideas right now.Thanks for any help.   
  What kind of data are you modelling?Have you heard of Zipf's Law? Does it seem appropriate here?   ;  So your question is really vague but this might help you...I'm terrible for not remembering his name, and this is the really really short version but:There was a guy who was hired by IBM to figure out the source of some interference / data corruption on a chip they were making years back (like in the 70s i think)He came in and took a look and after a while realized the graph of the corruption looked exactly like the historical price of tea in china.bing bada boom, chaos theory was born.If you want to model more realistic data, check out chaos theory and fractals.Or, you can take some real world data (perhaps the average temperature for a year for where you live) and use that as the basis.::shrug::if you give some more details about your requirements, people can probably help you out more   ;  I'm not sure exactly what you're looking for, either. But you might look at hourly Dow-Jones averages vs Fortune 500 averages. They aren't identical, but they follow the same long-term trends.;  You might also want to consider Benford's law if you want the data to look real.   ;  Such data set cannot be generate synthetically.Assume you develop an algorithm that can accurate model your problem. This will require analyzing individual contributing factors, and modeling them as equation.In other words - finding an algorithm to generate a data set is a dual problem of your rating algorithm. When you have one, you can construct the other.For testing and training purposes, this make such generation somewhat redundant, since any data generated will inevitably try to match your other algorithm. Given any synthetically constructed data set, the best your rating will ever be able to match are the actual input parameters to your generation algorithm.   ;  Zahlman: Yes that seems very appropriate, thanks!I'm going to model the same data as the movie dataset. So, I will have X number of movies (or items, entities, whatever) and Y number of users. I want to be able to fake the Y users each rating n items on a scale from 1 to 10. And I want it to look like the distributions in the movie dataset I actually have.Atrix256: Sorry for the vagueness, hopefully the explanation above clears things up a bit? I think that Chaos Theory is going to come in very useful!Antheus: What you were saying is that modeling my data needs the rating algorithm and the algorithm needs to have the data already modeled?? There is no rating algorithm to begin with - I just want to assign random ratings to each of the items, and these random assignments need to look real. Thanks for other dataset suggestions too, guys.   ;  Not sure what your asking for exactly but if you haven't looked into Perlin Noise perhaps you should. This link explains it well.http://freespace.virgin.net/hugo.elias/models/m_perlin.htmHope that helps   ;Quote:Original post by IFooBarAntheus: What you were saying is that modeling my data needs the rating algorithm and the algorithm needs to have the data already modeled?? There is no rating algorithm to begin with - I just want to assign random ratings to each of the items, and these random assignments need to look real.For it to look real, someone must have a preconception of what 'real' looks like, with this data. This can't be universal - it is dependent on your application. eg. Real-looking data for size of planets in metres will not look real if measuring size of human in metres. Not only is the scale vastly different, but the distribution and range is different.Once you have found out what your definition of real is, you can use that to generate your data. At the least for numerical data you probably need a mean value, a distribution type, and any parameters to that distribution.   