|
 Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
 How Your Intuition Can Fool You 
#13255199 - 09/27/10 05:46 PM (13 years, 6 months ago) |
|
|
If I asked you to take a random collection of real-world data, for example:
your electric bill last month
your age
your weight
the size of the solar system
the amount of your last IRS return
a random number printed in today's newspaper
the probability of rain tomorrow
the number of letters in your pet's name
the temperature of the surface of the sun
the number of people currently online on Shroomery
the length of the Nile river
today's Dow stock index
the area of Lake Michigan
Take all those numbers and look ONLY at the first digit. Would any particular number appear more often than any other or would 1 through 9 appear evenly often?
Intuitively, it seems the distribution would be even (all numbers appear equally often), but they don't. The digit 1 will appear in that list 6 times more often than the digit 9!
--
OK, first a quick math primmer. Bear with me, it's short and the story that comes next pretty amazing.
Logarithms are mathematical objects that can be used to simplify working with large numbers. They're tedious and error prone to calculate by hand, so long ago, before calculators, they were computed ahead of time and published in huge tables called a logarithm book. When a scientist needed to know the logarithm of a particular number, instead of calculating it himself, he just went to the library and looked it up in the book. Today a calculator or computer does the same job.
End of primmer. Now the story.
Back in 1881, mathematician Simon Newcomb noticed that the pages containing the logarithms for numbers that begin with the digit 1 (for example: 104, 1289, 1988734, and so on) were filthy dirty and worn from repeated use. He also noticed that the pages became progressively cleaner as the first digit in question approached 9.
This meant that people were looking up the logarithms for numbers beginning with 1 more often than those beginning with 2 and so on until the least often looked up numbers were those beginning with 9.
Then he made the astonishing deduction that any collection of random real-world data would show the same distribution. He even formulated it formally. Here it is for the math geeks:
The probability that first digit of random real-world data will be d is log10 (1 + 1 / d) for d = 1, 2, ..., 9
About 50 years after Newcomb, physicists Frank Benford independently made the same discovery about logarithm books in his library and even came up with the exact same equation. The effect is now called Benford's Law even though it was discovered by Newcomb.
Here's the astonishing part.
Take any random set of real-world data (like the list at the beginning of this post) put them all together and look at the first digit of each. The digit 1 will appear 30% of the time, the digit 2 will appear 17% of the time and so on to the digit 9 which will appear only 4.6% of the time. The larger the sample of data, the more exactly this distribution emerges.
The probability can be calculated from the above equation. Here it is graphically:
Also, the effect is what mathematicians call Scale Invariant. That means that if you analyze the length of the 100 longest rivers on Earth, the Benford distribution will emerge no mater if you measure in miles, kilometers, inches, millimeters, or cubits.
Here's a real example. Take the list of the 60 tallest structures on Earth and analyze the distribution of the first digit Here's what you get in meters and in feet:
Code:
First Meters Feet
Digit Count % Count %
1 26 43.3% 18 30.0%
2 7 11.7% 8 13.3%
3 9 15.0% 8 13.3%
4 6 10.0% 6 10.0%
5 4 6.7% 10 16.7%
6 1 1.7% 5 8.3%
7 2 3.3% 2 3.3%
8 5 8.3% 1 1.7%
9 0 0.0% 2 3.3%
This bizarre, completely counter-intuitive result is so consistent that the IRS uses it to spot phony tax returns because tax cheats use random numbers on their returns. Random numbers are uniformly distributed and so don't conform to Benford's Law. By analyzing returns and flagging for audit those whose first digits don't conform to the Benford distribution, they've caught a lot of tax cheats. That's how consistent the Benford distribution is in nature.
Forensic accountants use it routinely to discover insurance fraud, accounting fraud, and even voting fraud. Apparently, the recent disputed elections in Iran don't conform to the Benford distribution.
I really hope this doesn't go over the math-challenged head because it's pretty amazing if you understand what I just said.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Cups
technically "here"
Registered: 12/24/09
Posts: 1,925
|
 Re: How Your Intuition Can Fool You [Re: Diploid]
#13255225 - 09/27/10 05:53 PM (13 years, 6 months ago) |
|
|
Yes, yes...but what does it all mean Basil?
That there is order even in the most random of the randomness?
--------------------
What's up everybody?!
|
 Techie
Is Broken
Registered: 09/17/10
Posts: 23
Loc: UK 
Last seen: 13 years, 6 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13255243 - 09/27/10 05:59 PM (13 years, 6 months ago) |
|
|
fUck
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Cups]
#13255281 - 09/27/10 06:11 PM (13 years, 6 months ago) |
|
|
but what does it all mean
Even though this was discovered 130 years ago, it is still a little mysterious why it happens. Mathematicians have formulated various constructions to get a handle on it, but no really good explanation has emerged. It's intuitive for some statistical populations that grow logarithmically, like salaries, but the effect appears in every sort of statistical sample except those with Gaussian or normal distributions, like IQ or height which the overwhelming majority of people fall into a narrow range so the effect is absent.
There's a deeper, slightly more technical analysis here:
http://www.tphill.net/publications/BENFORD%20PAPERS/TheFirstDigitPhenomenonAmericanScientist1996.pdf
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
circastes
Big Questions Small Head
Registered: 01/14/10
Posts: 8,781
Loc: straya 
Last seen: 7 years, 10 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13255390 - 09/27/10 06:35 PM (13 years, 6 months ago) |
|
|
More proof that we are all One.
--------------------
My solitude...
My shield...
My armour...
TESTED
WITH
FULL
FORCE
|
NetDiver
Wandering Mindfuck
Registered: 08/24/09
Posts: 6,024
Loc: Everywhere and Nowhere
Last seen: 1 year, 8 months
|
Re: How Your Intuition Can Fool You [Re: circastes]
#13255448 - 09/27/10 06:47 PM (13 years, 6 months ago) |
|
|
Quote:
circastes said:
More proof that we are all One.
|
laserpig
Weedmaster_P
Registered: 04/28/09
Posts: 7,468
Last seen: 11 years, 10 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13255453 - 09/27/10 06:48 PM (13 years, 6 months ago) |
|
|
i don't fully understand the math you just presented, but i would like to address the topic of intuition generally:
intuition does not give you answers
intuition pushes you towards the right questions
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: laserpig]
#13255513 - 09/27/10 07:01 PM (13 years, 6 months ago) |
|
|
That's why intuition pushes you toward the scientific method.
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
laserpig
Weedmaster_P
Registered: 04/28/09
Posts: 7,468
Last seen: 11 years, 10 months
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13255543 - 09/27/10 07:05 PM (13 years, 6 months ago) |
|
|
it really does
even as a little kid, i "intuitively" did experiments when i was curious about stuff
for example: "wow! magnifying glasses burn paper! will they burn wet paper? i bet they won't. wow! they will! it just takes a li'l longer!"
ka-blam -- science
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: laserpig]
#13256090 - 09/27/10 08:41 PM (13 years, 6 months ago) |
|
|
Actually I believe that it won't. The paper must first dry before the gases are release which are what burns. So actually wet paper won't burn.
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
laserpig
Weedmaster_P
Registered: 04/28/09
Posts: 7,468
Last seen: 11 years, 10 months
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13256119 - 09/27/10 08:47 PM (13 years, 6 months ago) |
|
|
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13256227 - 09/27/10 09:03 PM (13 years, 6 months ago) |
|
|
Quote:
laserpig said:
it really does
even as a little kid, i "intuitively" did experiments when i was curious about stuff
for example: "wow! magnifying glasses burn paper! will they burn wet paper? i bet they won't. wow! they will! it just takes a li'l longer!"
ka-blam -- science
yep, this is logic and is all science is
Do something and think why it happened, then see if your reasoning holds up after further experience
people criticising science and its abilities, limitations, relative to other modlities seem to almost always have little understanding of what science is and never seem to show anything better.
Quote:
Diploid said:
but what does it all mean
Even though this was discovered 130 years ago, it is still a little mysterious why it happens. Mathematicians have formulated various constructions to get a handle on it, but no really good explanation has emerged. It's intuitive for some statistical populations that grow logarithmically, like salaries, but the effect appears in every sort of statistical sample except those with Gaussian or normal distributions, like IQ or height which the overwhelming majority of people fall into a narrow range so the effect is absent.
There's a deeper, slightly more technical analysis here:
http://www.tphill.net/publications/BENFORD%20PAPERS/TheFirstDigitPhenomenonAmericanScientist1996.pdf
Interesting article and great post. I enjoyed it : )
Off hand it would seem one would be more frequent simply because it is the first number which may be used to represent signifigant data, and hence it will be more likely to occur than the higher digits in measurements simply because of the fact that it is always among the digits that are greator or less than the digit representing the first digit in the data. This being the case, measurements would always be of suficient magnitude to be represented by one but not necessarily of sufficient magnitude to be represented by a larger digit- if that makes sense
Additionally, I would assume that the magnitude of measurements scale up at a rate higher than linear- i.e. when measuring things of a particular nature, that the data more quickly doubles from 1000+x to 2000+x then 10+x to 20+x (where x is the magnitude of the data, the order of magnitude or whatever)
If this is the case, it would seem natural that the number of data points begining with low digits would be signifigantly larger than those with high digits as there would be more data points in the range of 1,000+x to 1,999+x then 2,000+x to 2,999+x
I would think that some mathmatical reasoning could be derived from the fact that measurements often relate to variables with non-linear realtionships to the data produced: area of a circle is pi radius squared, for example. as the radius increases, the area increases more quickly, hence there will be more data points between intervals of lower magnitude- and these will more often, geometrically greater probability, be lower digits than higher.
Just off the cuff here, so probably something I missed in there. Obviously, just cuz I can say it doesn't mean it holds water mathmatically or experimentally- not claiming this
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13256259 - 09/27/10 09:08 PM (13 years, 6 months ago) |
|
|
Quote:
Icelander said:
Actually I believe that it won't. The paper must first dry before the gases are release which are what burns. So actually wet paper won't burn.
I would imagine the drying before burning simply happens because usually the ignition point of the paper is a higher temperature than the boiling point of the liquid, i.e. water, and since it takes a bit to warm the paper, especially the more massive wet paper with water's higher heat capacity, it is likely to dry before ignition.
I don't see any reason this temporal happenstance is required to burn the paper though. If you had a presurized ogygen atmosphere, for example, at some point you could surely burn wet paper- though obviously the water doesn't burn but you could keep it from vaporizing substantially by having it at equilibrium with ice
Interesting question
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: johnm214]
#13256318 - 09/27/10 09:21 PM (13 years, 6 months ago) |
|
|
I actually thought of that. I wasn't really sure that under all conditions wet paper wouldn't burn. But it's not really the paper burning right? It's the gases released by the heating of the paper that burn and the ash is what is left minus the gas? Is that right?
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13256782 - 09/27/10 10:49 PM (13 years, 6 months ago) |
|
|
Quote:
Icelander said:
I actually thought of that. I wasn't really sure that under all conditions wet paper wouldn't burn. But it's not really the paper burning right? It's the gases released by the heating of the paper that burn and the ash is what is left minus the gas? Is that right?
I'm not sure, but I suspect that yes, the paper really burns (if we're distinguishing between paper and vapor from the paper). The stuff paper is made out of isn't particularly volatile so it wouldn't seem vapor would form first, but combustion has all sorts of crazy chemical reactions that you'd never normally expect to observer, so I really can't say.
You may be thinking of volatile organic liquids when you wonder whether it is the vapor which combusts, however; these seem dissimilar. Things like gasoline combust as vapor due to the fact that they give off vapor constantly so long as they are liquid, and have a pretty low boiling point, much lower than the combustion point. Because you need an oxidizer (oxygen) to burn fuels (paper or gasoline), and the gasoline liquid is a relatively dense (compared to vapor or the atmosphere), there's not enough oxygen in close contact with the gasoline molecules as a liquid under normal conditions to support combustion. When it vaporizes it'll be a molecule surrounded by tons of oxygen in between the next molecule of gasoline(not one molecule in a dense sea of many) floating about, and can easily contact and react with oxygen. It's not so much that the gasoline liquid won't combust, its that under normal circumstances there's always some vapor that's going to combust much sooner than the liquid will (i.e. if you put a match to gasoline) so since the combustion temp will be much lower for the vapor, the vapor will always start burning long before the liquid got close to its ignition temp.
If you pressurized oxygen with the liquid though I'm sure it would eventually combust as you increase the pressure, even if you ensure essentially no vapor is present.
Basically, paper seems pretty porous (allowing better oxygen-fuel surface area, though solids are much worse than vapors in forming these points of contact) and since it doesn't seem like it would put off vapor to start the combustion, I'm guessing that this isn't what is burning. I'm sure it does put off some portion of combustible gas while burning, I just doubt this is what initiates the combustion or that its a very important part of it realtive to the direct burning of the solid.
Just my thoughts though, I don't know of any authoritative answer on this.
Edited by johnm214 (09/27/10 10:56 PM)
|
g00ru
lit pants tit licker
Registered: 08/09/07
Posts: 21,088
Loc: georgia, us
Last seen: 5 years, 4 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13256906 - 09/27/10 11:23 PM (13 years, 6 months ago) |
|
|
This argument rests on the assumption that intuition would tell you that the data would be evenly distributed amongst the digits. To me that's not intuition, that's a best guess. Intuition is somfin differnt
--------------------
check out my music!
drowse in prison and your waking will be but loss
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: johnm214]
#13257810 - 09/28/10 07:23 AM (13 years, 6 months ago) |
|
|
Well I'm pretty sure that when wood burns it's the gases released that are burning and the ash is the non combustible mineral leftovers.
http://www.qa02.com/Science/how-does-wood-burn.html
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
OrgoneConclusion
Blue Fish Group
Registered: 04/01/07
Posts: 45,441
Loc: Under the C
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13257916 - 09/28/10 07:57 AM (13 years, 6 months ago) |
|
|
Quote:
Icelander said:
That's why intuition pushes you toward the scientific method.
Tell intuition to GET OFF MY FUCKING BACK! 
--------------------
|
MushroomTrip
Dr. Teasy Thighs
Registered: 12/02/05
Posts: 14,794
Loc: red panda village
Last seen: 3 years, 1 month
|
Re: How Your Intuition Can Fool You [Re: circastes]
#13258094 - 09/28/10 08:49 AM (13 years, 6 months ago) |
|
|
Quote:
circastes said:
More proof that we are all  .
--------------------
    
All this time I've loved you
And never known your face
All this time I've missed you
And searched this human race
Here is true peace
Here my heart knows calm
Safe in your soul
Bathed in your sighs
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
|
Quote:
OrgoneConclusion said:
Quote:
Icelander said:
That's why intuition pushes you toward the scientific method.
Tell intuition to GET OFF MY FUCKING BACK!
YOU CAN'T HANDLE INTUITION!
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
OrgoneConclusion
Blue Fish Group
Registered: 04/01/07
Posts: 45,441
Loc: Under the C
|
Re: How Your Intuition Can Fool You [Re: Icelander]
#13258159 - 09/28/10 09:05 AM (13 years, 6 months ago) |
|
|
--------------------
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
 Re: How Your Intuition Can Fool You [Re: laserpig]
#13258205 - 09/28/10 09:19 AM (13 years, 6 months ago) |
|
|
i don't fully understand the math you just presented
OK, let me try to simplify a little.
Let's say you took a random list of real-world numbers. Things like the your electric bill, the length of the Mississippi river, the odometer reading in your car. Just any random set of real-world numbers like that.
Now think about them for a second. Would you say that this collection of numbers would have some sort of pattern or since they came from just anything that randomly occurred to you to pic, that there would be no pattern at all... that the numbers would be completely random?
Intuition tells me that there would be no pattern. Why should there be? It's just a jumble of random numbers you picked out. But there IS a pattern.
If you looked only at the first digit of each of those number and threw away the rest, you would find that 1 appears six times for every 9.
Here's an actual example. Let's look at the length (in feet) of the 20 tallest structures in the world by category:
Code:
Burj Khalifa 2,717
KVLY TV Mast 2,063
Guangzhou TV Mast 2,001
CN Tower 1,815
BREN Tower 1,516
Lualualei Transmitter Tower 1,503
Petronas Tower 1,482
Ekibstuz Power Station 1,377
Dimona Radar 1,312
Belmont Transmitter 1,272
Kiev TV Tower 1,263
Gerbrandy Tower 1,203
Yangtze River Crossing 1,137
Millau Viaduct 1,122
Nurek Dam 987
Grande Dixence Dam 935
Nanjing River Crossing 843
NTT Building 790
Hassan Mosque 751
Fuhrlander Wind Turbine 689
Niederaussem Power Station 673
St Louis Arch 656
Anaconda Smelter 630
Olympic Statium Canada 585
San Jacinto Monument 574
Chicabo Temple 570
Singapore Flyer 568
Ulm Minster 548
National Flag Square 548
Kenedy Space Center VAB 541
Now look at the first digit only. Here's that list:
2
2
2
1
1
1
1
1
1
1
1
1
1
1
9
9
8
7
7
6
6
6
6
5
5
5
5
5
5
5
Do you see 1 appearing more often than 2, and 2 appearing more often than 3 and so on until 9 which appears only twice?
Here's the distribution that always emerges:
First How Often
Digit It Appears
1 30.1%
2 17.6%
3 12.5%
4 9.7%
5 7.9%
6 6.7%
7 5.8%
8 5.1%
9 4.6%
This happens again if we do it in meters instead of feet. Or yards, or inches, or any unit you like. It always happens.
That's kinda weird. The pattern also holds if you analyze the second digit instead of the first, but it's not as prominent. If you analyze the last digit, they are completely random, as intuition expects.
So why should it be that the digit 1 appears first in random collections of data more often than 2, and 2 more often than 3 and so on? For some kinds of data, like salaries that grow faster as time passes, this makes sense. When you make $10,000 your raise is maybe $500, but when you make $100,000 your raise is $5,000, so it wraps around to 1 more often. In that case of logarithmic salary growth the Benford distribution makes intuitive sense.
But for random data like how many people are logged into Shroomery every day of last month, or the odometer reading for every car in your town, there's no obvious reason why this should happen, but somehow it does and it's weird.
I hope that made more sense.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Kickle
Wanderer
Registered: 12/16/06
Posts: 17,979
Last seen: 18 minutes, 40 seconds
|
|
Wouldn't this throw off almost all statistical analyses on randomized experiments using more than 1 group?
I.e. a between-subjects experimental design
--------------------
Why shouldn't the truth be stranger than fiction?
Fiction, after all, has to make sense. -- Mark Twain
Edited by Kickle (09/28/10 09:27 AM)
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Kickle]
#13258248 - 09/28/10 09:29 AM (13 years, 6 months ago) |
|
|
Wouldn't this throw off almost all statistical analyses on randomized experiments?
Not necessarily because experimental data usually follows a Gausian distribution. A "Bell Curve". The Benford distribution doesn't apply in that case. Also, statistical analysis is looking at the magnitudes, not the first digits.
Take, for example, IQ scores. The vast majority of people fall right around 120 or so. Almost nobody is at 50 or 150 so there is a huge skew and Benford is absent.
Or take the height of people. Almost everyone is 4, 5, or 6 feet. But almost nobody is 10 feet tall or 2 feet short. Again, no Benford.
Also, statistical analysis is done on the entire number (it's magnitude), not just the first digit. There is no Benford pattern in the magnitude of real-world data. It's only in the first digit.
But I'll tell you something else that's weird. The wear pattern on the buttons of calculators show a distinct Benford distribution. The button for 1 is always six times more worn out than the button for 9. 
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13258260 - 09/28/10 09:33 AM (13 years, 6 months ago) |
|
|
Maybe it's because we see ourselves as a singularity. I = 1
Could happen.
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
Kickle
Wanderer
Registered: 12/16/06
Posts: 17,979
Last seen: 18 minutes, 40 seconds
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13258262 - 09/28/10 09:33 AM (13 years, 6 months ago) |
|
|
What the fuck normal curve...
What the fuck benford distribution...
What the fuck, math.
--------------------
Why shouldn't the truth be stranger than fiction?
Fiction, after all, has to make sense. -- Mark Twain
|
OrgoneConclusion
Blue Fish Group
Registered: 04/01/07
Posts: 45,441
Loc: Under the C
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13258301 - 09/28/10 09:44 AM (13 years, 6 months ago) |
|
|
How does the 'counting effect' play into this?
For example: every thread starts at page 1 100% of the time.
Some will go onto page 2 and fewer to page 3 and so on.
Now this may not fully explain your seemingly random data, but it sure accounts for lots of stuff.
--------------------
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
|
Yeah, it does seem intuitive in the case of thread pages because the length of threads tends to be short so 1 and 2 page threads prevail.
But, but then if you do the analysis in terms of the number of words in threads, or the number of letters, or the number of commas or how many times the word "the" appears, or even how many misspellings there are in a collection of threads, the pattern emerges.
I can't imagine any reason why it would happen in those other cases but apparently it does.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
TheSkyInYourEye
Stillness Bathing
Registered: 04/07/10
Posts: 185
Last seen: 7 years, 7 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13258613 - 09/28/10 11:11 AM (13 years, 6 months ago) |
|
|
Seemed intuitive from the on start to me.
--------------------
Subtlety is the language of experience.
|
NewWavePeace
Incorporated
Registered: 07/20/10
Posts: 989
Loc: New Jersey
Last seen: 9 years, 8 months
|
|
My intuition has never fooled me.
When I second guess my intuition I get fooled.
--------------------
See I'm at one with the waves, whereas my wifes breasts are at one with her knees.
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13258862 - 09/28/10 12:06 PM (13 years, 6 months ago) |
|
|
Quote:
Icelander said:
Well I'm pretty sure that when wood burns it's the gases released that are burning and the ash is the non combustible mineral leftovers.
http://www.qa02.com/Science/how-does-wood-burn.html
The reason I would doubt this being the main source of the combustion is because paper doesn't give off vapor normally, and the actual molecules in the paper are very large- sort of like a natural plastic. The gasoline molecules on the other hand are much smaller.
To even vaporize the molecules would be essentially impossible so it would have to be broken down, and this would require combustion anyways that would liberate energy. So while I'm sure your correct that their are burning vapors, it seems to me this can't start the combustion and isn't primary to it like the gasoline scenario.
Just my reasoning though, I don't actually know the answer.
Quote:
OrgoneConclusion said:
How does the 'counting effect' play into this?
For example: every thread starts at page 1 100% of the time.
Some will go onto page 2 and fewer to page 3 and so on.
Now this may not fully explain your seemingly random data, but it sure accounts for lots of stuff.
essentially what I reasoned in part above. I agree with this
Quote:
Diploid said:
Yeah, it does seem intuitive in the case of thread pages because the length of threads tends to be short so 1 and 2 page threads prevail.
But, but then if you do the analysis in terms of the number of words in threads, or the number of letters, or the number of commas or how many times the word "the" appears, or even how many misspellings there are in a collection of threads, the pattern emerges.
I can't imagine any reason why it would happen in those other cases but apparently it does.
for measurements your always going to have a list of data points, and as the number of points goes to infinity the plot will approch a continuous function that approximates the data. Since there's going to be points between 1+x and 2+x where x is the magnitude but there may not be points between 2+x and 3+x due to the magnitude exceeding that of the data, the same principle orgnoe mentions would seem to hold.
Quote:
Diploid said:
[...]for random data like how many people are logged into Shroomery every day of last month, or the odometer reading for every car in your town, there's no obvious reason why this should happen, but somehow it does and it's weird.
It is indeed counterintuitive but I think the non-linear dependance of the data to variables (such as the area of a circle example I mention above and your refrence to the logarithmic functions) generally shows that there will be more data points between an interval of lower magnitude than one of higher magnitude, even if the interval is of equal range, as indeed it would be in the case of 1+x to 2+x vs 2+x to 3+x
It seems this would be a very powerful 'force' leading to this result.
This is a great thread and an excellent example of why statistics and open reasoning is useful for all manner of things, and why conclusory dogmatic claims or common sense should not be relied upon where decent data is available.
|
Ahimsa
µdose
Registered: 01/11/07
Posts: 1,827
Last seen: 4 years, 4 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13259027 - 09/28/10 12:47 PM (13 years, 6 months ago) |
|
|
In one word 'beautiful'. Thanks man, that blew my mind today! 
Edit: Hold on. Could this be used for predicting the lottery more correctly?
Edited by Ahimsa (09/28/10 12:56 PM)
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Ahimsa]
#13259269 - 09/28/10 01:40 PM (13 years, 6 months ago) |
|
|
Nopers, the lottery isn't a measurement, its a random selection of digits.
And besides, this doesn't concern the values, only the first signifigant digit representing the values.
|
spectralis
Registered: 08/29/09
Posts: 196
|
Re: How Your Intuition Can Fool You [Re: johnm214]
#13259326 - 09/28/10 01:51 PM (13 years, 6 months ago) |
|
|
when you look at a digital stop watch, it's the Benford effect?
Quote:
counting
|
DieCommie
Registered: 12/11/03
Posts: 29,258
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13259338 - 09/28/10 01:53 PM (13 years, 6 months ago) |
|
|
Great thread, and some great responses.
I can help but think that these so called random numbers are not completely random. I think the list of tallest building really illustrated that point.
|
andrewss
precariously aggrandized
Registered: 08/17/07
Posts: 8,725
Loc: ohio
Last seen: 3 months, 30 days
|
Re: How Your Intuition Can Fool You [Re: DieCommie]
#13259455 - 09/28/10 02:14 PM (13 years, 6 months ago) |
|
|
i dunno bout yall but i aint take kindly to all this math and numbers and science
--------------------
Jesus loves you.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: DieCommie]
#13259539 - 09/28/10 02:26 PM (13 years, 6 months ago) |
|
|
I think the list of tallest building really illustrated that point.
It's the list of tallest structures BY CATEGORY. In other words, the tallest building, tallest radio tower, tallest bridge, and so on.
In any case, look up some other sample set like 100 random numbers picked out of today's newspaper and see it for yourself. I've been looking at this in detail for a while now and I'm still freaked out over it.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
DieCommie
Registered: 12/11/03
Posts: 29,258
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13259599 - 09/28/10 02:34 PM (13 years, 6 months ago) |
|
|
Yea, but its still not completely random. Buildings cant be built to any random height. They exist in an interval, an interval that spans an order of magnitude. So you get all the 1s from the upper portion of that interval, and the other numbers come by conveniently ignoring 0 as the first digit in the lower part of the interval. My initial thought is that if you use a base that is much larger than the interval (or much smaller), then this pattern will disappear. This pattern appears where the base is of similar size to the interval that the random numbers fall on. So if you have random numbers in (0,1) then you will get a even distribution. But if you have random numbers in (.5, 1.5) and you ignore the leading zero, then half of your numbers will start with a 1.
Just my first thoughts... I havent yet read your links, so maybe Im way off.
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13259879 - 09/28/10 03:33 PM (13 years, 6 months ago) |
|
|
I'm still freaked out over it.
God's jus fuckin wit ye.
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: DieCommie]
#13260062 - 09/28/10 04:07 PM (13 years, 6 months ago) |
|
|
Yea, but its still not completely random. Buildings cant be built to any random height.
Alright, that's a fair critique. But how about this:
I just analyzed the first 54 threads currently listed in this forum, excluding the stickies. I opened each thread, copied the entire text content into Microsoft Word and used its word count function to create the following table:
Code:
Thread Word
Title Count
god as a sadist and the odd idea of heaven... 7351
Lucid Dream 4198
the cure for death led to a 100% suicide rate... 8953
Does reality exist? 3516
How Your Intuition Can Fool You 8569
Doubts about psychedelics from Albert Hofmann, LSD's discoverer 1765
Man leaves behind 1905-page suicide note 1240
Talking philosophy instead of living it. 1993
The Origin of the Mushroom's Shape 2587
The Salesmen of Science 2672
"What happened before the big bang?" 2636
Karma! 1135
Just curious what's your guys feeling on DMT? 3929
Could the Buddha eat a Big Mac WScott 8352
Wilderness experiences. Your opinions? 1615
questionnaire for atheists...(atheist only please) 12808
Happiness or Love? 2090
sense of self 4926
Hierarchy 790
Bus adverts a human rights issue... 1183
Sing! Sing a song!!!! 1168
"God, grant me the serenity..." 4991
Can I meet "God" while tripping...if I'm athiest? 7052
Scientocracy. 3738
What do you do when you've lost your mojo? 5531
The End of Discovery? 3995
Double slit experiment - you will enjoy this 2011
"To speculate on the nature of reality... 12338
If there is no God(s)... 2564
I am 'who' i am! 2918
A typical atheist day 993
Any good videos/quotes about attachment? 1930
Mind Altering Science – An OPEN Conference on Psychedelic Research 4619
Lieing to become happy? 1657
an ambiguous maxim of sorts 1504
How do you prepare for death....? 1962
Homophobia linked to homosexual arousal 2059
What is energy? 6365
Grey matter helps you look inwards. 2475
Toddler crawls across freeway 2328
1900+ page Suicide Note 4750
Yes God? I don't think so. 33056
Some Social problems are rooted in population dynamics. 3220
The Euthyphro Dilemma 5143
Anomalies. 1208
Would you want to see it coming? 3569
The god argument - worthwhile? 34347
What we need. 3049
The trick to Life 4645
Linking people good memories and ascociating it with you.. 1145
Photo of outdoor bondage festival: what do you make of this? 9133
Stza Crack thee great philospher 2586
Human Nature Vs Social Conditioning 10232
the powers of human credulity *DELETED* 971
Here's the list of first digits:
7
4
8
3
8
1
1
1
2
2
2
1
3
8
1
1
2
4
7
1
1
4
7
3
5
3
2
1
2
2
9
1
4
1
1
1
2
6
2
2
4
3
3
5
1
3
3
3
4
1
9
2
1
9
And here's the distribution:
First Times It Percentage
Digit Appears
1 16 30%
2 11 20%
3 9 17%
4 6 11%
5 2 4%
6 1 2%
7 3 6%
8 3 6%
9 3 6%
It's a small sample set, only 54, so the curve is rough, but you can see it's clearly there. Add another couple hundred samples and it will fit the base 10 logarithm curve almost exactly.
I'm totally amazed by this. I didn't really believe it until I saw it for myself.
|
Icelander
The Minstrel in the Gallery
Registered: 03/15/05
Posts: 95,368
Loc: underbelly
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260102 - 09/28/10 04:14 PM (13 years, 6 months ago) |
|
|
It's proof of magic. Admit it sucker.
(I have to make comments like this because math is a second to the 10th power language for me.)
--------------------
"Don't believe everything you think". -Anom.
" All that lives was born to die"-Anom.
With much wisdom comes much sorrow,
The more knowledge, the more grief.
Ecclesiastes circa 350 BC
|
OrgoneConclusion
Blue Fish Group
Registered: 04/01/07
Posts: 45,441
Loc: Under the C
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260143 - 09/28/10 04:23 PM (13 years, 6 months ago) |
|
|
Still seems like the same counting issue I brought up earlier.
You don't have to go through 9000 letters to get to 1000 letters, but you have to pass through 1000 letters to get to 9000.
Road more travelled and all - unless I am missing something.
--------------------
|
I AM SWIM
doin' thangs
Registered: 12/24/08
Posts: 9,999
Loc: Feels Changsta Man
Last seen: 4 days, 5 hours
|
|
You still need to go through 9000 letters to get to /over9000
|
Revelation
ॐ
Registered: 08/04/01
Posts: 6,135
Loc: heart cave
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260505 - 09/28/10 05:35 PM (13 years, 6 months ago) |
|
|
Here is one I just did by pressing "Random Article" on Wikipedia, then Select All in firefox, pasting in word and doing a word count.
Article - Word Count
1. Benford's law - 2974
2. Second Five-Year Plan - 333
3. Stetsonville, Wisconsin - 736
4. Mohammad-Reza Shafiei Kadkani - 315
5. Dannike - 355
6. Atlasov Island - 765
7. Flores Tiger - 309
8. List of Asturian consorts - 551
9. William Booth (footballer) - 290
10. Mark W. Hemphill - 420
11. Fat Family - 305
12. Punchbowl Harvest - 457
14. Manitoba Schools Question - 1108
15. Renishaw Central railway station - 448
16. Kohlsville, Wisconsin - 320
17. Milk bottle - 1261
18. Escravos River - 303
19. WordArt - 745
20. Ortenaukreis - 837
21. Fritigern - 1047
22. Anne-Laure Viard - 440
23. Kenelm Lee Guinness - 1357
24. Green turtle - 5705
25. Jack Hibbert (footballer) - 399
26. Paradox (artist) - 892
27. Antón District - 343
28. RWD-9 - 1175
29. Gmina Stary Targ - 537
30. Barley yellow streak mosaic virus - 335
31. Pitcairn XO-61 - 440
32. Donji Žirovac - 237
33. Ryan Delahoussaye - 578
34. Mega Man X5 - 2046
35. Jellico, California - 575
36. List of newspapers in Uruguay - 266
37. Hortonia, Wisconsin - 794
38. 1996 Guam League - 315
39. Maniac Mansion (TV series) - 2027
40. Goran Sukno - 331
41. Phillip McCallen - 675
42. Amasa Walker - 673
43. All-Ireland Intermediate Club Hurling Championship - 353
44. Bihar County - 934
45. Hong Kong Trade Development Council Trade Fairs - 1764
46. Witter Bynner Poetry Prize - 314
47. Crew Gold - 355
48. Australian Maritime Safety Authority - 429
49. Gary A. Robbins - 659
50. Bermuda at the 1999 Pan American Games - 473
List of first digits
2
3
7
3
3
7
3
5
2
4
3
4
1
4
3
1
3
7
8
1
4
1
5
3
8
3
1
5
3
4
2
5
2
5
2
7
3
2
3
6
6
3
9
1
3
3
4
6
4
1 occurs 6 times
2 occurs 6 times
3 occurs 15 times
4 occurs 7 times
5 occurs 5 times
6 occurs 3 rimes
7 occurs 4 times
8 occurs 2 times
9 occurs 1 times
so, in order of most frequency
3, 4, 2 & 1, 5, 7, 6, 8 , 9
So it didn't work. Why the fail? A little after I started I thought I should have used the number of characters per page instead of number of words, as many seemed to be around the 300 word mark. So if the average wiki page is around 300 words this method would not yield a truly random set to work with. Correct?
I'm going to try again but use the Wordcount tool to find the number of characters in the txt instead. Do you think that would be random enough?
--------------------
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
|
You don't have to go through 9000 letters to get to 1000 letters, but you have to pass through 1000 letters to get to 9000.
Road more travelled and all
Yeah, but you still have to go through 10-90 and 100-900 and 1000-9000, so it's a wash.
Besides, if the road more traveled is right, the Benford distribution would show up in a set of pure random numbers too. It doesn't. It only shows up in random REAL-WORLD numbers. Pure random numbers distribute perfectly flat.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Revelation]
#13260556 - 09/28/10 05:43 PM (13 years, 6 months ago) |
|
|
So it didn't work. Why the fail? A little after I started I thought I should have used the number of characters per page instead of number of words, as many seemed to be around the 300 word mark. So if the average wiki page is around 300 words this method would not yield a truly random set to work with. Correct?
Yeah, I think it's a sampling problem. Like I said before, sampling things like the height of people who are almost all 4, 5, or 6 feet won't work.
Wiki article length are too uniform to work.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260652 - 09/28/10 06:01 PM (13 years, 6 months ago) |
|
|
Actually, I just started to do the same with random Wiki articles and the vast majority are 1,000 to 2,000 characters. Something with a wider dispersion would be a better test. I'm going to try random Google pages and see where it goes.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260736 - 09/28/10 06:16 PM (13 years, 6 months ago) |
|
|
Quote:
DieCommie said:
Great thread, and some great responses.
I can help but think that these so called random numbers are not completely random. I think the list of tallest building really illustrated that point.
Yep.
If they were random this wouldn't work. This only works for measurements or experimentally determined numbers et cet
Quote:
Diploid said:
Yea, but its still not completely random. Buildings cant be built to any random height.
Alright, that's a fair critique. But how about this:
I just analyzed the first 54 threads currently listed in this forum, excluding the stickies. I opened each thread, copied the entire text content into Microsoft Word and used its word count function to create the following table:
That stuff isn't random either. Its part of language, not random letters, and the rules of grammer require certain arrangements as well as other non-random influences.
Right? or am I missing something?
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260928 - 09/28/10 06:51 PM (13 years, 6 months ago) |
|
|
Here's another one:
Code:
State Population Area Sq
Kilometers
Alabama 4,661,900 131,426.36
Alaska 686,293 1,481,346.00
Arizona 6,500,180 294,313.30
Arkansas 2,855,390 134,856.00
California 36,756,666 403,931.96
Colorado 4,939,456 268,628.39
Connecticut 3,501,252 12,548.49
Delaware 873,092 5,060.84
Florida 18,328,340 139,760.29
Georgia 9,685,744 149,975.85
Hawaii 1,288,198 16,635.49
Idaho 1,523,816 214,313.75
Illinois 12,901,563 143,961.90
Indiana 6,376,792 92,895.10
Iowa 3,002,555 144,700.05
Kansas 2,802,134 211,899.88
Kentucky 4,269,245 102,895.05
Louisiana 4,410,796 112,825.06
Maine 1,316,456 79,932.21
Maryland 5,633,597 25,314.54
Massachusetts 6,497,967 20,305.51
Michigan 10,003,422 147,121.68
Minnesota 5,220,393 206,188.95
Mississippi 2,938,618 121,488.57
Missouri 5,911,605 178,413.92
Montana 967,440 376,977.95
Nebraska 1,783,432 199,097.57
Nevada 2,600,167 284,448.03
New Hampshire 1,315,809 23,227.01
New Jersey 8,682,661 19,209.94
New Mexico 1,984,356 314,310.60
New York 19,490,297 122,283.70
North Carolina 9,222,414 126,160.91
North Dakota 641,481 178,647.02
Ohio 11,485,910 106,054.83
Oklahoma 3,642,361 177,846.71
Oregon 3,790,060 268,631.09
Pennsylvania 12,448,279 116,075.50
Rhode Island 1,050,788 2,706.54
South Carolina 4,479,800 77,981.95
South Dakota 804,194 196,541.25
Tennessee 6,214,888 106,751.54
Texas 24,326,974 678,051.12
Utah 2,736,424 212,751.98
Vermont 621,270 23,957.39
Virginia 7,769,089 102,547.99
Washington 6,549,224 172,348.17
West Virginia 1,814,468 62,361.73
Wisconsin 5,627,967 140,662.25
Wyoming 532,668 251,487.85
Here are the distributions:
Population
1 14
2 6
3 5
4 5
5 5
6 8
7 1
8 3
9 3
Area in Square Kilometers
1 27
2 14
3 2
4 1
5 1
6 2
7 2
8 0
9 1
Notice again the characteristic logarithmic drop-off. It starts way up and drops quickly approaching 0 at the limit. Looks like atomic decay of a radioisotope, which is a logarithmic curve.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: johnm214]
#13260950 - 09/28/10 06:55 PM (13 years, 6 months ago) |
|
|
That stuff isn't random either.
Random numbers don't exhibit this property. Randomly selected real-world data does.
The Wiki link explains it well:
Benford's Law, also called the first-digit law, states that in lists of numbers from many (but not all) real-life sources of data, the leading digit is distributed in a specific, non-uniform way. According to this law, the first digit is 1 almost one third of the time, and larger digits occur as the leading digit with lower and lower frequency, to the point where 9 as a first digit occurs less than one time in twenty. This distribution of first digits arises whenever a set of values has logarithms that are distributed uniformly, as is approximately the case with many measurements of real-world values.
This counter-intuitive result has been found to apply to a wide variety of data sets, including electricity bills, street addresses, stock prices, population numbers, death rates, lengths of rivers, physical and mathematical constants, and processes described by power laws (which are very common in nature). The result holds regardless of the base in which the numbers are expressed (except for trivial bases), although the exact proportions change.
In 1972, Hal Varian suggested that the law could be used to detect possible fraud in lists of socio-economic data submitted in support of public planning decisions. Based on the plausible assumption that people who make up figures tend to distribute their digits fairly uniformly, a simple comparison of first-digit frequency distribution from the data with the expected distribution according to Benford's law ought to show up any anomalous results. Following this idea, Mark Nigrini showed that Benford's law could be used as an indicator of accounting and expenses fraud. In the United States, evidence based on Benford's law is legally admissible in criminal cases at the federal, state, and local levels.
In June 2010, consultants working for political website Daily Kos used Benford's law, among other tools, to find serious flaws in the data collected by polling company Research 2000 (R2K). This led to the termination of R2K's contract with Daily Kos, and possible litigation.
Benford's law can only be applied to data that is distributed across multiple orders of magnitude. For instance, one might expect that Benford's law would apply to a list of numbers representing the populations of UK villages beginning with 'A', or representing the values of small insurance claims. But if a "village" is a settlement with population between 300 and 999, or a "small insurance claim" is a claim between $50 and $100, then Benford's law will not apply.
This last paragraph goes to why random Wiki articles don't work. They're almost all 2,000 to 3,000 characters long.
|
laserpig
Weedmaster_P
Registered: 04/28/09
Posts: 7,468
Last seen: 11 years, 10 months
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13260967 - 09/28/10 06:59 PM (13 years, 6 months ago) |
|
|
woah i just understood how this distribution happens
fyi i never read the explanation in OP, i just wanted to interject whatever opinion i said (i already forgot what it was)
but anyway that is very cool shit
my layman's explanation:
if you are counting upwards, once any digit falls into the 9 position (visualizing one of those analog flip-clocks i guess), the next digit only needs to cycle once before that 9 is replaced
but if you have a digit in the
oh shit that explanation doesn't work
 i know exactly what i'm trying to say, but i don't know how to say it
i guess that's why the OP explanation was in math language
|
Revelation
ॐ
Registered: 08/04/01
Posts: 6,135
Loc: heart cave
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13265676 - 09/29/10 03:59 PM (13 years, 6 months ago) |
|
|
Size in kilobytes of 50 pdf books sorted by name
First Digit Time it occurs
1 13
2 5
3 11
4 7
5 3
6 3
7 3
8 4
9 0
Weird...
--------------------
|
Middleman
Registered: 07/11/99
Posts: 8,399
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13267053 - 09/29/10 09:03 PM (13 years, 6 months ago) |
|
|
Quote:
Random numbers don't exhibit this property. Randomly selected real-world data does.
So what does this mean to you Diploid?
Scale invariance is the key to Einstein's problem of infinite non-destructive compression and the true explanation of gravity. Cool thread.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
 Re: How Your Intuition Can Fool You [Re: Middleman]
#13267210 - 09/29/10 09:35 PM (13 years, 6 months ago) |
|
|
That the universe is a very weird place.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
 Re: How Your Intuition Can Fool You [Re: Diploid]
#13267313 - 09/29/10 10:00 PM (13 years, 6 months ago) |
|
|
Log in to view attachment
BTW if any math geek is interested, here are four great documents that go deep into the theory behind Benford's Law. They're not too dense but you need integral equations, set theory, and some statistics to grok them.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13267327 - 09/29/10 10:04 PM (13 years, 6 months ago) |
|
|
Log in to view attachment
You can only attache one file per post...
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13267332 - 09/29/10 10:04 PM (13 years, 6 months ago) |
|
|
Log in to view attachment
.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13267336 - 09/29/10 10:05 PM (13 years, 6 months ago) |
|
|
Log in to view attachment
..
|
johnm214
Registered: 05/31/07
Posts: 17,582
Loc: Americas
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13267846 - 09/30/10 12:05 AM (13 years, 6 months ago) |
|
|
Thanks for the info, read the article you linked above, but that was my point, in part.
I just misunderstood your use of the term random to refer to the selection rather than the population from which you choose which is what I thought you were saying 
|
jimbotron
Patty-Cake Enthusiast
Registered: 02/24/09
Posts: 2,324
Last seen: 10 years, 4 months
|
Re: How Your Intuition Can Fool You [Re: Diploid] 1
#13268872 - 09/30/10 08:27 AM (13 years, 6 months ago) |
|
|
Quote:
Diploid said:
but what does it all mean
Even though this was discovered 130 years ago, it is still a little mysterious why it happens. Mathematicians have formulated various constructions to get a handle on it, but no really good explanation has emerged.
I think it's pretty easy to understand intuitively.
The most famous part of it, of course, is that if you take a set of real-world numbers, like bank account balances, most of them will start with 1.
Why?
Well, say you've got $100. To get to $200 you have to double your money. That's a much bigger difference than going from $800 to $900. If your bank account is on the order of a few hundred dollars, it's probably between $100 and $200.
And then, of course, you make the jump to $1000. Lot of ground to cover on the way to $2000, unless you're already rich, in which case there's not that much of a difference between $8000 and $9000.
Etc, etc, etc. That's how I think of it, anyway. Heck, most billionaires are estimated to have between $1-2bn net worth, go figure. The rest of the logarithmic distribution is easier to explain with a graph than with words, and it occurs to me that we're really talking about probability densities here, which means that a little integral calculus can tell you everything you need to know.
Anyway, the prominence of 1 is my favorite part of this law, and it's pretty easy to grasp.
--------------------
BEST TEAM IN THE UNIVERSE
  
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: jimbotron]
#13268997 - 09/30/10 09:03 AM (13 years, 6 months ago) |
|
|
Well, say you've got $100. To get to $200 you have to double your money. That's a much bigger difference than going from $800 to $900. If your bank account is on the order of a few hundred dollars, it's probably between $100 and $200.
I don't think you're quite getting it.
Your explanation doesn't make any sense. You're assuming that most people have a low hundred dollar amount in the bank, which I don't think is true. Why wouldn't most people have $80 or $90 in the bank? Or $800 or $900. You just arbitrarily decided the typical bank balance out of nowhere.
Besides, if you do the analysis in Japanese Yen, or Mexican Pesos, or Chinese Yuan, the distribution is still present. In other words, if you convert a $100 US account to Mexican pesos (7 cents to one peso) it comes to 1,428 pesos, but the Benford distribution will still appear whatever the unit.
Also, your explanation doesn't explain why opening the newspaper and selecting randomly 100 numbers appearing in random stories that day also conform to Benford's Law.
Or why, as posted a couple posts back, the file size in bytes of a random sampling of PDF documents conform.
The population of all the US states conforms.
The surface area of all the US states (both in miles and kilometers) also conforms to the Benford distribution. I posted the exact count a few posts back. Take a look.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
DieCommie
Registered: 12/11/03
Posts: 29,258
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13269235 - 09/30/10 10:18 AM (13 years, 6 months ago) |
|
|
Its not the unit, its the base (base 10). This is what I, and others, alluded to earlier and I still think it is the reason. We purposefully choose the base such that most things we measure span an order of magnitude. It makes it easier to comprehend and discuss numbers. And if most measured things span an order of magnitude, then many of the measurements will start with a 1.
Convert those numbers to base 2 and, surprise surprise they will all start with 1 (if you ignore the leading zero). Convert those numbers to a base billion system, and they will all start with a unique character. In between those two extremes you can fine tune your base to provide whatever results you want.
|
Diploid
Cuban
Registered: 01/09/03
Posts: 19,274
Loc: Rabbit Hole
|
Re: How Your Intuition Can Fool You [Re: DieCommie]
#13269269 - 09/30/10 10:28 AM (13 years, 6 months ago) |
|
|
It works for all non-trivial bases. Of course in base 1 or 2 it wouldn't work. It works in hex and octal and all other bases besides 1 and 2.
From the Wiki:
More precisely, Benford's law states that the leading digit d (d ∈ {1, …, b − 1} ) in base b (b > 2) occurs with probability
This quantity is exactly the space between d and d + 1 in a logarithmic scale. (Note that the limitation b>2 excludes the trivial cases of binary (b=2), in which all numbers begin with 1, and unary systems (b=1) in which the number d is represented by d tally marks.)
In base 10, the leading digits have the following distribution by Benford's law, where d is the leading digit and p the probability:
Convert those numbers to a base billion system, and they will all start with a unique character.
That's not true if the sample size is commensurately large. Banfords holds for all non-trivial bases. Read the Wiki article and particularly the PDF documents I linked. They completely shoot down your explanation.
Besides, if what you say is true, then why desl Benford also hold for the SECOND digit, only less prominently? And why does the curve approach EXACTLY the logrithm10 curve instead of just generally tending toward favoring 1? With a sufficiently large sample size, it's exactly the log10 curve.
--------------------
Republican Values:
1) You can't get married to your spouse who is the same sex as you.
2) You can't have an abortion no matter how much you don't want a child.
3) You can't have a certain plant in your possession or you'll get locked up with a rapist and a murderer.
4) We need a smaller, less-intrusive government.
|
DieCommie
Registered: 12/11/03
Posts: 29,258
|
Re: How Your Intuition Can Fool You [Re: Diploid]
#13269294 - 09/30/10 10:36 AM (13 years, 6 months ago) |
|
|
.
Edited by DieCommie (11/16/16 10:15 AM)
|
|
Threaded
Previous
Index
Next
Edit 
Reply 
1 2 3 4 5 all )
Swami
Icelander
OrgoneConclusion
TameMe
trendal