Friday, April 14, 2006
Heresy! Tar and feather him! (Part 3)
To begin the construction of probabilistic model one must first start with a mechanical understanding of the underlying process being observed. In other words, what is an appropriate mathematical fontway to describe the process (or what is a most appropriate or probable way). Is it a binary choice or one with n choices? Are we looking at the number of occurences or the time between occurrences? What is a proper mathematical statement that captures the interaction of the individual agents under observation? Here's two examples to make things clearer.
Take a cable company. Every month they send a bill to the customer who then makes a decision to cancel the subscription or keep it (of course, I understand that the cancellation decision can be made before then, but the result is the same - that the customer is dropped in the next billing cycle). This is a periodic, binary decision. Round 1, yes or no? If no, stop. If yes, Round 2, yes or no? So on and so forth. The individual level interaction can then be described by this following mathematical construct. Customers have a probability r that they keep the subscription and probability (1-r) that they drop. Then the probability that a customer survives to n rounds should be (1-r) * (r)^(n-1).
Going out on a limb a bit now. Take a billboard company. They must calculate the amount of exposure one of their billboards get over the course of a week. They utilize driving surveys to collect the data. Drivers diligently report their weekly trips and the company measures the number and frequency of exposure. The underlying factors that is driving the drivers (bad word choice here) cannot be pinpointed. Therefore, we cautiously treat it as a random variable. Picking a RV, let's go with the Poisson (why Poisson? Why not gamma? That's a good question. Why not gamma? Why not something else? It just seemed like a good, simple thing to try first. If there is something lacking in the Poisson choice such as over- or under- dispersion of the actual data, we could try something else. Why not trying something else?). Then tentatively, we can venture to say that individual level billboard exposure can be characterized by (L^x * e^L)/x! for probability that the individual saw the billboard x times, where L is lambda which doubles for the rate and e is the exponential function.
Comprendez vous? Still more to come.
Take a cable company. Every month they send a bill to the customer who then makes a decision to cancel the subscription or keep it (of course, I understand that the cancellation decision can be made before then, but the result is the same - that the customer is dropped in the next billing cycle). This is a periodic, binary decision. Round 1, yes or no? If no, stop. If yes, Round 2, yes or no? So on and so forth. The individual level interaction can then be described by this following mathematical construct. Customers have a probability r that they keep the subscription and probability (1-r) that they drop. Then the probability that a customer survives to n rounds should be (1-r) * (r)^(n-1).
Going out on a limb a bit now. Take a billboard company. They must calculate the amount of exposure one of their billboards get over the course of a week. They utilize driving surveys to collect the data. Drivers diligently report their weekly trips and the company measures the number and frequency of exposure. The underlying factors that is driving the drivers (bad word choice here) cannot be pinpointed. Therefore, we cautiously treat it as a random variable. Picking a RV, let's go with the Poisson (why Poisson? Why not gamma? That's a good question. Why not gamma? Why not something else? It just seemed like a good, simple thing to try first. If there is something lacking in the Poisson choice such as over- or under- dispersion of the actual data, we could try something else. Why not trying something else?). Then tentatively, we can venture to say that individual level billboard exposure can be characterized by (L^x * e^L)/x! for probability that the individual saw the billboard x times, where L is lambda which doubles for the rate and e is the exponential function.
Comprendez vous? Still more to come.
Thursday, April 13, 2006
Interlude: Update on character warfare
On March 31, 2006, I wrote a post mentioning that the U.N. announced that it will cease using traditional characters in its publications. When the news was released, there was quite an uproar until one observant someone pointed out the fact that the U.N. has already de facto stopped using traditional characters when official recognition was switched from Taipei to Beijing. This got me thinking about why the U.N. would make mention of it now, decades later after the fact. My best guess is that this is another CCP ploy to, how to put this properly, rub Taiwan's nose in poop. They've been doing this for a long time now, so it no longer comes as a surprise. Until the news came out weeks ago, I wasn't even aware that the U.N. used traditional characters.
In English we say, "There's no use crying over spilt milk."
In English we say, "There's no use crying over spilt milk."
Wednesday, April 12, 2006
Heresy! Tar and feather him! (Part 2)
Regression is limited in three ways. First, there is the equal variance assumption. Of course, this assumption can be lifted with more advanced methods, but it does not change the fact that this assumption is a built-in artifact of the design of the regression method. Well, what's wrong with equal variances? My question is: Is it safe to make such a limiting assumption? Is the world we live in truly a homoscedastic world? This assumption and these questions are analogous to the Homo Economicus problem in economics. Are people rational, forward looking, utility maximizers?
The only reason regression starts out with the homoscedasticity assumption is because it must. Otherwise, why would anyone make such a nonsensical, limiting assumption and spend time and energy trying to find ways around it when the data doesn't fit this requirement?
Real people vary in their behaviors and they vary at varying rates, most of the time for unobservable reasons. Probabilistic models work by not imposing unreasonable demands on the data in the first place. Rather they let the data tell them how the population in question differs. Thus, the heteroscedasticity inherent in many observable phenomenon is not a nuisance that has to be dealt with, but a fact to be embraced. In fact, non-equal variance is something that makes the data all the more richer in the results it produces and the information that can be garnered from them.
The only reason regression starts out with the homoscedasticity assumption is because it must. Otherwise, why would anyone make such a nonsensical, limiting assumption and spend time and energy trying to find ways around it when the data doesn't fit this requirement?
Real people vary in their behaviors and they vary at varying rates, most of the time for unobservable reasons. Probabilistic models work by not imposing unreasonable demands on the data in the first place. Rather they let the data tell them how the population in question differs. Thus, the heteroscedasticity inherent in many observable phenomenon is not a nuisance that has to be dealt with, but a fact to be embraced. In fact, non-equal variance is something that makes the data all the more richer in the results it produces and the information that can be garnered from them.
Tuesday, April 11, 2006
Heresy! Tar and feather him!
Someone said the 'D' word in applied probability models today. Yes, he mentioned 'demographics.' Hogwash! Heresy! Tar and feather this blasphemer! Actually, let's step back and examine this more carefully.
The idea he asked was pretty simple. He wanted to build 'ideal customer' profiles with mathematical modeling. I believe what he had in mind was using some mixture of statistical methods like regression analysis, data mining, or cluster analysis to identify significant 'traits' that can be used to predict some sort of desired behavior. In this case, buying behavior of cable services from a national service provider that wishes to remain undisclosed. After giving him the staring treatment, we then tried hard not to burst out laughing. Thoughts of laughter was soon followed by thoughts of vengeful violence. Before I forget, I must mention in passing that this particular person was sitting in and not actually enrolled in the course.
What's wrong with all this? Is this class just filled with liberals who would argue that putting people in buckets and bins is morally reprehensible? Far from it. We have another reason for our...Intolerant behavior. For the uninitiated, it may be difficult to explain the mechanics, so instead, let me give a few examples over the next few days to illustrate.
Consider the statement: people are inherently different. This is the first law of probabilistic modeling. In those days, we believed this to be the one and only truth (strike that! I admit I've been watching FMA recently). Back to the subject. People are just different in their behaviors no matter how you try to cut across demographics. Even if there is some gain from throwing people into different bins, that difference pales in comparison against the intra-bin heterogeneity that exists. If you want to look at income with respect to race, amongst Caucasians you have people like Bill Gates and Warren Buffet and families barely subsisting in West Virginian Appalachia. Amongst African-Americans you have the Oprah's and Denzel's and people earning 19K a year living in West Philadelphia.
When it is appropriate to apply probability models, a probabilitist who is initiated in the first law understands that information is not leveraged from grouping observations. Rather it is in leveraging the heterogeneity amongst the population that the most information can be gained. If we can simulate the underlying mechanics of the decision process and account for the fact that people are different in their behaviors, we can build models that forecast into the future with great predictive accuracy.
The idea he asked was pretty simple. He wanted to build 'ideal customer' profiles with mathematical modeling. I believe what he had in mind was using some mixture of statistical methods like regression analysis, data mining, or cluster analysis to identify significant 'traits' that can be used to predict some sort of desired behavior. In this case, buying behavior of cable services from a national service provider that wishes to remain undisclosed. After giving him the staring treatment, we then tried hard not to burst out laughing. Thoughts of laughter was soon followed by thoughts of vengeful violence. Before I forget, I must mention in passing that this particular person was sitting in and not actually enrolled in the course.
What's wrong with all this? Is this class just filled with liberals who would argue that putting people in buckets and bins is morally reprehensible? Far from it. We have another reason for our...Intolerant behavior. For the uninitiated, it may be difficult to explain the mechanics, so instead, let me give a few examples over the next few days to illustrate.
Consider the statement: people are inherently different. This is the first law of probabilistic modeling. In those days, we believed this to be the one and only truth (strike that! I admit I've been watching FMA recently). Back to the subject. People are just different in their behaviors no matter how you try to cut across demographics. Even if there is some gain from throwing people into different bins, that difference pales in comparison against the intra-bin heterogeneity that exists. If you want to look at income with respect to race, amongst Caucasians you have people like Bill Gates and Warren Buffet and families barely subsisting in West Virginian Appalachia. Amongst African-Americans you have the Oprah's and Denzel's and people earning 19K a year living in West Philadelphia.
When it is appropriate to apply probability models, a probabilitist who is initiated in the first law understands that information is not leveraged from grouping observations. Rather it is in leveraging the heterogeneity amongst the population that the most information can be gained. If we can simulate the underlying mechanics of the decision process and account for the fact that people are different in their behaviors, we can build models that forecast into the future with great predictive accuracy.
Wednesday, April 05, 2006
On devilish distributions and deviant divination Part 4
This series has been interupted a number of times. Here are the Parts One, Two, and Three.
Oddly enough, where Ehrenberg indoctrinates his followers down under, he doesn't teach them the underlying mechanics that gives rise to the model. Therefore you have a bunch of people who only know how to 'apply' the model (blindly). Unfortunately, because their knowledge is limited, they don't do the model justice. Who can blame them? They are simply told, "Oh, you know need to know the underlying mathematical constructs. They are too complex." Yeah, right.
Usually, this is the time when people's eyes glaze over, but believe me when I say that the coolness of this model is on par with the Black-Scholes option pricing equation for European options. I mean, who knew intuitively that the value of European style option prices can be modeled as the movement of heat through metal pipes?
The logic behind every piece of the model is actually quite simple. So let me walk through it. First, let all category purchases be described by a Poisson process with a rate parameter lambda. That is to say, the arrival of purchases come at a time-invariant rate, proportional to the unit of time. However, this assumption is WAY too limiting since it's ridiculous to believe that people are homogenous and therefore all possess the same rate parameter lambda. It's closer to the truth to say that people vary in their purchasing rates, therefore, let's define lambda not as a fixed value, but a random variable. Let this random variable have a Gamma distribution defined by shape parameter r and scale paramet alpha. If you integrate the product of the two, the result is a negative binomial distribution defined by parameters r and alpha.
That's the discription of the arrival process of purchases over the category, but it does not sort them into individual brands. The brand-choice decision turns out to follow a Dirichlet distribution also known as a multinomial distribution. It's an extended version of the binomial expect instead of two options, you can have up to n choices. Purchases are sorted into each brand with probability P sub i, where i indexes each particular brand. The summation of all the P's must equal 1. Once again, it would be naive to assume that the Dirichlet multinomial parameters would be homogenous throughout the population, so let's allow for the probability parameters to be random. In particular, the let the distribution of brand purchasing probabilities for each brand be a multivariate Beta distribution.
Integrating the entire mess (it's actually not as hard as it looks), you get something that looks like this where n is the total number of category purchases, x is the number of brand purchases, k is the number of brands in the category, and alpha is the parameter giving rise to the shape of the Dirichlet multinomial with k vectors. Okay, that was WAAAY too much math to know.
Oddly enough, where Ehrenberg indoctrinates his followers down under, he doesn't teach them the underlying mechanics that gives rise to the model. Therefore you have a bunch of people who only know how to 'apply' the model (blindly). Unfortunately, because their knowledge is limited, they don't do the model justice. Who can blame them? They are simply told, "Oh, you know need to know the underlying mathematical constructs. They are too complex." Yeah, right.
Usually, this is the time when people's eyes glaze over, but believe me when I say that the coolness of this model is on par with the Black-Scholes option pricing equation for European options. I mean, who knew intuitively that the value of European style option prices can be modeled as the movement of heat through metal pipes?
The logic behind every piece of the model is actually quite simple. So let me walk through it. First, let all category purchases be described by a Poisson process with a rate parameter lambda. That is to say, the arrival of purchases come at a time-invariant rate, proportional to the unit of time. However, this assumption is WAY too limiting since it's ridiculous to believe that people are homogenous and therefore all possess the same rate parameter lambda. It's closer to the truth to say that people vary in their purchasing rates, therefore, let's define lambda not as a fixed value, but a random variable. Let this random variable have a Gamma distribution defined by shape parameter r and scale paramet alpha. If you integrate the product of the two, the result is a negative binomial distribution defined by parameters r and alpha.
That's the discription of the arrival process of purchases over the category, but it does not sort them into individual brands. The brand-choice decision turns out to follow a Dirichlet distribution also known as a multinomial distribution. It's an extended version of the binomial expect instead of two options, you can have up to n choices. Purchases are sorted into each brand with probability P sub i, where i indexes each particular brand. The summation of all the P's must equal 1. Once again, it would be naive to assume that the Dirichlet multinomial parameters would be homogenous throughout the population, so let's allow for the probability parameters to be random. In particular, the let the distribution of brand purchasing probabilities for each brand be a multivariate Beta distribution.
Integrating the entire mess (it's actually not as hard as it looks), you get something that looks like this where n is the total number of category purchases, x is the number of brand purchases, k is the number of brands in the category, and alpha is the parameter giving rise to the shape of the Dirichlet multinomial with k vectors. Okay, that was WAAAY too much math to know.
Tuesday, April 04, 2006
Interlude: sacred mission inviolate?
Recently, I re-read Frank Herbert's God Emperor of Dune. Back in high school, I've heard a librarian mention to a computer technician in passing that it was probably the most pointless five hundred pages of text he's every read. Unfortunately for him, I think he missed the point about the book.
Leto at the end of the previous volume in the Dune series merges with the native sandtrout of Arrakis and in doing so gains a lifespan of that stretches across millennia. He consolidates his power over the intergalactic empire that his father left to him and rules it with an iron fist. He becomes known as the God-Emperor for two reasons: his longevity and his prescient ability. Leto was born, like his father, with the ability to see completely into the future and all the way back into the beginning of time. It's an amazing ability that allows his to know everything before it happens and know how to change things to prevent undesirable things from happening. Thus, Leto rules the galaxy as a tyrant unlike any the human race has ever seen. The situation is compounded by the galaxy's dependence on the 'spice', a geriatric drug that can only be produced on Arrakis, the capitol city of the empire. Leto rules not only be his awesome power, but also by a perverted form of hydraulic despotism. By controlling the distribution of spice, he can control all the major political power groups in his empire.
If the story was only about that, then the God-Emperor of Dune would just be another power monger's fantasy. The crux of understanding Leto throughout the book is understanding what he meant when he mumbles about the 'Golden Path'. It's awfully confusing at first, but once the reader is initiated into the underlying meaning of the phrase, everything clears up. Leto as the ruler over all humans is heavily burdened his stewardship. When he looks into future with his powers and in almost all its possible outcomes, he sees the ultimate destruction of the human race, either by their own hands or at the hands of thinking machines. He cannot bear to allow something like that to happen, thus he sacrifices his humanity to lengthen his life and uses every resource at his disposal to steer humanity onto the only path where survival is possible. After three millennia, countless purges, and other innumerable atrocities, his resolution to save the human race from itself remains unshakeable.
This is just a small slice of the many ideas Frank Herbert explores in God-Emperor of Dune, but it gets me to think about who is it that knows what is best for society? How is Leto's argument any different from that of modern day dictators and other authoritarian regimes? Could they actually be justified if they do no sway from doing what is right for society? Maybe this is how Singapore's administration can justify its use of power. I wonder if the CCP thinks of themselves in the same way. After all, who can refuse to be lorded over by a power that claims it is there to save you from yourself?
Leto at the end of the previous volume in the Dune series merges with the native sandtrout of Arrakis and in doing so gains a lifespan of that stretches across millennia. He consolidates his power over the intergalactic empire that his father left to him and rules it with an iron fist. He becomes known as the God-Emperor for two reasons: his longevity and his prescient ability. Leto was born, like his father, with the ability to see completely into the future and all the way back into the beginning of time. It's an amazing ability that allows his to know everything before it happens and know how to change things to prevent undesirable things from happening. Thus, Leto rules the galaxy as a tyrant unlike any the human race has ever seen. The situation is compounded by the galaxy's dependence on the 'spice', a geriatric drug that can only be produced on Arrakis, the capitol city of the empire. Leto rules not only be his awesome power, but also by a perverted form of hydraulic despotism. By controlling the distribution of spice, he can control all the major political power groups in his empire.
If the story was only about that, then the God-Emperor of Dune would just be another power monger's fantasy. The crux of understanding Leto throughout the book is understanding what he meant when he mumbles about the 'Golden Path'. It's awfully confusing at first, but once the reader is initiated into the underlying meaning of the phrase, everything clears up. Leto as the ruler over all humans is heavily burdened his stewardship. When he looks into future with his powers and in almost all its possible outcomes, he sees the ultimate destruction of the human race, either by their own hands or at the hands of thinking machines. He cannot bear to allow something like that to happen, thus he sacrifices his humanity to lengthen his life and uses every resource at his disposal to steer humanity onto the only path where survival is possible. After three millennia, countless purges, and other innumerable atrocities, his resolution to save the human race from itself remains unshakeable.
This is just a small slice of the many ideas Frank Herbert explores in God-Emperor of Dune, but it gets me to think about who is it that knows what is best for society? How is Leto's argument any different from that of modern day dictators and other authoritarian regimes? Could they actually be justified if they do no sway from doing what is right for society? Maybe this is how Singapore's administration can justify its use of power. I wonder if the CCP thinks of themselves in the same way. After all, who can refuse to be lorded over by a power that claims it is there to save you from yourself?
Monday, April 03, 2006
Interlude: constants and fixed costs
Last week, someone presented me with two ideas that I found highly interesting so I document them here.
First, the product of income and free time is a constant. I pondered that statement for a few moments. True, but with two caveats. This constant only holds in a short-run case. I can imagine that my 'constant' would be dramatically different if I did not finish college than if I did. This constant would also probably shift as one gets promoted: MD's work fewer hours than analysts and can actually take their families on African safaris. If the time period in consideration takes into account the time before and after some event that exogenously alter income or time, the constant would not hold.
The second caveat is that people who possess high wealth or have received significant windfalls are not bound by this dismal equation. Think: lottery winners and heirs to fortunes. They have no need of possessing income if they wealth they have can provide for all their needs. Speaking of needs, this is a nice segway into the second idea.
Fixed costs rise as income rises. As 'wants' become 'needs' they move from being a variable cost or life to a fixed cost of life. This usually happens as income rises to support a higher standard of living. "I just have to have that 64' plasma-screen deluxe home entertainment center." This is also related Marx's critique of capitalism - that the realm of 'necessity' expands making the realm of freedom harder to realize. In a sense, if you are still 'needing' something, you cannot truely be happy. Only when wealth and material possession has become immaterial to you do you begin to search for the source of genuine self-fulfillment. In that context, your work can be liberating. You enter the state of Eudaimonia - happiness and complete fulfillment of the human potential.
First, the product of income and free time is a constant. I pondered that statement for a few moments. True, but with two caveats. This constant only holds in a short-run case. I can imagine that my 'constant' would be dramatically different if I did not finish college than if I did. This constant would also probably shift as one gets promoted: MD's work fewer hours than analysts and can actually take their families on African safaris. If the time period in consideration takes into account the time before and after some event that exogenously alter income or time, the constant would not hold.
The second caveat is that people who possess high wealth or have received significant windfalls are not bound by this dismal equation. Think: lottery winners and heirs to fortunes. They have no need of possessing income if they wealth they have can provide for all their needs. Speaking of needs, this is a nice segway into the second idea.
Fixed costs rise as income rises. As 'wants' become 'needs' they move from being a variable cost or life to a fixed cost of life. This usually happens as income rises to support a higher standard of living. "I just have to have that 64' plasma-screen deluxe home entertainment center." This is also related Marx's critique of capitalism - that the realm of 'necessity' expands making the realm of freedom harder to realize. In a sense, if you are still 'needing' something, you cannot truely be happy. Only when wealth and material possession has become immaterial to you do you begin to search for the source of genuine self-fulfillment. In that context, your work can be liberating. You enter the state of Eudaimonia - happiness and complete fulfillment of the human potential.
0 Comments:
Post a Comment