A: “When I was a kid, I really enjoyed looking at graphs, plots and maps. My parents and I could not make of what was behind the interest. At the same time, I was also heavily interested in education. My mother was a teacher and the first set of statistics I ever encountered were standardized test scores. I strived to understand what the scores attempted to say about me, and why such scores and tests are so trustworthy. When the stakes increased with the AP and SAT exams, I began reading articles published by the Educational Testing Service and learned a ton about how these tests are constructed to minimize bias, and how scores are comparable across forms. It fascinated me how much science goes into these tests, but in the end of the day they are still just one factor [...]]]>

Recently, I participated in an email interview about what being a Statistics major entailed, how I got interested in the field and the future of Statistics. I figured this might be of interest to those that are contemplating majoring in Statistics, or considering a career in Data Science.

A: “When I was a kid, I really enjoyed looking at graphs, plots and maps. My parents and I could not make of what was behind the interest. At the same time, I was also heavily interested in education. My mother was a teacher and the first set of statistics I ever encountered were standardized test scores. I strived to understand what the scores attempted to say about me, and why such scores and tests are so trustworthy. When the stakes increased with the AP and SAT exams, I began reading articles published by the Educational Testing Service and learned a ton about how these tests are constructed to minimize bias, and how scores are comparable across forms. It fascinated me how much science goes into these tests, but in the end of the day they are still just one factor in the whole picture of a student. This niche interest lead me to statistics, psychometrics in particular, and although I no longer study psychometrics, I found what I learned to be incredibly valuable.”

Q2: “I noticed you have bachelor’s, master’s, and doctoral degrees in statistics. How did your graduate study build on what you learned in your undergraduate program?”

A: “For me, the undergraduate and graduate programs were night and day. The undergraduate program focused more on modeling and data analysis. The graduate program focused more on thinking about data and how to develop a scientific “common sense” about how to work with, express and make automated decisions based on data. The graduate program was much more mathematically and computationally intensive than the undergraduate major. My graduate study actually built more on my mathematics major in college because many of the concepts in graduate statistics require knowledge of linear algebra, numerical analysis and real analysis. Fortunately, our statistics major requires upper division math courses.”

Q3: What was the most interesting part of majoring in statistics? What did you find most challenging?

A: “The most interesting part of majoring in statistics was seeing how many fields can grow and transform based on insights from data and statistics. In my case, I found it most interesting seeing how it integrates and interacts with computer science. Every time someone surfs through Facebook, enters a Google search, or looks at an item on Amazon, data about what you are doing are collected and algorithms process this data to enrich the experience by, say,recommending books and offering special deals on Amazon, recommending friends and showing relevant stories on Facebook, and the most groundbreaking of all: returning relevant search results.

The most challenging part for me was the mathematical theory. Although I loved math, I sometimes had trouble connecting the theory to the application, and statistics is such an applied field. I look at it as a rite of passage and once I saw enough theory relevant to my interests, the learning process became easier.”

Q4: How do you apply what you learned in your statistics education in your current line of work?

A: “It is ironic, but it is the more basic concepts of statistics and probability that I use everyday rather than the complicated models I learned. Concepts such as independence, confidence, power, accuracy etc. are important building blocks for building my own models, or for choosing an existing one from those that I learned in school.

I always start with some exploratory analysis such as computing some statistics and making plots that show relationships clearly. Then I set explicit guidelines for the input and output of the model I want to build and note any critical assumptions that are violated or that must be met. I then try several different methods and models and validate their results using common metrics taught in undergraduate statistics before settling on a final model configuration.”

Q5: What skills did you learn in the statistics major that you find useful for work and everyday life?

A: “The training in mathematics I received as part of the statistics major taught me how to think logically, and this is very important in my work in computer science. I think patience was another very important skill I learned. I love what I do, and sometimes I take for granted that others have the same mathematical training that I do because I am so entrenched in it. Through my experience teaching as well as consulting as a student, I gained a better sense of the challenges and difficulties many people face when thinking about and interpreting statistics and how to better communicate results and ideas.”

Q6: Any advice for students who are considering majoring in statistics?

A: “My advice for students majoring in statistics is to choose an additional major or minor that uses statistics and is of interest to the student. I do not consider statistics to be a “standalone” major. When interviewing for a job, employers want to know why an interviewee is passionate about their company. For example, if interviewing for a finance company, the company wants to hear about passion for finance, and see education or experience in such fields. Another way to accomplish this instead of double majoring is to do some internships, projects or research in a field of interest.”

Conclusion: Finally, could you tell me a little about yourself for an intro bio we will include before the Q&A interview? For instance, what university(ies) did you attend, what degree(s) have you earned, what is your current job title, where do you work and for how long (you can be general here, or include a link to your professional website or blog if you have one), and what are you career goals?

A: “I attended University of California, Los Angeles (UCLA) for my B.S. (Statistics, Mathematics of Computation), two M.S. (Statistics and Computer Science) and Ph.D. (Statistics). I currently work for an Internet advertising startup in Santa Monica, CA as Chief Data Scientist/Research Engineer, and have been working in the field for three years. Whenever I get a free moment, I write about statistics, data mining and computer science topics on my blog at http://www.bytemining.com. I plan on dedicating the rest of my life working with and communicating about data and turning online phenomena into knowledge that can be used to progress technology and change the world!”

Look, if I am at a casino, I am well aware that the odds are against me, so why even try to think that I can use statistics to make money in this way? Although I love numbers and math, the stuff flows through my brain all day long (and night long), every day. If the goal is to enjoy and have fun, I do not want to sit there crunching probability formulas in my head (yes that’s fun, but it is also work). So that leaves me at the video Poker machines enjoying the free drinks. Another positive about video Poker is that $20 can sometimes last a few hours. So it should be no surprise that I do not agree with using Poker to teach probability.  Poker is an extremely superficial way to introduce such a powerful tool and gives the impression that probability is a way to make a quick buck, rather than as an important tool in science and society. The only [...]]]>

Whenever I tell people in my family that I study Statistics, one of the first questions I get from laypeople is “do you count cards?” A blank look comes over their face when I say “no.”

Look, if I am at a casino, I am well aware that the odds are against me, so why even try to think that I can use statistics to make money in this way? Although I love numbers and math, the stuff flows through my brain all day long (and night long), every day. If the goal is to enjoy and have fun, I do not want to sit there crunching probability formulas in my head (yes that’s fun, but it is also work). So that leaves me at the video Poker machines enjoying the free drinks. Another positive about video Poker is that $20 can sometimes last a few hours. So it should be no surprise that I do not agree with using Poker to teach probability.  Poker is an extremely superficial way to introduce such a powerful tool and gives the impression that probability is a way to make a quick buck, rather than as an important tool in science and society. The only time that I have used Poker in teaching (besides when required), is to cover the hypergeometric distribution and sampling without replacement.

Since I took Intro Probability Theory, I have always wondered what to do in the following situation. Say a pair of cruddy low cards appear on the first draw. The game only awards money for pairs of jacks or better. If all I have in the hand is a pair of low cards and no face cards, my decision is easy: hold the pair of low cards. But what if there is at least one face card showing (no other pairs)? Pictorially this looks like

The conundrum:

1. Hold the two low cards and deal, hoping for a three of a kind, or
2. Hold the two low cards AND one of the face cards, hoping for a three of a kind, OR a pair of Jacks of Better.

Under each of these decisions, which yields the highest probability of winning something and which one yields the highest payout? This problem can be solved exactly by using combinatorics, conditional probability and expectation, but since a video poker game is basically a simulator (though likely biased), I wrote my own simulation. For the answer, scroll to the end!

Data Structure

In most card games, we would want to store the state of the game: the outstanding cards in the deck(s), and the hand(s) of each player. In standard video poker, there is one deck, and one player, so only the player hand needs to be recorded because every card in the deck is either in the hand, or it is not. One obvious way to represent the hand is as an array of denomination/suit tuples in an array. Unfortunately, this data structure requires other data structures to store the possible suits, and possible denominations. It is also more tedious to detect certain kinds of wins. For this simulation, I use a 13 x 4 matrix where each row is a different denomination, and each column is each of the four suits. This matrix allows us to easily see which cards are possible to be dealt. Additionally, this matrix, as well as vector-based languages such as R, make it easy to detect wins. Such a matrix looks like the following for the hand 2♠ 5♣ 8♥ 8♣ A♦

         

where Cij denotes a card, i is the denomination and j is the suit and H is the player’s hand in question.

Poker wins are not disjoint. A three of a kind involving Jacks is also a pair of Jacks or better, etc. When checking wins, I start with the lowest paying win, and move up to Royal Flush, only keeping track of the highest win. Thus, this algorithm detects a four-of-a-kind involving Queens as Jacks or Better, two pairs of Queens, and a three-of-a-kind of Queens, but only counts it as the highest win, the four-of-a-kind.

1. Pair of Jacks or Better: a pair of Jacks, Queens, Kings or Aces. In A, this is simply the condition that at least one row in rows 10 through 13 has a row sum greater than 1.
2. Two pair: two pairs of anything. In A, this is the condition that at least two rows have a sum greater than 1.
3. Three of a kind: three of any card. In A, this is the condition that at least one row has a sum of at least 3.
4. Straight: all 5 cards can be permuted such that they form an ascending sequence: A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A. This case is interesting and will be discussed in a bit.
5. Flush: all 5 cards are of the same suit. In A, this is the condition that at least one column has a sum of at least 5.
6. Full House: one three-of-a-kind, and a pair of anything. In A, this is the condition that a row has sum 3, and another row has sum 2.
7. Four of a Kind: 4 of any card. In A, this is the condition that a row has sum 4.
8. Straight Flush: the 5 cards can be permuted to form an ascending sequence and are all of the same suit. In A, this is simply the condition that we have a straight and a flush in the same hand.
9. Royal Flush: a straight flush with the Ace as the high card. In A, this is simply the condition that we have a straight flush AND the sum of row 13 is 1.

Bug alert: It just occurred to me that there are many more wrap-around straights such as Q, K, A, 2, 3. This will be fixed this evening.

From basic computer science and number theory, every natural number can be written as the sum of distinct powers or 2 and the representation of such an integer is unique. Furthermore, the sum of n successive powers of 2 is divisible by . After some experimentation I came up with the following rule: if all of the row sums are 0/1 and the integer representation of this binary vector is divisible by , then A is a straight. The only straight that does not fit this pattern is the wrap-around straight: J, Q, K, A, 2 which can be checked manually.

1. Randomly generate a hand containing a pair of low cards (2-10) and at least one face card.
2. Hold the pair of low cards. Under strategy 2, hold one (and only one) of the face cards.
3. Discard the unheld cards from the deck and draw 2 or 3 cards at random from the same deck.
4. Check for wins.
5. Increment a win counter.
6. Repeat steps 1-5 tons of times, recording the percentage of hands that yielded a win, of the n games/hands played.

Results: Hold the Pair of Low Cards Only

My usual strategy is to always hold the low pair and take one face card along for the ride. That way, I hopefully match one of the two denominations I hold. My parents on the other hand, always told me to hold the low pair only, because that gives one more card (degree of freedom) for a win. It turns out they were right. Each game consisted of 1,000 hands. A percentage of these hands yields a win. This percentage is a random variable, so I ran this simulation to play 1,000 games. The table below shows the distribution of the win percentages.

Note that under strategy 1 (hold low pair only), all wins are more likely than under strategy 2! Of course, the estimate in the last column is an average; the mean in this case. The plot below shows the distribution of win percentages for both strategies. 

The Code

The code for my simulation is below. Note that it can easily be modified for your own target hands of interest. In my simulation, certain functions were never used because certain winning hands were not possible. 

DISCLAIMER: I did this for fun, and it is possible that there are bugs or problems with my code, algorithm or simulation. The results seem correct because I empirically I seem to do about the same using either strategy, and in a gambling perspective, an 8% discrepancy is not likely to set off bells in the head.

The above resources are available not only for Wikipedia, but for other Wikimedia Foundation projects such as Wiktionary, Wikibooks and Wikiquotes.

As Wikipedia readers will notice, the articles are very well formatted and this formatting is generated by a somewhat unusual markup format defined by the MediaWiki project. As Dirk Riehl stated:

There was no grammar, no defined processing rules, and no defined output like a DOM tree based on a well defined document object model. This is to say, the content of Wikipedia is stored in a format that is not an open standard. The format is defined by 5000 lines of php code (the parse function of MediaWiki). That code may be open source, but it is incomprehensible to most. That’s why [...]]]>

Lately I have doing a lot of work with the Wikipedia XML dump as a corpus. Wikipedia provides a wealth information to researchers in easy to access formats including XML, SQL and HTML dumps for all language properties. Some of the data freely available from the Wikimedia Foundation include article content and template pages article content with revision history (huge files) article content including user pages and talk pages redirect graph page-to-page link lists: redirects, categories, image links, page links, interwiki etc. image metadata site statistics

The above resources are available not only for Wikipedia, but for other Wikimedia Foundation projects such as Wiktionary, Wikibooks and Wikiquotes.

As Wikipedia readers will notice, the articles are very well formatted and this formatting is generated by a somewhat unusual markup format defined by the MediaWiki project. As Dirk Riehl stated:

There was no grammar, no defined processing rules, and no defined output like a DOM tree based on a well defined document object model. This is to say, the content of Wikipedia is stored in a format that is not an open standard. The format is defined by 5000 lines of php code (the parse function of MediaWiki). That code may be open source, but it is incomprehensible to most. That’s why there are 30+ failed attempts at writing alternative parsers.

For example, below is an excert of Wiki-syntax for a page on data mining.

'''Data mining''' (the analysis step of the '''knowledge discovery in databases''' process,<ref name="Fayyad"> or KDD),
a relatively young and interdisciplinary field of [[computer science]]<ref name="acm" />
{{cite web|url=http://www.sigkdd.org/curriculum.php |title=Data Mining Curriculum |
publisher=[[Association for Computing Machinery|ACM]] [[SIGKDD]] |date=2006-04-30 |accessdate=2011-10-28}}
</ref><ref name=brittanica>{{cite web | last = Clifton | first = Christopher | title = Encyclopedia Britannica: Definition
of Data Mining | year = 2010 | url = http://www.britannica.com/EBchecked/topic/1056150/data-mining |
accessdate = 2010-12-09}}</ref> is the process of discovering new patterns from large [[data set]]s
involving methods at the intersection of [[artificial intelligence]], [[machine learning]], [[statistics]] and
[[database system]]s.<ref name="acm"> The goal of data mining is to extract knowledge from a data set in a
human-understandable structure<ref name="acm" /> and involves database and [[data management]],
[[Data Pre-processing|data preprocessing]], [[statistical model|model]] and [[Statistical inference|inference]]
considerations, interestingness metrics, [[Computational complexity theory|complexity]] considerations, post-processing
of found structure, [[Data visualization|visualization]] and [[Online algorithm|online updating]].<ref name="acm" />

I was epicly worried that I would spend weeks writing my own parser and never complete the project I am working on at work. To my surprise, I found a fairly good parser. Since I am working on named entity extraction and ngram extraction, I wanted to only extract the plain text. If we take the above junk and extract only the plain text, we would get 

Data mining (the analysis step of the knowledge discovery in databases process, or KDD), a relatively young
and interdisciplinary field of computer science is the process of discovering new patterns from large data sets
involving methods at the intersection of artificial intelligence, machine learning, statistics and database systems.
The goal of data mining is to extract knowledge from a data set in a human-understandable structure and involves
database and data management, data preprocessing, model and inference considerations, interestingness
metrics, complexity considerations, post-processing of found structure, visualization and online updating.

and from this we can remove punctuation (except sentence terminators .?!), convert to lower case and perform other pre-processing text mining steps. There are many, many Wikipedia parsers of various qualities. Some do not work at all, some work only on certain articles, some have been abandoned as incomplete and some are slow as molasses.

I was delighted to stumble upon Wikipedia Extractor, a Python library developed by Antonio Fuschetto, Multimedia Laboratory, Dipartimento di Informatica, Università di Pisa, that extracts plain-text from the Wikipedia XML dump file. The script is heavily object-oriented, and it is very easy to modify and extend for other purposes. For me, it is the easiest parser to use and yields the best quality output although there are other options.

Pros

• Very easy to run; it’s just a Python script.
• Yields high quality output; no stray wikisyntax garbage.
• Highly object-oriented; easy to extend and embed in text mining projects.
• Object-oriented style makes it easier to parallelize with lightweight processes (written by the user).
• Allows specifying the maximum size of each produced file (good for sending to S3).
• It is written in Python.

Cons

• Far too slow. Python profilers show major overhead involved in regex search and replace, and string replacement.
• Is not perfect, but one of the best I have seen. For some reason, Wikilinks are converted to HTML links. Correcting this required modifying the source code.
• Retooling the package to work with Hadoop Streaming is not too difficult, but requires some work and grokery that should be easier.

Wikipedia Extractor is good for offline analysis, but users will probably want something that runs faster. Wikipedia Extractor parsed the entire Wikipedia dump in approximately 13 hours, on one core, which is quite painful. Add in further parsing and the processing time becomes unbearable even on multiple cores. A Hadoop Streaming job using Wikipedia Extractor as well as too much file I/O between Elastic MapReduce and S3 required 10 hours to complete on 15 c1.medium instances. 

Ken Weiner (@kweiner) recently re-introduced me to the Cloud9 package by Jimmy Lin (@lintool) of Twitter which fills in some of these gaps. I avoided it at first because Java is not the first language I like to turn to. Cloud9 is written in Java and designed for use with Hadoop MapReduce in mind. There is a method within the package that explicitly extracts the body text of each Wikipedia article. This method calls the Bliki Wikipedia parsing library. One common problem with these Wikipedia parsers is that they often leave syntax still in the output. Jimmy seems to wrap Bliki with his own code to do a better job of extracting high quality text only output. Cloud9 also has counters and functions that detect non-article content such as redirects, disambiguation pages, and more.

Developers can introduce their own analysis, text mining and NLP code to process the article text in the mapper or reducer code. An example job distributed with Cloud9 which simply counts the number of pages in the corpus took approximately 15 minutes to run on 8 cores on an EC2 instance. A job that did more substantial required 3 hours to complete, and once the corpus was refactored as sequence files, the same job took approximately 90 minutes to run.

Conclusion

I am looking forward to playing with Cloud9 some more… I will take 90 minutes over 10 hours any day! Wikipedia Extractor is an impressive Python package that does a very good job of extracting plain text from Wikipedia articles and for that I am grateful. Unfortunately, it is far too slow to be used on a pay-per-use system such as AWS or for quick processing. Cloud9 is a Java package designed with scalability and MapReduce in mind, allowing much quicker and more wallet friendly processing.

LexisNexis has open-sourced its alternative to Hadoop, called High Performance Computing Cluster. The code is available on GitHub. For years the code was restricted to LexisNexis Risk Solutions. The system contains two major components:

The protocol that drives the whole process is the Enterprise Control Language which is said to be faster and more efficient than Hadoop’s version of MapReduce. A picture is a much better way to show how the system works. Below is a diagram from the Gigaom article from which most of this information originates.

To me, Roxie seems much more exciting because it seems to complement (or replace) several technologies currently in the space. I do not know all the details, but it seems to potentially encapsulate technologies such as HBase, Hive, RabbitMQ and MemcacheDB, technologies that are common used to query and [...]]]>

A month ago, I wrote about alternatives to the Hadoop MapReduce platform and HPCC was included in that article. For more information, see here.

LexisNexis has open-sourced its alternative to Hadoop, called High Performance Computing Cluster. The code is available on GitHub. For years the code was restricted to LexisNexis Risk Solutions. The system contains two major components:

• Thor (Thor Data Refinery Cluster) is the data processing framework. It “crunches, analyzes and indexes huge amounts of data a la Hadoop.”
• Roxie (Roxy Radid Data Delivery Cluster) is more like a data warehouse and is designed with quick querying in mind for frontends.

The protocol that drives the whole process is the Enterprise Control Language which is said to be faster and more efficient than Hadoop’s version of MapReduce. A picture is a much better way to show how the system works. Below is a diagram from the Gigaom article from which most of this information originates.

To me, Roxie seems much more exciting because it seems to complement (or replace) several technologies currently in the space. I do not know all the details, but it seems to potentially encapsulate technologies such as HBase, Hive, RabbitMQ and MemcacheDB, technologies that are common used to query and speed data to a web frontend.

My opinion on HPCC is mixed. Although Hadoop has already taken off in usage, LexisNexis is a very strong institution and could potentially convince some corporate users to use their system instead — those that do not want to use Microsoft’s Dryad project. I do not see HPCC being a Hadoop killer, just as I do not see Spark or any other alternative to be a Hadoop killer. However, if HPCC does become a strong alternative, I sense this could be trouble for some of the newer players in the Hadoop field such as HortonWorks and MapR. I do not have much of an interest in studying business and competition, but Hadoop Summit 2011 showed that the Hadoop space has become crowded, and small breakthroughs such as another company developing a similar project is enough to add volatility and uncertainty for all involved.

<< My review of Day 1.

I am summarizing all of the days together since each talk was short, and I was too exhausted to write a post after each day. Due to the broken-up schedule of the KDD sessions, I group everything together instead of switching back and forth among a dozen different topics. By far the most enjoyable and interesting aspects of the conference were the breakout sessions.

Keynotes

KDD 2011 featured several keynote speeches that were spread out among three days and throughout the day. This year’s conference had a few big names.

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year. AdMeld did an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!

Mining and Learning on Graphs Workshop 2011

I had originally planned to attend the 2-day workshop Mining and Learning with Graphs (MLG2011) but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first talk I arrived for was Networks Spill the Beans by Lada Adamic from the University of Michigan. Adamic’s presented work involved inferring properties of content (the “what”) using network structure alone (using only the “who”: who shares with whom). One example she presented involved questions and answers on a Java programming language forum. The research problem was to determine things such as who is most likely to answer a Java beginner’s question: a guru, or a slightly more experienced user? Another research question asked what dynamic interactions tell us about information flow. [...]]]>

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year. AdMeld did an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!

Mining and Learning on Graphs Workshop 2011

I had originally planned to attend the 2-day workshop Mining and Learning with Graphs (MLG2011) but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first talk I arrived for was Networks Spill the Beans by Lada Adamic from the University of Michigan. Adamic’s presented work involved inferring properties of content (the “what”) using network structure alone (using only the “who”: who shares with whom). One example she presented involved questions and answers on a Java programming language forum. The research problem was to determine things such as who is most likely to answer a Java beginner’s question: a guru, or a slightly more experienced user? Another research question asked what dynamic interactions tell us about information flow. For this example, Adamic used data from the virtual world SecondLife. Certain landmarks (such as a bench) can be bookmarked by users and certain gestures (like a kiss) can be studied. This made my ears rise. SecondLife is a treasure trove of cool data. Is there a way to access it? It looks there might be a way to access some of it including monetary valuation, market purchases, and several APIs for different aspects of SecondLife.  I will have to look into that later though. Adamic concluded with a discussion of Twitter as a social network, but I was starting to fall asleep from my hectic and early morning. The gist of her talk, and many other talks in this field, was to combine semantic variables (NLP) with topological variables (SNA) to predict som other semantic variables. This talk was very digestible, and very interesting (despite my lack of sleep), but featured some of the worst visualizations I have ever seen (area plots representing correlations across multiple levels of an ordinal variable), but that was minor. Of course, Nathan might disagree .

Social network analysis, and network analysis in general, is a field that I really want to sink my teeth into. The difficulty I have is that the discussion of this field seems to involve so much vernacular that is specific to the field that everything seems so much more difficult than it really is.

At this point I took off to lunch. Just across from the Hyatt (a beautiful hotel by the way) is Seaport Village, a beautiful waterfront park containing nice landscaping, shops, restaurants, all with the ocean in the background. There is no beach there — the village backs right up to the water. Across the bay is some type of military complex and Coronado Island. I had a $7 hot dog, followed by a chocolate-covered strawberry and a peanut butter cup from the nearby candy store. It was such a nice day I walked around for a while, grabbed a strawberry shake and then headed back for the next session… the one I had been waiting for!

Afternoon Tutorial: Probabilistic Topic Models
David Blei, Princeton

My dissertation topic is related to Latent Dirichlet allocation (well, topic modeling in general), so I was definitely interested to hear what the father of LDA had to say. Since this was a 3 hour tutorial, I was expecting that Blei would start with the unigram model, and then discuss Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Indexing (pLSI) building up to LDA. Instead, Blei started with LDA and for good reason! In this post, I will not summarize the mechanics of Latent Dirichlet Allocation as that is another post entirely. For some introduction, see here. LDA and its extensions can be used to model the evolution of topics over time, to model the connections among topics, and to predict links among objects in a network. Topic modeling is a case study in machine learning rather than a field in itself; topic modeling draws on several different concepts including Bayesian statistics, time series analysis, hierarchical models, Markov chain monte carlo (MCMC), Bayesian non-parametric statistics and sparsity. In LDA, a document is represented as a mixture of topics (some hypothetical quantity that captures content clustering), and a topic is a distribution over words in a vocabulary.

Again, this is a high-level description of what was discussed. A full mechanical analysis would require dozens of pages. LDA is just a probabilistic model. As such, there are established ways for estimating the parameters of the model as well as the topic assignments. Some of these include mean field variational methods, expectation propagation, Gibbs sampling, collapsed Gibbs sampling, collapsed variational Bayes and online variational Bayes. Each of these estimation methods has its own advantages and disadvantages. Blei showed the LDA and pLSI have a lot in common. Unlike LDA, pLSI uses maximum likelihood estimations (and the EM algorithm) for parameter estimation; pLSI tends to overfit badly. The hyperparameter α adds regularization to the ϴ parameter in the LDA model. [Sorry to refer to these random parameters, but it is difficult to describe without them. See the links mentioned earlier for an overview of LDA.]

Preprocessing. A lot of preprocessing must be performed before computing a topic model. First, we should remove stopwords, which are words that provide absolutely no clues to the content of the text. If we leave stopwords in the corpus when computing the model, we may end up with meaningless topics that are described with only stopwords, due to their high probability. Second, Blei mentioned that stemming is a good idea, but modern stemming algorithms tend to be too aggressive. If resources allow, I think it would be useful to have humans manually strip words to their root words. Multiword phrases such as “black hole” are also an issue. With sufficient resources, one could ask human labelers to identify these phrases and recode them as a single word by replacing the space between words with an underscore. Hanna Wallach (U. Mass) has a paper that describes how to identify multiwork phrases by using n-grams. Blei has a similar paper that discusses an algorithm called TurboTopics. He also mentioned that a standard statistical hypothesis test such as chi-squared, permutation tests, or a nested hypothesis test would also be sufficient, though inefficient. I have not thought of how this would work however. Finally, remove rare words because they can lead to local optima in the likelihood surface probably yielding inefficient computation.

Some hairy details. One of the parameters that makes LDA useful is  α. α is a hyperparameter in the LDA model that determines the sparsity of draws from the underlying Dirichlet distribution. α is typically a small number; Blei mentioned that 0.01 is a good a priori value for α. As α gets larger, the distribution of topics tends towards the uniform (each topic equally likely) distribution and as α approaches 0, we get sparser draws, meaning more peaked topic probabilities. Setting α to be ridiculously small (i.e. 0.001) may yield a single topic dominating the model. α can be chosen, or we can fit α to the data using cross-validation or some other method. He also discussed the parameter η.

Open source software. We quickly (flash of an eye) went through a list of some open-source LDA implementations:

• LDA-C (variational EM), Blei.
• HDP (hierarchical Dirichlet processes), Blei
• LDA (R package, collapsed Gibbs), Jonathan Chang, Data Scientist, Facebook.
• Lingpipe, alias-i
• Mallet (collapsed Gibbs), UMass
• Gensim (online and batch LDA), Radim Řehůřek

To my delight, Blei seemed to favor the R package (although Gensim is a nice Python implementation). The R package not only contains LDA, but several other models including RTMs, MMSB and sLDA which will be discussed later. It is supposedly fast as well. The output from the R package can be visualized using the Topic Model Visualizer by Allison Chaney.

The beauty of LDA is that it can be embedded in many more complicated models. Some applications of these extensions include word sense, graphs and hierarchies. Before delving into specifics, there are a couple of changes to the LDA model that motivate the next topics.

1. The probability of observing word w given a set of topics β and a set of topic labels z is given by P(w|β,z) which is multinomial. The distribution of P(w|ß,z) can be changed depending on what we are modeling. For example, for count data, P(w|β,z) can be Poisson. This drastically changes the model, however. In LDA, P(w|ß,z) is multinomial which is convenient because it is the conjugate prior of the Dirichlet distribution.
2. The characteristic LDA posterior distribution can be used in more creative ways…

Correlated Topic Model. In LDA, all topics are considered independent of each other, and this is usually unrealistic. CTM allows the topics to be correlated. For example, a paper classified as about calculus is more likely to also be classified as about physics, than it is to be classified as about sewing. Blei mentioned that CTM allows for better prediction, likely because it is more realistic. CTM is also more robust to overfitting. The main distinction from LDA is that ϴ follows the logistic normal distribution instead of the Dirichlet distribution.

Dynamic Topic Model. DTM models how each individual topic changes over time. One example Blei showed involved a topic that could be labeled “technology”. In the late 1700s, this topic contained the words “coal”, “steel” (I am making it up from memory…probably badly…bear with me) and in 2011 contained the words “silicon” and “solar”. The main distinction from LDA is two-fold: assuming the topic at time t is normally distributed with the topic at time t-1 as the mean and some variance. That is,

and

instead of multinomial.

A limitation of DTM is that it does not handle the death of a topic gracefully.

Supervised LDA. In sLDA, we associate each document with an external variable. For example, a document may be a Yelp review containing text. The external variable associated with the Yelp review may be the number of stars in the associated rating. We can use sLDA to use the topics estimated by LDA as regressors to predict this external variable Y. Various types of regression can be performed from standard linear regression to the generalized linear model (GLM).The Yelp example would likely use an ordered logit model for Y.

Relational Topic Models. RTM applies sLDA to every pair of documents in a corpus and attempts to use content to predict connectedness in a graph. For example, given the content on my Facebook profile, one could use sLDA to predict what kind of reaction I would have to an ad (i.e. click or no click) and this could be used for targeted ad serving, or any other type of recommendation engine. Think collaborative filtering! RTM is also good for certains types of data that have spatial/geographic dependencies.

Ideal Point Topic Models were barely touched upon since we were running short on time (although we voted to extend the session by 30 mins and Blei happily obliged). They seem particularly useful in political science for predicting roll call votes.

Bayesian Non-Parametric Models are a hot topic but are too complicated to describe here. In LDA, the number of topics is determined a priori and remains fixed throughout the model. In real life, topics can be “born” and can “die” off and we may not know a priori how many topics to use. One can model the latter situation as a Chinese Restaurant Process where each table is associated with a topic. Furthermore, a Chinese Restaurant Franchise can be used for modeling hierarchies (hLDA). In CRF, there is a corpus level restaurant where each table is a parameter and a topic (called plates). Then, each document has its own Chinese restaurant where each table is associated with a customer in the corpus level Chinese restaurant. Blei recommended a book by Hjort.

Algorithms. The last few minutes were dedicated to discussing inference algorithms for LDA, particularly Gibbs sampling and variational Bayes. Gibbs sampling is very simple to implement, though Blei stated that it does not work for DTM or CTM because the assumptions of conjugacy (multinomial/Dirichlet) are violated. Variational Bayes is more difficult to implement, but handles non-conjugacy in CTM and DTM much better.

Plenary Sessions

The plenary sessions consisted of several thank-yous and awards. The committee provided some humor which gave some humility to the long process of writing and submitting papers. They went over paper acceptance statistics and read some of the funnier comments that reviewers gave, one of which was something like “It is clear that the author did not read this paper before submitting it.” I don’t know how many times I have said that in various situations. The committee handed out awards for best paper and best dissertation. This year’s KDD Cup competition was a contest similar to the Netflix challenge, but involved music recommendation. The winner was the National Taiwan University, for the fourth straight year in a row I am told. The innovation award went to a researcher dear to my heart, Ross Quinlan, who developed the C4.5 decision tree modeling software.

For more information about topic modeling software, see David Blei’s website at http://www.cs.princeton.edu/~blei which contains code for most if not all of these topic models. For the notes from the tutorial, see http://www.cs.princeton.edu/~blei/kdd-tutorial.pdf.

Hadoop has become the de facto standard in the research and industry uses of small and large-scale MapReduce. Since its inception, an entire ecosystem has been built around it including conferences (Hadoop World, Hadoop Summit), books, training, and commercial distributions (Cloudera, Hortonworks, MapR) with support. Several projects that integrate with Hadoop have been released from the Apache incubator and are designed for certain use cases:

Hadoop is meant to be modeled after Google MapReduce. To store and process huge amounts of data, we typically need several machines in some cluster configuration. A distributed filesystem (HDFS for Hadoop) uses space across [...]]]>

It’s been a while since I have posted… in the midst of trying to plow through this dissertation while working on papers for submission to some conferences.

Hadoop has become the de facto standard in the research and industry uses of small and large-scale MapReduce. Since its inception, an entire ecosystem has been built around it including conferences (Hadoop World, Hadoop Summit), books, training, and commercial distributions (Cloudera, Hortonworks, MapR) with support. Several projects that integrate with Hadoop have been released from the Apache incubator and are designed for certain use cases:

• Pig, developed at Yahoo, is a high-level scripting language for working with big data and Hive is a SQL-like query language for big data in a warehouse configuration.
• HBase, developed at Facebook, is a column-oriented database often used as a datastore on which MapReduce jobs can be executed.
• ZooKeeper and Chukwa
• Mahout is a library for scalable machine learning, part of which can use Hadoop.
• Cascading (Chris Wensel), Oozie (Yahoo) and Azkaban (LinkedIn) provide MapReduce job workflows and scheduling.

Hadoop is meant to be modeled after Google MapReduce. To store and process huge amounts of data, we typically need several machines in some cluster configuration. A distributed filesystem (HDFS for Hadoop) uses space across a cluster to store data so that it appears to be in a contiguous volume and provides redundancy to prevent data loss. The distributed filesystem also allows data collectors to dump data into HDFS so that it is already prime for use with MapReduce. A Data Scientist or Software Engineer then writes a Hadoop MapReduce job.

As a review, the Hadoop job consists of two main steps, a map step and a reduce step. There may optionally be other steps before the map phase or between the map and reduce phases. The map step reads in a bunch of data, does something to it, and emits a series of key-value pairs. One can think of the map phase as a partitioner. In text mining, the map phase is where most parsing and cleaning is performed. The output of the mappers is sorted and then fed into a series of reducers. The reduce step takes the key value pairs and computes some aggregate (reduced) set of data such as a sum, average, etc. The trivial word count exercise starts with a map phase where text is parsed and a key-value pair is emitted: a word, followed by the number “1″ indicating that the key-value pair represents 1 instance of the word. The user might also emit something to coerce Hadoop into passing data into different reducers. The words and 1s are sorted and passed to the reducers. The reducers take like key-value pairs and compute the number of times the word appears in the original input.

After working extensively with (Vanilla) Hadoop professional for the past 6 months, and at home for research, I have found several nagging issues with Hadoop that have convinced me to look elsewhere for everyday use and certain applications. For these applications, the though of writing a Hadoop job makes me take a deep breath. Before I continue, I will say that I still love Hadoop and the community.

• Writing Hadoop jobs in Java is very time consuming because everything must be a class, and many times these classes extend several other classes or extend multiple interfaces; the Java API is very bloated. Adding a simple counter to a Hadoop job becomes a chore of its own.
• Documentation for the bloated Java API is sufficient, but not the most helpful.
• HDFS is complicated and has plenty of issues of its own. I recently heard a story about data loss in HDFS just because the IP address block used by the cluster changed.
• Debugging a failure is a nightmare; is it the code itself? Is it a configuration parameter? Is it the cluster or one/several machines on the cluster? Is it the filesystem or disk itself? Who knows?!
• Logging is verbose to the point that finding errors is like finding a needle in a haystack. That is, if you are even lucky to have an error recorded! I’ve had plenty of instances where jobs fail and there is absolutely nothing in the stdout or stderr logs.
• Large clusters require a dedicated team to keep it running properly, but that is not surprising.
• Writing a Hadoop job becomes a software engineering task rather than a data analysis task.

Hadoop will be around for a long time, and for good reason. MapReduce cannot solve every problem (fact), and Hadoop can solve even fewer problems (opinion?). After dealing with some of the innards of Hadoop, I’ve often said to myself “there must be a better way.” For large corporations that routinely crunch large amounts of data using MapReduce, Hadoop is still a great choice. For research, experimentation, and everyday data munging, one of these other frameworks may be better if the advantages of HDFS are not necessarily imperative:

BashReduce

Unlike Hadoop, BashReduce is just a script! BashReduce implements MapReduce for standard Unix commands such as sort, awk, grep, join etc. It supports mapping/partitioning, reducing, and merging. The developers note that BashReduce “sort of” handles task coordination and a distributed file system. In my opinion, these are strengths rather than weaknesses. There is actually no task coordination as a master process simply fires off jobs and data. There is also no distributed file system at all, but BashReduce will distribute files to worker machines. Of course, without a distributed file system there is a lack of fault-tolerance among other things.

Intermachine communication is facilitated with simple passwordless SSH, but there is a large cost associated with transferring files from a master machine to its workers whereas with Hadoop, data is stored centrally in HDFS. Additionally, partition/merge in the standard unix tools is not optimized for this use case, thus the developer had to use a few additional C programs to speed up the process.

Compared to Hadoop, there is less complexity and faster development. The result is the lack of fault-tolerance, and lack of flexibility as BashReduce only works with certain Unix commands. Unlike Hadoop, BashReduce is more of a tool than a full system for MapReduce. BashReduce was developed by Erik Frey et. al. of last.fm.

Disco Project

Disco was initially developed by Nokia Research and has been around silently for a few years. Developers write MapReduce jobs in simple, beautiful Python. Disco’s backend is written in Erlang, a scalable functional language with built-in support for concurrency, fault tolerance and distribution — perfect for a MapReduce system! Similar to Hadoop, Disco distributes and replicates data, but it does not use its own file system. Disco also has efficient job scheduling features.

It seems that Disco is a pretty standard and powerful MapReduce implementation that removes some of the painful aspects of Hadoop, but it also likely removes persistent fault tolerance as it relies on a standard filesystem rather than one like HDFS, but Erlang may impose some functionality that provides a “good enough” level of fault tolerance for data.

Spark

Spark is one of the newest players in the MapReduce field. Its purpose is to make data analytics fast to write, and fast to run. Unlike many MapReduce systems, Spark allows in-memory querying of data (even distributed across machines) rather than using disk I/O. It is of no surprise then that Spark out-performs Hadoop on many iterative algorithms. Spark is implemented in Scala, a functional object-oriented language that sits on top of the JVM. Similar to other languages like Python, Ruby, and Clojure, Scala has an interactive propt and users can use Spark to query big data straight from the Scala interpreter.

One wrinkle is that Spark requires installing a cluster manager called Mesos. I had some difficulty installing it on Ubuntu, but the development team was an amazing help, and made a few changes to the source and now it runs well. On the downside, Mesos adds a layer of complexity that we are trying to avoid. On the upside, Mesos allows Spark to co-exist with Hadoop and it can read any data source that Hadoop supports, and it “feels” light, similar to Disco’s server UI.

Spark was developed by the UC Berkeley AMP Lab. Currently, its main users are UC Berkeley researchers and Conviva. Hadoop Summit 2011 featured a talk on Spark by one of the developers, which I wrote about earlier this summer.

GraphLab

GraphLab was developed at Carnegie Mellon and is designed for use in machine learning. GraphLab’s goal is to make the design and implementation of efficient and correct parallel machine learning algorithms easier. Their website states that paradigms like MapReduce lack expressiveness while lower level tools such as MPI present overhead by requiring the researcher to write code that beats a dead horse.

GraphLab has its own version of the map stage, called the update phase. Unlike MapReduce, the update phase can both read and modify overlapping sets of data. Recall that MapReduce requires data to be partitioned. GraphLab accomplishes this by allowing the user to specify data as a graph where each vertex and edge in the graph is associated memory. The update phases can be chained in such a way such that one update function can recursively trigger other update functions that operate on vertices in the graph. This graph-based approach would not only make machine learning on graphs more tractable, but it also improves dynamic iterative algorithms.

GraphLab also has its own version of the reduce stage, called the sync operation. The results of the sync operation are global and can be used by all vertices in the graph. In MapReduce, output from the reducers is local (until committed) and there is a strict data barrier among reducers. The sync operations are performed at time intervals, and there is not as strong of a tie between the update and sync phases. What I mean is that the sync intervals are not necessarily dependent on some prior update completing.

GraphLab’s website also contains the original UAI paper and presentation, a document better explaining the abstraction, and there is even a Google Group for the GraphLab API. To me, GraphLab seems like a very powerful generalization, and re-specification, of MapReduce.

Storm

Recently, Nathan Marz of BackType made waves in the Twitter big data community with a blog post titled Preview of Storm: The Hadoop of Realtime Processing. Within a day, Storm became known as “Real-time Hadoop” to the chagrin of some developers from Apache. Hadoop is a batch-processing system — that is, give it a lot of fixed data and it does something with it. Storm is real-time — it processes data in parallel as it streams.

Marz writes that with their previous system, much time was spent worrying about graphs of queues and workers: where to send and receive messages, deploying workers and queues, and a lack of fault tolerance. Storm abstracts all of these complications away. Storm is written in Clojure, but any programming language can be used to write programs on top of Storm. Storm is fault-tolerant, horizontally scalable, and reliable. Storm is also very fast, with ZeroMQ used as the underlying message passing system.

Nathan Marz is a software developer at BackType, and made waves in 2010 with Cascalog. Cascalog really took off after his presentation at the 2010 Hadoop Summit, and I am delighted I got to see him present it. Storm will be open-sourced soon and I hope to write more about it later.

I included Storm in this post based on its colloquial name “Real-time Hadoop” — it is not clear to me whether or not Storm even uses MapReduce though.

HPCC Systems (from LexisNexis)

Perhaps the project with the least flattering name comes from LexisNexis, which has developed its own framework for massive data analytics. HPCC attempts to make writing parallel-processing workflows easier by using Enterprise Control Language (ECL), a declarative, data-centric language. I should note that SQL, Datalog and Pig are also said to be declarative, data-centric languages. A matter of fact, the development team has a converter for translating Pig jobs to ECL. HPCC is written in C++. Some have commented that this will make in-memory querying much faster because there is less bloated object sizes originating from the JVM. I also prefer C++ simply because it feels closer to human though — we think in terms of objects (object-oriented) at times, and a series of steps (procedural) at other times and use both thought processes together.

HPCC already has its own jungle of technologies like Hadoop. HPCC has two “systems” for processing and serving data: the Thor Data Refinery Cluster, and the Roxy Rapid Data Delivery Cluster. Thor is a data processor, like Hadoop. Roxie is similar to a data warehouse (like HBase) and supports transactions. HPCC uses a distributed file system.

Although details are still preliminary as is the system, this certainly has a “feel” for potentially being a solid alternative for Hadoop, but only time will tell.

With all these alternatives, why use Hadoop?

One word: HDFS. For a moment, assume you could bring all of your files and data with you everywhere you go. No matter what system, or type of system, you login to, your data is intact waiting for you. Suppose you find a cool picture on the Internet. You save it directly to your file store and it goes everywhere you go. HDFS gives users the ability to dump very large datasets (usually log files) to this distributed filesystem and easily access it with tools, namely Hadoop. Not only does HDFS store a large amount of data, it is fault tolerant. Losing a disk, or a machine, typically does not spell disaster for your data. HDFS has become a reliable way to store data and share it with other open-source data analysis tools. Spark can read data from HDFS, but if you would rather stick with Hadoop, you can try to spice it up:

Hadoop Streaming is an easy way to avoid the monolith of Vanilla Hadoop without leaving HDFS, and allows the user to write map and reduce functions in any language that supports writing to stdout, and reading from stdin. Choosing a simple language such as Python for Streaming allows the user to focus more on writing code that processes data rather than software engineering. Once code is written, it is easy to test from the command line:

cat a_bunch_of_files | ./mapper.py | sort | ./reducer.py

And, running and monitoring the job is similar to Vanilla Hadoop. Hadoop Streaming was my first introduction to Hadoop and it was quite pleasant.

Or, you could use a Hadoopified project that better solves the problem. Vanilla Hadoop can do some sophisticated stuff, but it suffers the problems I mentioned at the beginning of the post. Developers have created software that works on HDFS, but is geared toward different audiences. A Data Scientist may prefer Pig or Hive for data analysis whereas a Systems and Software Engineer may prefer a workflow solution (Oozie, Cascading etc.) and a (modern) DBA may want to use HBase. Each of these achieve different goals, but still rely on HDFS.

This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.

Keynotes

We first heard a series of keynote speeches which I will summarize. The first keynote was from Jay Rossiter, SVP of the Cloud Platform Group at Yahoo. He introduced how Hadoop is used at Yahoo, which is fitting since they organized the event. The content of his presentation was very similar to last year’s. One interesting application of Hadoop at Yahoo was for “retiling” the map of the United States. I imagine this refers to the change in aerial imagery over time. When performed by hand, retiling took 6 weeks; with Hadoop, it took 5 days. Yahoo also [...]]]>

I woke up early and cheery Wednesday morning to attend the 2011 Hadoop Summit in Santa Clara, after a long drive from Los Angeles and the Big Data Camp that lasted until 10pm the night before. Having been to Hadoop Summit 2010, I was interested to see how much of the content in the conference had changed.

This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.

Keynotes

We first heard a series of keynote speeches which I will summarize. The first keynote was from Jay Rossiter, SVP of the Cloud Platform Group at Yahoo. He introduced how Hadoop is used at Yahoo, which is fitting since they organized the event. The content of his presentation was very similar to last year’s. One interesting application of Hadoop at Yahoo was for “retiling” the map of the United States. I imagine this refers to the change in aerial imagery over time. When performed by hand, retiling took 6 weeks; with Hadoop, it took 5 days. Yahoo also uses Hadoop for fraud detection, spam detection, search assist, geotagging data/local indexing, ad targeting, predicting supply and demand and the aggregation and categorization of news stories. Jay also mentioned that Dapper runs models on data with Hadoop for ad personalization. Jay also mentioned that Big Data conferences all over the country are selling out.

Eric Baldeschwieler, the CEO of Hortonworks was next. Hortonworks seems to be a new company that spun off from Yahoo. Their goal is to provide commercial support and a full Apache Hadoop platform for users. Yes, they are very similar to Cloudera, and yes, they are competition. (Hortonworks and MapR both did a good job of not stepping on everyone’s toes in terms of how they presented themselves.) Cloudera provides its own distribution of Hadoop, which is of course similar to the Apache version. Hortonworks’ goal is to provide similar services, but with more transparency by using the Apache Hadoop distribution rather than wrapping its own. Paraphrasing Eric, Hortonworks is open-source from the ground up. A bit later, Sanjay Radia also of Hortonworks discussed Hadoop for the enterprise. Hortonworks has contributed, or is working on security (preventing users from deleting others’ data), service level agreements (SLAs), predictability and a Fair-Share scheduler.

Anant Jhingran, CTO of IBM discussed how Hadoop was used in IBM Watson. It seemed pretty obvious that Hadoop or some form of map-reduce was used in the system, but it did not seem to be highly publicized. Watson learned from 200 million pages of data, about 2-5TB and required between 3000 and 4000 Watts. Anant went quickly through a cool user interface representing a Jeopardy board and stated that the user interface to an artificial intelligence application is just important as the application itself. He also prefers the term IA (intelligence augmentation) over AI, and apparently this is a common distinction. To me, I interpret AI vs. IA to be artificial intelligence vs. knowledge discovery (data mining).

Karthic Ranganathan from Facebook discussed Facebook’s messaging system which was built on HBase, HDFS and MapReduce. Facebook sees 15 billion messages per month, excluding SMS and email, approximately 14TB of data! There are also 120 billion chat messages (25TB), for a grand total of almost 300TB per month. (I may have missed something as these numbers do not add up). Facebook uses HBase for the bodies of small messages, metadata, and for the search index. Facebook uses HBase because of its high write throughput and easy horizontal scalability. Facebook uses another system called Haystack for photos, bodies of large messages and attachments. Of course, HDFS is used for fault tolerance, scalability, checksums for data integrity and its MapReduce abilities. Profiles and services are partitioned by user. Each machine has 16 cores, with 12 1TB hard disks, and 48GB RAM (24GB used for HBase). Some things that Facebook would like contribute and improve: NameNode high availability and a second NameNode, better performance overall, and using flash memory to improve performance. Facebook often adds several columns to a table so that DevOps does not need to take the server offline to add new columns.

Big Data conferences all over the country are selling out.

Breakout Sessions

There were so many great sessions and I can only summarize the ones I attended. Check out the event agenda for abstracts on all sessions.

First I attended Web Crawl Cache – Using HBase to Manage a Copy of the Web. In this talk, we learned about Yahoo’s Web Crawl Cache (WCC) that collects and organizes data from Microsoft as a result of a search deal. These snapshots of the web are not only useful for search, but also for drilling into other avenues such as local assets, influence and language corpora. WCC uses HBase for several reasons: bulk load, MapReduce jobs are efficient, random access reads, a usable consistency model, and it is easy to dynamically add columns (this seems to contradict Karthick’s claim).

It was very difficult to pick a session for the 1:45 to 2:15 time slot. Options included Next Generation Hadoop, Scaling out Realtime Data (Facebook) and Building Kafka (LinkedIn). I admire the work and clout that LinkedIn has built over the past year or two, so I attended Jay Kreps session. LinkedIn’s data pipeline includes a lot of tracking, logging, metrics, messages and queuing. LinkedIn attempted to use messaging systems such as JMS and RabbitMQ. Streaming data is prevalent at LinkedIn such as search trends, click trends, invitation social networks etc. Kafka is LinkedIn’s solution for a distributed message queue; rather than polling for data, users subscribe to a data stream and data sources publish data to it. Kafka is 7000 lines of Scala, a functional and object-oriented language on top of the Java Virtual Machine (JVM). Kafka can produce about 250,000 messages per second (50 MB) and consume 550,000 messages per second (110 MB).

Next I attended another talk by Hortonworks, this time on HCatalog. HCatalog changes the way we think about data in HDFS. No longer do we need to worry about files and directories. Instead, HCatalog seems to add a layer of abstraction on top of HDFS that treats data as a set of tables. Tools such as Pig and Hive use this layer of abstraction, and currently Hive is tightly integrated with HCatalog. Hortonworks intends to add support for HBase and Streaming later this year.

I waited all day to see Matei Zaharia‘s talk on Spark. Zaharia is a graduate student at UC Berkeley and it was a nice change of pace to see a student present some work. Spark is a data processing platform that sits on top of the Mesos cluster management project (also produced by Berkeley). Mesos can handle 10,000s nodes, 100s of concurrent jobs and can be isolated in Linux containers (i.e. OpenVZ). Spark aims to extend MapReduce for iterative algorithms, and interactive low latency data mining. One major difference between MapReduce and Spark is that MapReduce is acyclic. That is, data flows in from a stable source, is processed, and flows out to a stable filesystem. Spark allows iterative computation on the same data, which would form a cycle if jobs were visualized. Resilient Distributed Dataset (RDD) serves as an abstraction to raw data, and some data is kept in memory and cached for later use. This last point is very important; Spark allows data to be committed in RAM for an approximate 20x speedup over MapReduce based on disks. RDDs are immutable and created through parallel transformations such as map, filter, groupBy and reduce. RDD immutability is similar to immutable types in functional programming languages. It does not mean that the dataset cannot change. Instead, it means that a new copy of the dataset is created, with the change included. The user can also perform actions on RDDs such as count, collect, etc. Some applications using Spark are traffic prediction (Berkeley), spam classification (Twitter), kmeans, alternating least squares matrix factorization, and network simulation.

The main takeaway from Hadoop Summit 2010 was Cascalog. I predict the main takeaway from Hadoop Summit 2011 is Spark.

One time at work I had a bizarre issue with corrupted data in HDFS. After that, I began blaming everything on HDFS. The next session Data Integrity and Availability of HDFS was englightening. HDFS takes good care of Yahoo’s data. We can trust Yahoo because if HDFS breaks, Yahoo begins losing money so they know what they are talking about! Yahoo’s goal is to have 60 PB online all the time. The key to HDFS reliability is replication. A replication factor of 3 (3 copies of every file? block?) is appropriate. A replication factor of 2 is also quite robust, but should only be used when there is a backup of the data because the probability of data loss is much higher. Yahoo has had issues with losing blocks (blocks are pieces of data, so lost blocks = data loss). There are a variety of reasons and most of them had nothing to do with HDFS. One cause of lost blocks is a bug in a Hadoop component like Pig, particularly a new version. In one incident, a new version of Pig opened a lot of files without closing them, and created a lot of abandoned files. In the speaker’s anecdotal case study, none of the incidents of data loss were caused by HDFS proper. Other causes of data loss encoutered were exhausting disk space, users hammering HDFS, and “other.” The speaker noted that NameNode high availability (a hot topic) would have only helped in 8 of the 36 incidents studied. Some ways of preventing data loss include resource allocation, selecting good tenants of a cluster, and fixing hardware errors quickly.

If your job isn’t running, it’s not likely caused by HDFS.

Bill Graham of CBS Interactive gave an interesting talk about using Hadoop to build a graph of users a content. Surprisingly, CBSi has quite a large arsenal of MapReduce enabled technologies: Chukwa, Pig, Hive, HBase, Cascading, Sqoop and Oozie. CBSi uses only 100 nodes with 500 TB of disk space for processing data associated with 235 million uniques (individuals, roughly). Mapping users to content should be easy, right? Well, some users have multiple identities, including anonymous identities. The goal is to create a holistic graph that “matches” all of the identities efficiently for uses such as ad targeting. CBSi’s needs in a Hadoop platform: rapid experimentation and data mining, and to power new site features and ad optimization. The main vehicle for representing data is a Pig RDF that allows for a kind of graph based join so to speak. CBSi hopes to add Oozie, Azkaban, HCatalog and Hama (graph processing) to its arsenal.

MapR was a very prominent sponsor of Hadoop Summit. M. C. Srivas presented a technical discussion of MapR’s capabilities and how it differs from Apache Hadoop. MapR is a full distribution of Hadoop and is 100% compatible with the Apache distribution and projects such as Pig and Oozie. MapR is fast and boasts high availability by rethinking the NameNode. The NameNode is a bottleneck since 60% of file operations are metadata. The NameNode and its limitations limit the size of a cluster. To resolve some problems with the NameNode, MapR turns every server into a metadata server. Since metadata is seldom retrieved, it is paged to disk so more RAM can be used for MapReduce proper. MapR distributes NameNode functionality and provides full random read and write semantics as well as export to NFS. With the distributed NameNode, runaway tasks no longer take down the NameNode. MapR has some lofty performance goals. While HDFS can handle 10-50PB, MapR can handle 1010 EB (exabytes). While HDFS can handle 2000 nodes in a cluster, MapR can handle 10,000 or more. It was mentioned at BigDataCamp that MapR does not rely on HDFS at all.

The final session I attended was Avery Ching’s talk on Giraph: Large-scale Graph Processing on Hadoop. Unfortunately, Avery jumped right into the technical details of Giraph without giving a high level overview of the problem Giraph solves. Also, his slides were in 10 point font and I could not read them. Combine this with the fact that my brain was exhausted, so I wanted to head to the bar. Vanilla Hadoop incurs too much overhead for graph data processing. Yahoo used MPI in the past for graph data but it had no fault tolerance and was too generic. Giraph is a library for iterative graph processing. Giraph is fault-tolerant and dynamic. Giraph takes a vertex centric approach to graph data. I found this interesting because most of my work is edge centric. Overall, Giraph is similar in goal to Pregel, but available to non-Googlers and has no single point of failure (except those incurred by Hadoop).

Now I have to catch my breath with some wine, beer and cheese at the nice happy hour reception afterwards. It was a long day, and a great day at Hadoop Summit 2011 and I will of course be back next year. I have no clue what is in store for me next year. Will the NameNode be removed as the single point of failure? Will other open-source software start integrating Hadoop? We shall see…

And now it is time to head back to Los Angeles.

Have a happy and safe Fourth of July!

The food/drinks/networking event must have been amazing because it was very difficult to get everyone to come to the main room to start the event! We started with a series of lightning talks from some familiar names and some unfamiliar ones.

Chris Wensel, the developer of Cascading, is also the founder of Concurrent, Inc. Cascading is an alternate API for Map-Reduce written in Java. With Cascading, developers can chain multiple map-reduce jobs to form an ad hoc workflow. Cascading adds a built-in planner to manage jobs. Cascading usually infers Hadoop, but Cascading can run on other platforms including EMC Greenplum and the new MapR project. RazorFish and BestBuy use Cascading for behavioral targeting. Flightcaster uses a domain specific language (DSL) [...]]]>

It has been a while since I have been to Silicon Valley, but Hadoop Summit gave me the opportunity to go. To make the most of the long trip, I also decided to check out BigDataCamp held the night before from 5:30 to 10pm. Although the weather was as predicted, I was not prepared for the deluge of pouring rain in the end of June. The weather is one of the things that is preventing me from moving up to Silicon Valley.

The food/drinks/networking event must have been amazing because it was very difficult to get everyone to come to the main room to start the event! We started with a series of lightning talks from some familiar names and some unfamiliar ones.

Chris Wensel, the developer of Cascading, is also the founder of Concurrent, Inc. Cascading is an alternate API for Map-Reduce written in Java. With Cascading, developers can chain multiple map-reduce jobs to form an ad hoc workflow. Cascading adds a built-in planner to manage jobs. Cascading usually infers Hadoop, but Cascading can run on other platforms including EMC Greenplum and the new MapR project. RazorFish and BestBuy use Cascading for behavioral targeting. Flightcaster uses a domain specific language (DSL) written in Clojure on top of Cascading for large data processing jobs. Etsy uses a DSL written in JRuby as a layer on top of Cascading. Of course, the big player is BackType. Cascalog combines Cascading with the Datalog language to provide a declarative language for working with data and map-reduce. Wensel noted that one disadvantage of Pig and Hive that Cascading addresses is that Pig and Hive lack a physical planner. Workflow managers such as Oozie and Azaband can run Cascading jobs as part of a workflow. Version 2.0 of Cascading removes Hadoop as a dependency and will allow users to run Cascading jobs on data that is in RAM rather than on disk.

James Falgout from Pervasive DataRush presented the second lightning talk. Pervasive’s products seem to use this “dataflow” paradigm that attempts to fill in features that are missing in map-reduce. The basic description compared dataflow to the Unix shell pipeline with message passing. James showed an example dataflow that a user could configure visually. Pervasive is working on integrating dataflow with Hive.

Guy Harrison from Quest Software introduced their system Toad for Cloud Databases. Toad attempts to merge data from several different data sources for analysis such as Hive, MongoDB, and Cassandra. Unfortunately, Guy’s thick Australian accent made his humorous talk unintelligible to me (hearing loss?).

Steve Wooledge from AsterData (now part of Teradata) discussed the company’s product goal of taking a standard relational database system and integrating map-reduce on top of it. Such a system is flexible and allows both SQL-like access as well as programmatic access to data. This hybrid row-oriented and column-oriented datastore can be used for path and pattern matching, text processing and graph traversal among the usual tasks. nPath is a product that enhances a system with transactional data analytics (click analysis, sessionalization).

Andrew Yu from EMC presented some of EMC’s data analytics products. I wrote about EMC in an earlier blog post so I will spare the details. EMC offers a data warehouse product as well as a hybrid, pre-configured system containing its Greenplum warehouse and map-reduce built-in.

Ben Lee from Foursquare discussed how big data is used at Foursquare and gave some statistics about its service. This was by far the most interesting talk to me. Foursqaure offers realtime suggestions of places to visit based on the user’s history, and the user’s friends’ histories based on day of week and time of day. Foursquare has 10 million users, 50 million venues, and 750 million check-ins. There are over 3 million check-ins per day. 10,000 developers use Foursquare’s API. MongoDB is the main datastore and Scala is used for the front end. Back end data processing uses Hadoop (both vanilla, and Streaming) as well as Flume, Elastic MapReduce, and S3. Ben displayed an awesome visualization of check-in data; researchers took check-ins from New York City and performed sentiment analysis on the text attached to the check-in. The visualization suggested that people were the “happiest” in Manhattan.

Paul Baclace introduced some software called Phatvis that allows developers to visualize map-reduce jobs. It is his hope that the visualization can be used to fine tune Hadoop parameters based on evidence from prior jobs. The source can be found here

Of course, the fun in every “unconference” is the circus known as scheduling the sessions. Some of the proposed sessions:

• Big Data 101 / Intro to Hadoop
• Extract MapReduce Data into Relational Database High Performance Database
• “ETL was Yesterday” What’s next?
• Operations of Hadoop Clusters
• SQL / NoSQL Why not Both? (Aster)
• Geodata
• Big Data Retention / Compression
• Business Intelligence and Hadoop
• Data Management Lifecycle
• Distributions of Hadoop
• Hadoop for Bioinformatics and Healthcare

The topics did not seem exciting this time, and seemed to have a lot of overlap with presentations at Hadoop Summit, but I found two (we could only attend two) that stood out.

Session 1: Operating a Hadoop Cluster

Thank goodness managing a Hadoop cluster is not in my job description (only small clusters I use for research). Charles Wimmer, the lead of the Operations track for Hadoop Summit, lead this discussion and much of the discussion dovetailed off of incidents that occurred at Yahoo. A popular topic of discussion was backup. There is no such thing as “backing up” a Hadoop cluster we agreed. Any data that is important should be replicated, preferrably 3 times, or transmitted in parallel over a pipe to multiple data centers. One strict limitation of replication is that if some new release of Hadoop, or some new Hadoop distribution contains a bug that corrupts the data, all replicates may also be corrupted.

Discussion then turned to hardware. Yahoo uses high-density storage nodes with 6 drives each containing 2-3TB of space. Charles mentioned that a common problem with Hadoop is that it is difficult to keep the CPUs busy especially in a server with 8 Nehalem processors (8 CPUs or 8 cores?). The major reason for this is that the main bottleneck in map-reduce jobs is the network I/O required in the shuffle phase as data comes out of the mappers. The map phase is the most CPU bound phase. Wimmer, and several others, made one thing clear: use SATA, not SAS. Apparently SATA and SAS drives have similar read performance (I believe I misheard that) for practical purposes. The original Google map-reduce was based on commodity hardware and quantity is more important than quality (within reason). For this reason, SATA provides a lot more space for your data. The same amount of space is an order of magnitude more expensive for SAS drives.

The next topic of discussion was the NameNode as the single point of failure. Apparently the MapR system does not use the HDFS, and recovering from a lost NameNode is not as severe as it is for Hadoop. Hadoop 0.20.2 also supposedly introduces sharding, called NameNode federation, where the namespace is divided over several NameNodes.

Hadoop has some issues with certain types of scalability, particularly with the JobTracker. When a large job with a large number of mappers and reducers finish quickly, the TaskTrackers send an influx of messages to the JobTracker and it gets overwhelmed. To prevent users from thrashing a cluster, use a capacity scheduler to put hard caps on queues. There was also some high level discussion of QoS-like functionality among users and sophisticated monitoring of jobs. Map-Reduce NextGen improves scalability by allocating a JobTracker to each individual job whose purpose is solely to monitor resource allocation. The biggest feature Charles would like to see is high availability NameNodes.

Yahoo boasts an impressive 22 clusters each containing between 400 and 4200 nodes. A fellow from AOL indicated that AOL has a cluster of size close to 1000. Is AOL coming back from the dead?

Thank goodness managing a Hadoop cluster is not in my job description…

Session 2: Geodata

I do not get the opportunity to work with geographical data often, so I was curious to see what these folks had to say. The discussion was lead by a fellow named Brian from OSGeo. The largest point that I took away from this talk was that not an incredible amount of thought has been dedicated to Big Geodata, particularly how to store and process it. PostgreSQL and PostGIS are a few ways to store and analyze manageable amounts of data, but not large data. MongoDB is one solution but has its issues. A fellow from Foursquare mentioned that MongoDB cannot shard across geographic data, but I could not hear precisely what he said to that effect. The biggest challenge seems to be a lack of an indexer capable of indexing a large amount of geospatial data aside from the standard RTree implementation. I believe that geodata as well as streaming data and multimedia are some of the biggest unsolved problems in Big Data.

Anyways, on to Hadoop Summit!

As our motivating example, consider we find some really cool graphic on the web and we want to know where it likely originated (i.e. art, a meme). In such a search engine, we could upload the graphic and get results containing the exact image, or images that are very similar such as variations of the image (crop, resize, borders, different effects), modifications of the image (consider Obama-izing [...]]]>

Recently, I have been thinking about alternate ways of specifying search queries other than with text. A couple of weeks ago I came across a piece of music that I could not identify. I thought it would be a huge win for a search engine to allow me to upload this piece, and it would present me with matches, or near matches to other pieces that sound similar, or have similar characteristics. Some services already exist. Shazam allows a user to place a microphone near playing music and it will identify the artist and song. Some uses of search-by-sound:

• Music identification (“solved” – Shazam)
• Music personalizaton and recommendation (“solved” – Pandora)
• Identification of the source of a sound (i.e. a species of bird, a musical instrument, an inanimate object)
• MP3 and media file search
• Finding material that violates copyright

As our motivating example, consider we find some really cool graphic on the web and we want to know where it likely originated (i.e. art, a meme). In such a search engine, we could upload the graphic and get results containing the exact image, or images that are very similar such as variations of the image (crop, resize, borders, different effects), modifications of the image (consider Obama-izing [the campaign logo] someone’s picture), and semantically similar images (different photos of the same object or person). Wouldn’t this be cool? A billion-dollar idea, right?

Well, Google apparently beat me (and millions of others I’m sure) to it with its search-by-image feature on Google Images. I uploaded a photo of myself to see what I would get. We see my school website (where the image originated), as well as several other sites that use my Gravatar. Not too bad.

On the results page, users can also provide some type of labeled data to Google. I am not exactly sure what it is used for yet, but note the text in the search bar: “Describe this image.” Upon entering my name, Google found another photo that looks almost identical to the first one — a variation.

Below are the “visually related” images that were presented to me (before I labeled my photo in the search bar):

I see Steve Jobs (I am honored), but 7 out of 16 images are women, and of the men, we look nothing alike. I know, I know, “visually related” refers to similarity in pixels between images, but I expected more. In these images, we see a lot of red and blue hues.

Let’s try something that will generate many more hits: a popular meme…

The image I uploaded was originally posted on Amazon S3, and is linked to by the above two web pages. Google does a much better job when using a URL rather than uploading an image for obvious reasons. More interestingly, the “visually similar” images show variations and modifications of the same image, based on pixel similarity.

And we get also see web pages containing a copy of the image (not linked to the original S3 file):

But this Isn’t Good Enough Yet!

Google “Search-by-Image” is an awesome first step, and I look forward to seeing more as it is undoubtedly coming. For search-by-image (or search-by-multimedia) to be useful, it must also take “semantic” or conceptual knowledge into account, just like with text search. That is, if I upload a photo of myself, I should get back other photos of myself from various (hopefully authorized) sources. Or, if we upload a photo of the Eiffel Tower, we should get back integrated search results containing other images of the Eiffel Tower as well as text results with information about the Eiffel Tower, and perhaps a tourist’s video or documentary.

One may at first believe that the O RLY search used some semantic knowledge; however, all of the images share a large number of pixels and these images are likely just “visually similar” as stated. Using semantic knowledge, one may see results of other famous owls used in memes in addition to the variations and modifications of the O RLY owl.

All of the data collected by such a system would also provide a hell of a corpus for image and multimedia classification. Researchers could construct classifiers for detecting spammy multimedia, knockoff multimedia (second, third generation grain in images, waveform distortion in audio), pornographic content, as well as augmenting labeled and unlabeled multimedia with metadata. For example, suppose we take a picture of what I think is a rhodendron (inside joke for readers). With such a large corpus, I can upload the photo and have Google (or some other AwesomeSearch) retag the image as that of a hydrangea instead.

Uses of search-by-multimedia with semantic knowledge:

• Cross referencing objects or people on various different sites.
• Product search when textual information (or QR code) is not known
• Catching criminals
• Cataloging media
• Methods for multimedia spam detection
• Geolocation without use of GPS or WiFi, and location search
• Augmentation of metadata and tagging of objects, people, etc.
• Detecting adult, inappropriate or illegal content.
• Identification of actions from images, video or audio and retrieval of related information

Of course, search-by-multimedia poses the same challenges that we face in big data today:

• choosing and boosting the proper features
• collecting a significant and correctly labeled corpus
• fast processing of large datasets with new and existing machine learning algorithms
• efficient indexing and retrieval algorithms to match queries with probably results
• these things are easier said than done, but a lot of fun.

Search-by-multimedia is a very interesting concept and is exciting to think about. In this age of big data and technology, anything will possible. I look forward to the day where anything on the Internet can be found, no matter its content or medium.

To check out Google’s search-by-image, click here and then click on the camera icon.

The CPU/Processor

If you are reading this article, I probably do not need to explain what the CPU/processor does. For high performance computing, you will want to get a CPU that is very “fast” and also has multiple cores. The definition of the word “fast” is in the eyes of the beholder and typically refers to more than just clock speed (GHz). In the constant war between AMD and Intel, I stick with Intel. AMD processors are powerful, but they seem to have [...]]]>

I am usually pretty reserved with cash, but after working full-time for six months, I finally decided to spend some of my money on building a new research development server. This process was long overdue and the reason it took me so long to commit to this project was all of the new technology developed since building my last server. This “new technology” can be pretty confusing unless one specializes in computer architecture. I want to share what I have learned throughout this process, while giving some background. These are only my opinions, and I may be wrong on some things as I am not a hardware expert. I encourage you to read and learn more on your own.

The CPU/Processor

If you are reading this article, I probably do not need to explain what the CPU/processor does. For high performance computing, you will want to get a CPU that is very “fast” and also has multiple cores. The definition of the word “fast” is in the eyes of the beholder and typically refers to more than just clock speed (GHz). In the constant war between AMD and Intel, I stick with Intel. AMD processors are powerful, but they seem to have more of a market with gamers. Intel is my preference, but I have not yet run into anyone that feels strongly towards AMD for high-performance computing (HPC). There are two main processor lines under Intel: standard, and Xeon. Standard processors are your run of the mill CPUs that are found in consumer desktop machines. Xeon processors are designed for non-consumer server, workstation and embedded systems use. I do not consider researchers as “consumers,” we are producers, so the Xeon family is better suited to our needs. On the other hand, you may find that a standard CPU will fit your needs for your particular research or use case. Xeon processors typically have more cache and more multiprocessing capabilities…and they are a lot more expensive. For high-performance computing, I strongly suggest Intel Xeon.

After months of research, I have concluded that multiple Intel Xeon processors are better than one Intel Core i7. As of the time of this writing, it seems that i7 processors cannot be doubled (or tripled etc.) up like Xeons can. Like the AMD, the i7 seems to be favored by gamers and those needing a richer multimedia experience. 

In 2011, most CPUs in new systems have multiple cores. Each core can essentially run one process each. A system with n cores can run n processes simultaneously. Many CPUs are hyperthreading enabled, meaning that each core can actually run 2 threads simultaneously, bringing the total number of threads to 2n. But can’t the system already run multiple processes concurrently? We can run Firefox, TweetDeck, Thunderbird etc. concurrently, right? In practice, it seems that the CPU is processing multiple threads simultaneously. If we could slow down time to the micro level, one would see that the CPU works on one process at a time, then does a context switch to another process. Theoretically, this gives the illusion that the CPU is running multiple processes simultaneously.

While Intel makes great products, its inventory is a nightmare to navigate. There are several things that you must know to ballpark a particular CPU model.

• the model number (the most reliable!)
• the brand name specifies a group of CPU models satisfying similar use cases (Core [i3/i5/i7/i9], Core 2 Duo, Quad Core, Pentium, Xeon).
• the architecture/subarchitecture — specifies a type of processor within a brand, each containing many series (Nehalem, Westmere, Sandy Bridge are common ones these days)
• the chipset (not commonly referred to, examples: Tylersburg, Cougar Point, Panther Point)
• the platform which refers to a set of models (e.g. Harpertown, Jasper Forest, Gainestown, Prescott, Gulftown). Models within a series are typically only differentiated by clock speed (GHz).
• the socket type specifies the shape and size of the CPU. The CPU and the motherboard must have the same socket type (i.e. LGA1366, Socket 775)

As if this is not confusing enough, each Intel Xeon model number is prefixed with a letter for different use cases. The letter distinguishes CPUs with differing thermal dissipation power (TDP). (source)

• W stands for “Workstation” and is meant to be installed in pairs. This designation does not seem very common anymore. They typically run the fastest (clock speed) and the hottest. They require significant cooling.
• E is “mainstream (rack mount)” and the standard model of CPU. Although it is “standard,” there is nothing wrong with it performancewise, but will run hot even when idle.
• X stands for “performance” and are similar to E but provide for extra overclocking capabilities and have lower idle power draw.
• L stands for “power optimized” and are low voltage CPUs (60W or less) that are typically only used for data centers or rack servers. They typically do not come in the higher clock speeds etc.

For the Intel Xeon, model numbers indicate what configuration it is compatible with on the motherboard (source):

• 3xxx Xeons are designed to be used by themselves, as the only CPU on the motherboard.
• 5xxx Xeons are designed to be used in pairs; two CPUs on the motherboard.
• 7xxx Xeons are designed to be used in pairs, or in larger groups.

The 2 CPUs that I purchased are model Intel Xeon E5645. The Intel Xeon E5645 is part of the Gulftown platform of the Xeon family. It uses the Westmere subarchiture which is the 32 nm shrink of the Nehalem architecture spec and connects to the system bus using socket LGA1366. (This is the same architecture used for the i7-9xx series to make it more confusing) The E means that it is a “mainstream” CPU. Since it is a 5000 model, it is installed with another identical CPU on the same board.

The number of cores is important. Most chips in current desktops contain 2 or 4 cores. Higher end systems and servers may have 6, 8 or 10 cores per chip. Xeons with 8 and 10 cores per unit debuted in Q2 of 2011 and are very expensive (about $2000 for 8 cores). They also require a brand new socket type (LGA1367), which means a new, expensive motherboard. A CPU with more cores allows an application to perform several units of work per task; these processors allow higher bandwidth.

The clock speed (GHz) used to be the deciding factor for most people, until Moore’s Law broke down. Higher clock speed possibly allows a single process to complete faster. Since games typically use a limited number of threads and require quick performance, a single i7 is a good choice. The i7 has multiple cores, and also has a very high clock speed.

The cache size and speed is also important. The cache allows very high speed access to memory locations that are frequently accessed by copying the data from RAM into the CPU cache. Modern systems typically have three levels of cache: L1, L2 and L3. L1 cache is said to be the “closest” to the CPU, meaning the CPU queries the L1 cache first when performing a memory access. The L1 cache is the smallest. The L2 and L3 caches are accessed next in order, and L3 cache is larger than L2 cache. Very simply put, CPUs with larger caches (especially L1) are better.

Newer processors report CPU throughput as gigatransfers per second (GT/sec) which, like GHz, quantifies some measure of “speed.” Using GT/s, one can compute the number of bits the CPU can transfer per second as

Think of the cores vs. clock speed decision as a highway. Suppose the clock speed indicates the maximum speed limit on a single lane highway. A faster CPU corresponds to a single lane highway with a high speed limit. You will get to your destination faster. On the other hand, consider a one-lane vs. a two-lane highway, both with identical speed limits. If one lane is too busy for you, take the other lane. An increase in the number of cores increases the number of choices of lanes you can transition to. On the single-lane highway, you would need to slow down and wait for the cars in front you to move forward. By switching lanes, you may get to your destination faster, or you may not, but more driving is completed overall. 

Some time over the past 6 weeks I randomly saw a tweet announcing the “Data Scientist Summit” and shortly below it I saw that it would be held in Las Vegas at the Venetian. Being a Data Scientist myself is reason enough to not pass up this opportunity, but Vegas definitely sweetens the deal! On Wednesday I woke up at 6am to partake on the 5.5 hour voyage to Las Vegas.

The Pre-Party

The Venetian and all close hotels were booked, so I ended up at the Aria; a new experience. The hotel is beautiful and very ritzy. I had heard that the rooms were very technologically advanced but I wasn’t prepared for the recorded welcome message, music and automatic shades opening upon entry to the room. The Aria is a geek’s paradise. Everything is computerized. Key cards are “waved” rather than swiped, lights are turned on/off and dimmed by use case (“sleep”, “read” etc.), rather than manually. There are no paper “Do Not Disturb” signs; rather, a switch on the wall (or via TV) toggles an indicator light outside the door. And the best part… Internet is FREE!

Elastic Compute Cloud (EC2) is a service provided a Amazon Web Services that allows users to leverage computing power without the need to build and maintain servers, or spend money on special hardware. The idea is simple, the user “boots” up one or more machines and then accesses those machines as if they were logged into any other machine remotely. I used EC2 and Elastic MapReduce extensively for my M.S. thesis last spring, but mainly used its large memory capabilities rather than its potential for explicit parallelism.

Recently, I ran a crawling job on EC2 using a parellel crawler I wrote in Python with twill. Using EC2 poses its own challenges. Using parallel code poses more challenges. Combining these two facts with the fact that crawling is I/O bound can create some more interesting challenges. If you have taken a course in operating systems, you have heard this stuff over and over again. So have I, but I am stubborn. I tend to learn lessons from experience, and this was no exception. Through this series of posts, I want to point out difficulties and “gotchas” that are important to keep in mind when using EC2, and in this post, with [...]]]>

Elastic Compute Cloud (EC2) is a service provided a Amazon Web Services that allows users to leverage computing power without the need to build and maintain servers, or spend money on special hardware. The idea is simple, the user “boots” up one or more machines and then accesses those machines as if they were logged into any other machine remotely. I used EC2 and Elastic MapReduce extensively for my M.S. thesis last spring, but mainly used its large memory capabilities rather than its potential for explicit parallelism.

Recently, I ran a crawling job on EC2 using a parellel crawler I wrote in Python with twill. Using EC2 poses its own challenges. Using parallel code poses more challenges. Combining these two facts with the fact that crawling is I/O bound can create some more interesting challenges. If you have taken a course in operating systems, you have heard this stuff over and over again. So have I, but I am stubborn. I tend to learn lessons from experience, and this was no exception. Through this series of posts, I want to point out difficulties and “gotchas” that are important to keep in mind when using EC2, and in this post, with using parallelism in your code to accomplish large tasks.

Monitor your Instances

Monitoring your instances has two important benefits. First, to make sure that you are not maxing out resources on the machine. EC2 is “elastic.” With some clever programming, you can boot up more machines if you notice resources becoming scarce on your current machines, and then decommission them later when they are not needed.  I did not do this at first, and I ran into several issues.

Disk Space. The concept of a “disk” is very confusing in EC2. The AMI forms a disk, sort of. Above and beyond the OS and any other software and packages you may install as part of the AMI, you can use whatever free space is remaining to store output files. The total disk space used by the AMI seems to be configured at the moment the AMI is constructed. Thus, it is not a good idea to store files in the instance. I did this. Fortunately, I found out before it was too late that my “disk” was filling up. I wrote a cron job to copy all of my output files to /mnt every five minutes. Use /mnt to store your files as it has lots and lots of space; HOWEVER if you terminate your instance, the files are gone. This is still true if you use space within the instance. Once your job completes, upload your files to S3. s3cmd allows access to S3 from the command line, and with the modification here, you can upload and download files in parallel (a life saver for big batches). Another option is to create an EBS volume, mount it, and write files directly to the EBS volume. EBS space is much more expensive than S3 space.

Memory. On my first attempt, I maxed out memory to the point that the OS killed 6 of my 8 processes. This caused a huge blow in the performance of my crawler and rendered the extra money I spent on an extra large instance wasted. Monitor your job’s memory using top. If memory usage seems to grow too fast to your liking, consider using a memory profiler to make sure that there are no memory leaks in your code. I have found that long running Python processes eat up a lot of RAM, even if there are no explicit growths of data structures.

Additionally, maxing out RAM means that the disk will begin to swap. This is devastating to performance because this extra grinding of the disk decreases the total I/O throughput your job can handle. This is crucial for crawlers as files need to be written to disk quickly.

If after profiling you find that your job is still using too much RAM, consider caching, or using a high memory EC2 instance.

I/O Throughput. How fast your job consumes and produces data is a good way to determine if something is going amiss in your job, or with the other resources you are using. When I started my crawling job, I was crawling n pages per hour, but after twelve hours, the throughout decreased exponentially until it got so slow that I had to add more instances. One way to monitor throughput is to save the results of ls -latr –full-time to disk and extract the date/time of each file. Using a tool like R, you can quickly plot your I/O throughput over time using an aggregate(). A decrease in I/O throughput can be the result of many things: 1) swapping from exhausting RAM, 2) low disk space, 3) network congestion within AWS, 4) poor resource performance (if crawling, the resource would be the website being crawled), 5) hammering an external resource and/or HTTP throttling, 6) congestion in the Internet. For crawling, you may want to consider using several smaller instances rather than fewer larger instances. This way, you will be accessing the resource from many IPs and the result of being throttled should be lessened. Additionally, use instances that have “High” I/O performance; some are rated “Moderate” or “Low.” 

CPU. A general rule of thumb is that you can run n processes in parallel, for n cores. Additionally, if each core supports hyperthreading then the number of processes you can run is approximately 2n. If you run more than the suggested number, the price of context switching can slow down your performance. If you find the need to routinely exceed this guideline, use an instance with more cores.

When running parallel code, routinely do a ps aux | grep processName to make sure the correct number of processes is running. If any were killed, this will be noted in /var/syslog with a reason.

Financial metrics. Are you getting your money’s worth? Are you really using all of the cores you are paying for? Are you really using all of the memory you are paying for? This is up to you and your budget to dictate. But do not get carried away and assume that you must stay with the same instance size. Most AMIs can run on different instance types (except 64bit AMIs are restricted to m1.large and bigger).

Quarantine Essential Services

My crawler used Redis as a work queue. Each process could easily write new thread IDs and page numbers to the queue as they are discovered, and read thread IDs and page numbers from the queue as each process is ready to crawl a page. One problem that I faced was that I coupled the crawling operation with queue management into the same script, and ran the Redis server on a server where a crawler was running. This coupling posed two challenges. First, it can sustain nasty bugs. Whenever a process was created on the master Redis node, my code would wipe the Redis queue clean to prepare it for crawling (bug!). Flushing the queue, and the initial population of the queue should have taken place in two separate scripts. Due to my major bug, I wiped the entire queue clean in the middle of the crawl. Fortunately, I followed the advice in the next section.

Second, I had to be careful that my processes did not exceed RAM limitations. Because Redis is mainly an in-memory key-value store, it itself can hog up most of the RAM in the instance. For this reason, it is best to quarantine essential services such as queues to their own instances.

Document Everything

Log everything. Log every resource you are going to use (URLs for a crawl) and log everything that was done and any problems that arise. Using the directory structure (ls) as well as a log of what work was already performed, I was able to reconstruct and repopulate the work queue and essentially start where I left off. For my crawling operation, I wrote the following events to logs, each with a timestamp.

• Starting the crawl.
• Logging in to the site being crawled.
• Clearing and populating the queue.
• Visiting a thread’s first page.
• Discovering the number of pages of posts in the thread/inserting to the queue.
• HTTP redirects, when a thread has been moved.
• Visiting a thread ID that does not exist.
• Inadvertent logouts, marking work to be redone.
• Queuing inconsistencies.

An activity log verbosely documented everything that occurred without logging actual data. An inventory manifest indicated which URLs/forum posts had valid content and how many pages of content were associated with them. A standard directory listing indicated what work had been done. By cross referencing the manifest and the directory listing, it is easy to see which posts had not yet been processed. A system log prepared by the operating system also documents critical failures for you, such as lack of disk space or processes being killed.

When writing your logs, use the advice in the next section!

Take Care to not Clobber Files and Objects

It’s been said over and over again. Each process should hold as much of its own real estate as possible. When two or more processes write to the same object, corruption can occur unless there is a locking mechanism in place. If two processes write to the same file at the same time, you will notice garbled entries in your logs. This did not affect my crawled data because each file was written by a single process. The same can be true for reading data as well. When spawning multiple processes, I shared the same Redis connection with all of the processes. If two processes read from the queue at the same time, one process would get the correct data (a thread ID and page number) and the other would get “OK”, which was the result of the first process’ fetch operation. This is mostly my fault, but partially redis-py‘s fault for filling some buffer between Python and Redis with meaningless information (“OK”).

Each process should write is own log files. When opening a file, you can use the following:

import os
 OUT = open("mylogfile-%s.log" % str(os.getpid()), "w")
 ...
 OUT.close()

Crawler Specific: Set an Upper Bound

Crawling is fun, but you must practice moderation or it is easy to attempt to boil the ocean. When I first started, I would run a crawl, have it crash, and then deem the data out of date and start over from the beginning and crawl until there was nothing possible left to crawl. It is good to set an upper bound: “I will crawl 10 days worth of data”, or “I will only use threads created prior to May 1, 2011.”

One of the keys to success with EC2 is to get over the penny pinching. If you have a project, just take the plunge and do it on EC2 (if required). The amount you spend on the first few projects will save you more on future projects.

I ditched my iPhone this past December (good riddance) in favor of the Droid X (Android). I figured, on such an open source OS, Google must be doing the same thing. After surfing through Hacker News, it turns out I was right.

Compared to the iPhone though, getting the data on an Android phone is not simple.

One wrinkle with NumPy/SciPy that needs to be ironed out is the difficulty of installation on certain OSes, and particularly, architectures.The SciPy SuperPack has done a good job of taking care of this issue, but it has not yet been updated for 2.7.1 and manually hacking away at its script has not worked for me.

I cannot take credit for the instructions in this article. A brave warrior, Jeremy Conlin, somehow managed to figure out how to install 64-bit NumPy and SciPy, with 64-bit Python 2.7.1 on Snow Leopard; he posted the directions to the SciPy User mailing list on February 24. I followed the directions, and miraculously they worked. I am reproducing them here for Google bait.

Install Python 2.7.1

1. Download the universal Mac 2.7.1 installer here (Python 2.7.1 Mac OS X 64-bit/32-bit x86-64/i386 Installer). Typically, Python will be installed to /Library/Frameworks/Python.framework/Versions/2.7/, but may be in other locations.

2. Verify that your new version of Python is 64-bit enabled. Note: Python installations typically do not get toggled as the default Python, so find the location of the 2.7.1 Python executable. On my machine, it is /Library/Frameworks/Python.framework/Versions/2.7/bin/python. python2.7 should also work.

Load [...]]]>

Numpy and SciPy are packages for numerical computation and scientific computing, for Python.

One wrinkle with NumPy/SciPy that needs to be ironed out is the difficulty of installation on certain OSes, and particularly, architectures.The SciPy SuperPack has done a good job of taking care of this issue, but it has not yet been updated for 2.7.1 and manually hacking away at its script has not worked for me.

I cannot take credit for the instructions in this article. A brave warrior, Jeremy Conlin, somehow managed to figure out how to install 64-bit NumPy and SciPy, with 64-bit Python 2.7.1 on Snow Leopard; he posted the directions to the SciPy User mailing list on February 24. I followed the directions, and miraculously they worked. I am reproducing them here for Google bait.

Install Python 2.7.1

1. Download the universal Mac 2.7.1 installer here (Python 2.7.1 Mac OS X 64-bit/32-bit x86-64/i386 Installer). Typically, Python will be installed to /Library/Frameworks/Python.framework/Versions/2.7/, but may be in other locations.

2. Verify that your new version of Python is 64-bit enabled. Note: Python installations typically do not get toggled as the default Python, so find the location of the 2.7.1 Python executable. On my machine, it is /Library/Frameworks/Python.framework/Versions/2.7/bin/python. python2.7 should also work.

Load Python 2.7.1 and execute the code below. If you get 64, then you are ready to proceed.

 import sys
 from math import log
 log(sys.maxsize, 2) + 1
 

Another way is to execute the following. If you get 1099511627776 (and NOT 1099511627776L) you are in good shape.

 2**40
 

(This tip comes from Aaron Meurer’s SymPy blog)

Install gfortran

1. Download gfortran-4.2.3 here and double-click the file to mount the disk image. 

2. Create a temporary directory (I call it tmp here) and run the following commands.

 cd tmp
 mkdir gfortran
 cd gfortran
 pax -zrvf /Volumes/GNU\ Fortran\ 4.2.3/gfortran.pkg/Contents/Archive.pax.gz .
 cp -r usr/local/* /usr/local
 

Drop Down to the Root Shell

NOTE: From this point forward, I dropped to the root command prompt (sudo su -) so that I had full control over the environment. This is mainly due to the way to the shell deals with PYTHONPATH and the complications that can stem from it when installing with root privileges.

Locate the Python 2.7.1 distribution and find a directory that ends in site_packages. For example, for me this is /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/.

Assuming Bash is your shell of choice, enter the following

export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/

Or, for tcsh,

setenv PYTHONPATH /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/

Install distribute-0.6.14

The next few steps follow the typical Python package installation method. I use python2.7 rather than python just to make sure that my system is using 2.7.1 and not built in 2.6.1.

Distribute adds tools for installing Python modules including pip and easy_install.

 curl -O http://pypi.python.org/packages/source/d/distribute/distribute-0.6.15.tar.gz
 tar -xzvf distribute-0.6.15.tar.gz
 cd distribute-0.6.15
 python2.7 setup.py install
 

Install nose

Nose is al alternate test discovery and running process for unittest, similar to py.test.

Installation requires easy_install. I am not a fan of easy_install in the slightest. There are two important things to remember here:

1. PYTHONPATH must be set correctly, or easy_install will complain.
2. If you have easy_install for another version of Python, make sure you use the version for 2.7.1 and not some previous version or easy_install will complain.

On my system, easy_install for 2.7.1 is located at /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install so I use:

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install nose
 

Install NumPy

Finally, we can install NumPy. Version 1.5.1 is the first version that apparently works with Python 3.

 curl -O http://downloads.sourceforge.net/project/numpy/NumPy/1.5.1/numpy-1.5.1.tar.gz
 tar xzvf numpy-1.5.1.tar.gz
 cd numpy-1.5.1
 sudo python2.7 setup.py install
 

Then, open Python and execute the following to import the library, and test it. On my system, all tests passed.

 import numpy
 numpy.test()
 

Install SciPy

Next, install SciPy. Version 0.9 is the first version that is compatible with Python 3.

 curl -O http://downloads.sourceforge.net/project/scipy/scipy/0.9.0/scipy-0.9.0.tar.gz
 tar xzvf scipy-0.9.0.tar.gz
 cd scipy-0.9.0
 sudo python2.7 setup.py install
 

Then, open Python and execute the following to import the library, and test it. On my system, all but 14 tests passed.

 import scipy
 scipy.test()
 

Install readline

readline provides functions for completion and reading/writing of history files from the Python interpreter (up/down arrow).

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install readline
 

Install IPython

IPython provides an enhanced command line interface to the Python interpreter. It is much more pleasant to work with than the standard command line interface.

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install ipython
 

Install wxPython

wxPython is a graphical user interface toolkit for Python. Up until recently, this was the blocker in the process — there was no 64-bit version of wxPython for Mac. Download the DMG here and double click the installer. Versions for Cocoa and Carbon are provided, but Python 2.7.1 apparently requires the Cocoa version.

Install matplotlib

matplotlib provides the plotting interface which is crucial to SciPy. Installing matplotlib is always the most complicated part of the process.

 wget http://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1.tar.gz
 tar xzvf matplotlib-1.0.1.tar.gz
 cd matplotlib-1.0.1
 

Then, open the file make.osx, find lines that look like below, and make the appropriate changes

 PYVERSION=2.7
 ZLIBVERSION=1.2.5
 PNGVERSION=1.4.5
 FREETYPEVERSION=2.4.4
 

Finally, delete line 63. Line 63 looks like:

 cp .libs/libpng.a . &&\
 

Within the matplotlib-1.0.1 directory, download the following archives. Do not extract them!

1. zlib 1.2.5
2. libpng 1.4.5
3. freetype 2.4.4

Or execute the following commands within the matplotlib-1.0.1 directory.

 wget http://zlib.net/zlib-1.2.5.tar.gz
 wget http://downloads.sourceforge.net/project/libpng/libpng14/older-releases/1.4.5/libpng-1.4.5.tar.gz
 wget http://download.savannah.gnu.org/releases/freetype/freetype-2.4.4.tar.bz2
 

Finally, install matplotlib-1.0.1 using the following command

 PREFIX=$PYTHONROOT make -f make.osx deps mpl_install
 

where PYTHONROOT is the directory containing the pacakage tree. On my system, it is /Library/Frameworks/Python.framework/Versions/2.7/.

Finally, enter Python and test matplotlib.

 import matplotlib.pyplot as pyplot
 pyplot.plot([1,2,3])
 

If installation was successful, a plot like the following should appear.

Hopefully this is helpful to someone, and here’s to hoping a SciPy Superpack for Python 2.7.1 will be released soon!

RStudio plays to this analogy very well, and makes modifications where appropriate. RStudio provides many features that are lacking in the standard R GUI, and improves on features that do not work properly in the Windows R GUI. Over the past few days, I have been doing all of my R analysis within RStudio, shortly with the Desktop version, and mostly with the Server version. I will discuss mostly the server version [...]]]>

As most readers are probably aware, the free IDE for R, called RStudio, was recently released for general use and it immediately made huge waves within the R community. IDE stands for Integrated Development Environment. IDEs typically provides a rich set tools developing in some target language. For standard programming languages like C++ (VisualStudio) and Java (Eclipse or NetBeans), IDEs contain:

• an editor tailored to the target language. The editor typically has tab/auto-complete for variable names, functions and class methods and properties and also features syntax highlighting.
• a multiple document interface (MDI) where there may be several documents opened in different tabs.
• a window that interacts with the compiler, or a panel containing the console to the language, a la MATLAB, and even vanilla R’s GUI.
• a debugger
• a file browser and language reference.

RStudio plays to this analogy very well, and makes modifications where appropriate. RStudio provides many features that are lacking in the standard R GUI, and improves on features that do not work properly in the Windows R GUI. Over the past few days, I have been doing all of my R analysis within RStudio, shortly with the Desktop version, and mostly with the Server version. I will discuss mostly the server version since that is what I have been using. It is identical (AFAIK) to the desktop version, so you are not missing anything by using either version.

RStudio Server

First, installation is miraculously easy. I only had a few very minor glitches to deal with. Armed with sudo access to a machine on a research cluster at work, I was able to simply download the RPM and install it using the instructions provided on the web site. Then, all I had to do was fire up a browser and go to

http://servername.com:8787

and I was asked for my login credentials. But I couldn’t get in. This server authenticates using LDAP, but all I had to do was replace the contents of /etc/pam.d/rstudio with the contents or /etc/pam.d/login and I was able to login. But then there was a “unknown error.” Oh, the version of R that was installed was too old (2.8). I just did a yum upgrade R, and RStudio logged me in with no problems. What showed up on my browser screen was beautiful! It looked identical to the desktop version of RStudio.

Once logged in, I somehow have access to ALL of my files on the remote server. I can load my data (typically produced by Hadoop) already residing on the server, and I can save output, graphs, data and even the R session itself on the remote server! All while just clicking buttons. No commands to remember, no screwed up PDF files, and most importantly…. no scping files back and forth from the server just to create a plot (X worked well, but had limitations)!

Things I Love about R Studio

I will have to go panel by panel, but even then I will have missed cool features. I also will not discuss features that are already present in the MacOS X R GUI and are repeated and beautified in RStudio:
The R command prompt still looks the same. At first, my reaction was “Damn, what am I supposed to do?” But when the GUI finished loading, the familiar R command prompt appeared in all is 1970ish glory. I immediately started typing commands and seeing fields in the other panes populate and change to display different usages. It left me with a “oh, I see” feeling.

Saves R sessions correctly, and when I return to RStudio, ALL of my work is there! I could never get the save session/image function to work in R GUI. I gave up several years ago. In RStudio, it works properly, but you don’t even need it because… when you leave RStudio and then return, everything is there! The workspace (variables, functions, data, etc), the scripts you were working on, the plots, even the last dang help screen you looked at!

The Stop Execution button in the console actually works. When executing a long running computation in R GUI (that’s the first mistake), it is sometimes necessary to cancel the computation either because I made an error, or because the computation is killing my system’s performance. In R GUI, particularly on MacOS X, the Stop Execution button did absolutely nothing, because there was typically a spinning beachball preventing me from clicking it. Hitting ESC also did not work. In RStudio, clicking Stop actually seems to break out of the madness.

Workspace panel. The workspace panel displays the variables, functions, data frames and other objects that reside in the current workspace, a la MATLAB. From this panel, one can also switch or save workspaces. The user can also import a dataset from a text file using a trivial wizard (a la SPSS, etc.), or from a web URL. The user can also clear the workspace. A frequently overlooked command to do the same from the command line is rm(list=ls()), but that is no longer necessary to remember!

Clicking on a data frame object in the workspace pane, causes it to be displayed in a nice tabular format. It can also be printed to a local printer, or opened in a new window.

Clicking on a numerical value allows the user to change it by opening an in-place edit box. Clicking on other objects like lists, vectors and functions opens an edit window displaying the definition that created it.

Files panel. There is nothing really exciting to see here, except that by clicking the Upload button, I can upload files directly to the remote server just by selecting the file, without having to SCP!

Scripting panel. This is the second best feature of R studio and has the same feeling as the stock script editor that ships with R. The largest difference is that the editor in RStudio is stable. On MacOS X, the editor tends to garble 2-3 rows of code together on every single scroll. This editor does a better job of indentation than R GUI. When opening a function, R GUI tends to indent the body properly, but insert a closing } prematurely. RStudio’s editor also features auto-completion, a feature present in the command-line of R GUI and R, but not in the editor of R GUI. The user can also save their script on the remote server, print code to a local printer and search. Similar to MATLAB, the user can select one or more lines of code and run them by clicking the “Run Line(s)” button, rather than having to copy and paste lines. “Run All” is a point-and-click replacement for source.

The “Source on Save” function is interesting. If enabled, RStudio will run/source the script each time the script is saved. Honestly, I do not find this feature to be all that useful unless in the middle of debugging, and dangerous if not debugging. Suppose after a long 10-fold-cross-validation computation there is an error that we want to fix. We fix the error and save the script. Do we really want to run the computation again? If R were a compiled language, then yes. Since R is not a compiled language, this feature is not entirely useful in concept.

The “magic wand” icon contains what I suspect to be a growing collection of coding tools. Currently, the user can comment and uncomment a bunch of lines at once. This is particularly useful since, for some reason, there is no multiline comment flag in R. The user can also select a series of lines and wrap a function around them. This feature could be dangerous for those not familiar with coding but provides a very nice way to put a bunch of code into a function as an afterthought.

Plot panel. By far my favorite part of RStudio is the plot panel! All plots are saved in this panel, and the user move back and forth among plots that were already constructed. The Export button allows exporting a plot to user defined dimensions and save to the local machine as a PNG, or even copy it to the local machine’s clipboard! Of course, the PDF button produces a PDF file of the plot that can be saved on the local machine. If the plots are all too much, we can click “Clear All” and start again with a clean slate.

But, is it possible to create plots of larger size? I am sure there is, but I did not spend much time looking.

LaTeX and Sweave documents. From the File menu the user can create new documents including LaTeX and Sweave. Unfortunately, I cannot experiment more with these features because there is something amiss in my configuration. For students and researchers, having Sweave and LaTeX integrated with RStudio is a huge, huge, huge advantage. No longer must we copy/paste among different programs. To make the integration complete, BibTeX, Asymptote/TikZ/gnuplot whatever should be easily included by the user.

At any point if the user interface shows stale data, there is a Reload button to help you out by refreshing the entire RStudio interface.

Things that Need Improvement

I do not really have any complaints about RStudio, quite the opposite actually. However, there are some things that do not seem to work. I should note however, that I have not spent much (well, any) time debugging them. The developers are probably already working on some of them. Some of them are probably problems in my configuration and others are probably settings that I need to tweak.

No auto-completion of parentheses or quotation marks. This is a bummer, but not a deal breaker. On the other hand, as you type closing marks, RStudio highlights the matching mark.

The dataset view needs work. Columns can’t be resized. Other natural functionalities that seem to be missing are: column renaming (a call to names), cannot sort or order values by a column, and data manipulation (I didn’t say that). These missing features are a tad disappointing, but a hell of a lot better than displaying in the terminal.

Install packages in the packages panel does not work on our server’s configuration.

LaTeX cannot be found. Upon attempting to create a new LaTeX or Sweave document, I got a friendly notice (instead of a bizarre error message) saying that LaTeX is not installed. The problem is, it is installed and there does not seem to be anywhere in the GUI to configure its location. Additionally, some LaTeX templates would be useful.

In Conclusion…

My Workflow Before and After RStudio

Before RStudio



After RStudio

The developers of this open source project seemed to get it right on the first try. How the hell is that possible??? So has anyone switched from the big R to the big blue ball?

Web mining requires a lot more sophistication. The programmer in many cases does not know the full formatting of the data published on a web site. Additionally, this format may change over time. There are certain standards that do apply to certain types of data on the web, but one cannot rely on web developers to follow these standards. For example, the RSS Advisory Board developed a convention for the formatting of web pages so that browsers can automatically discover the links to the site’s RSS feeds. I have found in my research that approximately 95% of my sample actually implemented this convention. Not bad, but not perfect.

Always Have a Plan B, C, D, …

One might say that 95% is good enough. I am a bit obsessive when it comes to data quality, so I wanted to extract a feed for 99% of the sites I had on my list. Also, I am always leery [...]]]>

Programming defensively requires knowing the input that your code should be able to handle. Typically, the programmer may be intimately familiar with the type of data that his/her code will encounter and can perform checks and catch exceptions with respect to the format of the data.

Web mining requires a lot more sophistication. The programmer in many cases does not know the full formatting of the data published on a web site. Additionally, this format may change over time. There are certain standards that do apply to certain types of data on the web, but one cannot rely on web developers to follow these standards. For example, the RSS Advisory Board developed a convention for the formatting of web pages so that browsers can automatically discover the links to the site’s RSS feeds. I have found in my research that approximately 95% of my sample actually implemented this convention. Not bad, but not perfect.

Always Have a Plan B, C, D, …

One might say that 95% is good enough. I am a bit obsessive when it comes to data quality, so I wanted to extract a feed for 99% of the sites I had on my list. Also, I am always leery of bias. Could there be something special about these sites that do not implement RSS autodiscovery? Clearly, there are exceptions to my Plan A. So time to move to Plan B. I found that some of the sites in this 5% used FeedBurner to index their feeds, so Plan B was to use a regular expression to extract FeedBurner URLs. This only added another 1% (actually less than that) to my coverage.

((?:http|feed)://feeds.feedburner.com/.*?)

Next, Plan C took the domain name and simply slapped /feed to it and hope it sticks. I called this process “feed probing” and it added the remaining 3% that I was looking for. If Plans A, B and C all failed to find a suitable RSS feed, all hope is lost and we just skip this site (1% error).

On the other hand, there are times when it is the HTTP client or server cannot be trusted…

Common Python Exceptions in Web Mining

It is all too common to encounter an exception while web mining or crawling. Code must handle these errors gracefully by catching exceptions or failing without aborting. One method that works well is to provide a resume mechanism that restarts execution where the code left off, rather than having to start a multiple hour/day/week job over again! Below is a taxonomy of common problems (and their Python exceptions):

HTTP Errors. These occur frequently. Some are recoverable, and others are worth just throwing out a record over. The most common ones are below, but for more information, refer to RFC 2616.

• 404 Not Found: the web page you tried to download could not be found. Skip the record.
• 400 Bad Request: the server has deemed the client’s HTTP request as malformed. Either retry, or double check your code!
• 401 Unauthorized: authentication is required before proceeding. Either skip the record, or add authentication to your code.
• 403 Forbidden: you are trying to access something you are not allowed to access. This is a common HTTP error thrown when your program is being rate limited.
• 500 Internal Server Error: something on the server end is wrong. Either try again immediately, or wait a while and retry the request.
• 3xx Redirect: the web page you are trying to access has moved somewhere else. In my research, these are rare but common in practice. I choose to skip these sites. You may wish not to.

In Python, these can be caught as urllib2.HTTPError. It is also possible to specify actions based on the specific HTTP error code returned:

try:
    content = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
    if e.code == 404:
        print "Not Found"
    elif e.code == 500:
        print "Internal Server Error"

Server Errors “URLError”. These occur frequently as well and seem to denote some sort of server or connection trouble, such as “Connection refused” or site does not exist. Usually, these are resolved by retrying the fetch. In Python, it is very important to note that HTTPError is a subclass of URLError, so when handling both exceptions distinctly, HTTPError must be caught first.

try:
    content = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
    ...
except urllib2.URLError, f:
    print f.reason

Other Bizarreness. The web is very chaotic. Sometimes weird stuff happens. The rare, elusive httplib.BadStatusLine exception technically means that the server returned an error code that the client does not understand, but it can also be thrown when the page being fetched is blank. On a recent project, I ran into a new one: httplib.IncompleteRead which has little documentation. Both of these issues can usually be resolved by retrying the fetch. Both of these pesky errors (and more) can be handled by simply catching their parent exception: httplib.HTTPException.

try:
    content = urllib2.urlopen(url).read()
except httplib.HTTPException:
    #you've encountered a rare beast. You win a prize!

Everything Deserves a Second Chance

One common reaction to any error is to just throw the record out. URLErrors errors are so common, that it is probably unwise to do that if you are using the data for something. Typically, these errors go away if you try again. I use the following loop to catch errors and react appropriately.

attempt = 0
while attempt < HTTP_RETRIES:
    attempt += 1
    try:
        temp = urllib2.urlopen(url).read()
        break
    except urllib2.HTTPError:
        break
    except urllib2.URLError:
        continue
    except httplib.HTTPException:
        continue
else:
    continue

This code attempts to fetch URL a maximum of HTTP_RETRIES times. If the fetch is successful, Python breaks out of the loop. If a URLError or HTTPException occurs, we move on to another attempt of the fetch. If we encounter an HTTP error (not found, restricted etc), give up. Depending on the error, we can modify the code to retry on certain errors, and abort on others, but for my purposes, I do not care.

The Comatose Crawler

If you have ever done a large scale crawl on a web site, you are bound to encounter a state where your crawler becomes comatose – it is running, maybe using system resources, but is not outputting anything or reporting progress. It looks like an infinite no-op loop. I have encountered this problem since I started doing web mining in 2006 and did not, until just this past weekend, realize exactly why it was happening and how to prevent it.

Your crawler has sunk in a swamp, and is essentially trapped. For whatever reason, the HTTP server your code is communicating with maintains an open connection, but sends no data. I suppose this could be a deadlock-type situation where the HTTP server is waiting for an additional request (?), and the crawler is waiting for output from the HTTP server. It was my misunderstanding that the HTTP protocol had a built-in timeout, and I was relying on it. This is apparently not the case. There is a simple way to avoid this swamp, by setting a timeout on the socket sending the HTTP request:

import socket
...
HTTP_TIMEOUT = 5
socket.setdefaulttimeout(HTTP_TIMEOUT)
...
handle = urllib2.urlopen("http://www.google.com")
content = handle.read()
...

If a request to the socket goes unanswered after HTTP_TIMEOUT seconds, Python throws a urllib2.URLError exception that can be caught. In my code, I just skip these troublemakers.

Traceback (most recent call last):
  File "", line 1, in ?
  File "/usr/lib64/python2.4/urllib2.py", line 130, in urlopen
    return _opener.open(url, data)
  File "/usr/lib64/python2.4/urllib2.py", line 358, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.4/urllib2.py", line 376, in _open
    '_open', req)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 1021, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.4/urllib2.py", line 996, in do_open
    raise URLError(err)
urllib2.URLError: urlopen error="" timed="" out=""

With enough experience, dedication, blood, sweat, tears, and caffeine, data mining the jungle known as the World Wide Web becomes both simple and fun. Happy web mining!

Some of the ones that I frequently read, or are written by Twitter friends/followers (in no particular order):

When I accepted this position, one of my hopes/expectations would be to become professionally competent and confident in C, Java, Python, Hadoop, and the software development process rather than relying on hobby and academic knowledge. That is something a degree cannot help with. It has been a great experience, although very frustrating, but that is expected when jumping into development professionally.     

I am writing this post to chronicle what I have learned about using Hadoop in production and how it majorly differs from its use in my research and personal analysis. To start, I was asked to check out a huge stack of code from a Subversion repository. But then what?

But you’re a Computer Scientist! This should be easy!

The first part is true, but there is a stark difference between a garden variety computer scientist and one that converts from [...]]]>

Wow. I can’t believe it has been a month since I have posted. On December 1, I started a new chapter in my life, working full time as a Data Scientist at the Rubicon Project. Needless to say, that has been keeping me occupied, as well as thinking about working on my dissertation. For the time, I am getting settled in here.

When I accepted this position, one of my hopes/expectations would be to become professionally competent and confident in C, Java, Python, Hadoop, and the software development process rather than relying on hobby and academic knowledge. That is something a degree cannot help with. It has been a great experience, although very frustrating, but that is expected when jumping into development professionally.
    

I am writing this post to chronicle what I have learned about using Hadoop in production and how it majorly differs from its use in my research and personal analysis.
To start, I was asked to check out a huge stack of code from a Subversion repository. But then what?

But you’re a Computer Scientist! This should be easy!

The first part is true, but there is a stark difference between a garden variety computer scientist and one that converts from another field. Homegrown developers typically have an undergraduate degree in Computer Science (mine is close, but not purely CS). They have a strict and challenging curriculum of coursework, and their lives are peppered with summer internships. In these experiences, they are seasoned with professional work experience that they were not expected to have beforehand. Once they graduate, they have the skills necessary to work as a software engineer.

“Converts” typically have a love for Computer Science, but typically hold undergraduate degrees in related fields. Mine was Mathematics of Computation and Statistics. The Statistics B.S. was totally irrelevant. The Math of Computation B.S. was relevant in discovering my true interest, however, my life was consumed with writing proofs and solving puzzles. I only programmed for fun and to solve problems that I faced in daily life. Shortly after starting a Ph.D. in Statistics, I discovered that I wanted a career change to Computer Science and software engineering. Although I was subjected to the same curriculum as the undergraduates, it was rushed and did not provide the full experience the college CS majors received. Although I am booksmart in the fields, I come to the development world either being expected to know the ins and outs of engineering, or expected to pick it up quickly without the pampering. Learning engineering this way is fun, and not so mundane, but more frustrating.

But I though you already knew Java and Hadoop?

Well, that is all relatively speaking! The Java code I have written was not for Hadoop. The code that I write for my own research and hobby is much different from the code I am expected to write in production. In my research I always used Python (except in a few instances where I used Java) and the Hadoop Streaming package. Although Streaming is a great package, I feel it lacks the customization that a standard Hadoop job written in Java has. It also lacks the “meat” that the vanilla Hadoop distribution enjoys in the professional world. There are also performance gains from using vanilla Hadoop, as both the code and the framework are written in the same language and there is no interface among languages to deal with.

1. My code must execute successfully on tens and hundreds of thousands of files, not just some manageable subset that I have created.
2. I did not write the code that created these files, so there are bugs and intricacies that I must defensively program against.
3. These files are created in real time, on the fly, as real events are occurring. Stuff happens. Sometimes it’s not good stuff, and code must be able to work with (or toss out) that data. Data for my own research has been massaged and curated by moi before running in Hadoop. In production, this is not realistic.
4. The code must be efficient. These jobs must not take forever to run, because time can equal money: either wasted CPU cycles that others could use, or time in the cloud.
5. Development time must be used to integrate existing code rather than reinventing the wheel. My coworkers have already written a lot of code. I have been learning how to integrate this code into my own.
6. The developer must not go rogue with code and must code carefully. Crashing the cluster or maxing out the hard drive can have dire consequences.

Tip 0: Nobody’s Perfect

0.1 Prepare to be Frustrated. I cannot stress this enough. We all like to start new things thinking that we know close to everything we need to know to get started. Not true. Unless you literally have somebody sitting next to you whose job it is to train you, expect to do a lot of self learning and to be very frustrated. So, don’t get down on yourself, get plenty of sleep, eat, and drink plenty (after hours) if necessary.

0.2: Java Development can be Intimidating at First. To use somebody else’s Java code, just point the CLASSPATH to the src directory containing the package. If the source code is packaged correctly, this should work fine. The CLASSPATH can be set in your .bashrc file, or you can pass it using the -cp flag. Note that you must pass the CLASSPATH both to the compiler javac as well as the runtime java. This gets old really fast…more on this later.

When using a JAR file containing an archive of a package, the first line of each source file has a line that starts with package. This is essentially an “address” that allows you to point to that class without having to know its name, and is a good way to index the content of the JAR file.

0.3: Know your Limitations. A coworker would frequently chime in “use Eclipse!” Time and time again I have tried to use Eclipse and it makes me want to cry. Eclipse introduces its own level of complexity and customization to the build process that feels icky to me.

I have found that vim and ant have proven valuable for the time being. Adding an IDE just potentially adds another layer of misery to the learning process. The only IDE I ever recommend is Visual C++, and I am not even a Microsoft guy.

0.4: Make Subversion Play Nice. I did not realize that it is not necessary to check out the entire project. There is nothing wrong with checking out only the trunk directory. You can also give it a new name, so instead of having some meaningless directory called trunk, I can call it ryans_first_project:

svn co http://svn.mydomain.com/svn/project/trunk ryans_first_project

Tip 1: Partition your Data

Suppose we have n mappers and more than one of these mappers sees a key cat. We want to make sure that each instance of cat gets sent to the same reducer. One way to do this is to use a Partitioner. An ad-hoc way to do this is to just hash the key and get the remainder after dividing by some number k. This might also provide a way of controlling how many reduce tasks are spawned (I am not positive about that though).

Tip 2: The Key in the Mapper is Useless

If you use TextInputFormat, know that lines of text come into the mapper in key/value pairs and also leave the mapper in key/value pairs. The key coming INTO the mapper will be of this weird type LongWritable. It is useless. It is just the byte offset of the line in the file. What we really want to parse is the value coming into the mapper, and we emit the key and value to the next phase.

Tip 3: Use ant (or maven) to Configure and Build your Jobs

As I said, passing CLASSPATHs around .bashrc and on the command-line is a clusterf*ck. In my case, the code stack I am working with has a build.xml file. When I write source code, I don’t need to do anything, ant knows to compile the file because the build.xml file contains instructions to compile it.

Also, all and any libraries I need I just dump into a lib directory, and by simply adding the name of the library file to build.xml (it is obvious where to put it), it is automatically added to the CLASSPATH at compile time. To build the project, all I type is

ant some_target

and it spits out a cute little JAR file in the target directory, ready for me to use with Hadoop. Of course, this process actually builds the entire project, not just my code, but it only takes 10 seconds or so to build.

Tip 4: input_dir and output_dir are just Parameters

The Hadoop command given in tutorials usually has the following form

hadoop jar somejarfile.jar a.class.name input output

input and output are not “set” parameters, they are just plain old parameters that you can interact with in your Java programs by using the argv array. If you want to pass in 10 directories, you can do that!

Tip 5: The JAR file is the Key to Success

Once you have a JAR file built by hand or by ant, all you need to do is move that baby around to wherever you want to run the Hadoop job. Of course, this assumes that Hadoop is installed on the machines you want to use. Then, with one file, running the job is as simple as:

hadoop jar myjarfile.jar com.bytemining.my.package.name input_dir output_dir

Tip 6: When it Gets to be Stressful, It’s Nothing a Little Ping Pong Can’t Fix

I’ve only been at this for 2 weeks… there will undoubtedly be a part 2.

The next morning I made the short trek to the PayPal headquarters for a very long 9am-8pm day. Since I came up here for the camp, I wanted to make the most of it and paid the $30 for the morning session, even though I [...]]]>

My first time at ACM Data Mining Camp was so awesome, that I was thrilled the make the trip up to San Jose for the November 2010 version. In July, I gave a talk at the Emerging Technologies for Online Learning Symposium conference with a faculty member in the Department of Statistics, at the Fairmont. The place was amazing, and I told myself I would save up to stay there. This trip gave me an opportunity to check it out, and pretend that I am posh for a weekend . The night I arrived I had the best dinner and drinks at this place called Gordon Biersch. I had the best garlic fries and BBQ burger I have ever had. I ate it with a Dragonfruit Strawberry Mojito, the Barbados Rum Runner, and finished off with a Long Island Iced Tea, so the drinks were awesome as well. Anyway, to the point of this post…

The next morning I made the short trek to the PayPal headquarters for a very long 9am-8pm day. Since I came up here for the camp, I wanted to make the most of it and paid the $30 for the morning session, even though I had not intended on going originally.

Overview of Data Mining Algorithms with Dean Abbott

The paid morning session from 9-11:30 was led by Dean Abbott (@deanabb), the president of Abbott Analytics. It was an excellent overview of the basic data mining algorithms, but obviously 2 hours is not enough time to cover the algorithms in detail. When I first scanned through the slides I was concerned that I would be bored, but I actually learned a few things that made it worth it.

One of the first concepts I learned about was CHAID, (CHi-squared Automated Interaction Detector) a decision tree algorithm that can build wide n-ary trees rather than just binary trees like in CART. CHAID can also output a p-value, making diagnostic analysis more practical. I also did not know that decision trees could be used as a pre-analysis step to find interactions among variables. The output from this step can be used to construct better regression models including the proper interaction terms.

We moved on to linear regression and logistic regression which were obviously very basic. Next, we spent some time discussing neural networks. It is no secret that I detest neural networks. I don’t know what it is, but they annoy me to no end. It seems like there is very little science behind how to choose quantities such as the parameters, number of neurons or number of hidden layers. Maybe it is just me, but neural networks feel like a hack. Besides, anything that can be done with a neural network can be done using plain old statistics.

Dean also discussed other methods including nearest neighbors, Radial Basis Functions, Bayes Classifiers and Naive Bayes, Support Vector Machines (SVM).

At this point, we had to start rushing which was too bad. We briefly discussed ensemble methods including bagging, boosting (AdaBoost), and Random Forests. We spent about 5 minutes on unsupervised methods as well including k-means, Kohonen maps
(self-organizing maps). I am not sure what happened to principal components analysis (PCA), multidimensional scaling (MDS)
or independent components analysis (ICA). As I mentioned to a friend, unsupervised learning always gets the shaft. We had slides for association rules (Apriori algorithm), but we did not have time to discuss it. I was hoping semi-supervised learning, reinforcement learning and
recommendation systems would be mentioned, but there was not enough time even for what was on the agenda.

I wish we had more time. Unfortunately, there were way too many questions and a few individuals that wished to waste minutes debating and challenging the speaker.

Dean Abbott teaches a full, two-day course (not free) in data mining that may be of interest. Click here for more information. I usually would not post something like this, but he is an excellent, and practical speaker

What I found a bit surprising was that this session was at a Data Mining event. I would hope that most of the people in attendance had familiarity with a good amount of the material. The Netflix session at the previous ACM Data Mining Camp seemed to align better with the target audience of the day’s events. On the other hand, there were a ton of people in the session. Perhaps this was a good money-maker for the Bay Area ACM, because perhaps some people got their training on in the morning, and then left after lunch.

The eBay sponsored lunch was phenomenal, just like last time. I got a smoked ham sandwich and my little box also contained a bag of potato chips, an apple, and an oatmeal-raisin muffin looking thing (it was supposed to be a cookie but the baker got carried away).

Main Session

Next up was the main session which mainly consisted of a QA session with some experts in the field and also some job announcements from companies that sponsored the event.

• Somebody from SIGKDD announced the SIGKDD 2011 conference to be held in San Diego, CA in August 2011.
• A research engineer from eBay discussed the fact that many equate data mining with text mining and search. He drove home the point that at eBay, researchers are interested in other things such as social network analysis and valuing links.
• The Bayesian networks analysis tool BayesiaLab from Bayesia was introduced and the developers gave a shout-out to Judea Pearl over at UCLA. Dr. Pearl said about Bayesia, “This is good stuff!”
• LinkedIn talked about some of its new projects including CareerExplorer, that takes the professional graph and shows what a college student’s future career could potentially be. LinkedIn’s product team has engineers that specialize in machine learning, statistics, and data mining. They also host an “InDay” each month which is essentially its version of a hackday. They also mentioned that LinkedIn is investing very heavily in Hadoop, and they just tripled the size of their Hadoop cluster.
• Netflix is hiring “like crazy” and expanding internationally. Data mining engineers work on Cinematch technology and other projects.
• Joseph Rickert from Revolution Analytics introduced the crowd to its commercial version of R.
• Salford Systems talked a bit about its products including CART and Random Forests.
• SAS was also present and mentioned that it is looking for people that want to publish their books on data mining with them.

Large Data with R

Given that I gave a talk to the Los Angeles R Users’ Group on working with large datasets in R, I figured this would be an enlightening session. Unfortunately, the R skills that were covered were very basic, and it was little more than a commercial for Revolution Computing’s version of R. The take away from the session was basically just that the Revolution version has optimized methods that read the data into memory in chunks and operate on each chunk (perhaps) independently. This is nothing that a nice integration with Hadoop could not provide. No mention was made of the free open-source solutions for large datasets in R: bigmemory and ff.

If I had a time machine, I would have instead attended Rob Zinkov‘s talk on Sentiment Analysis. Rob is a second-year Ph.D. student in Computer Science at University of Southern California’s Information Sciences Institute and a member of the Los Angeles R Users’ Group.

Mahout

Next up was Ted Dunning discussing Mahout. I was elated to see practically each hand in the room shoot up when we were asked to vote on which sessions we wanted to attend. Mahout is a Java framework that provides scalable machine learning and data mining algorithms. Mahout code interacts with Hadoop to provide map-reduce functionality for algorithms. The purpose of Mahout is to provide early production quality scalable data mining. Some classification methods currently in Mahout include mixture modeling, Latent Dirichlet Allocation (LDA), logistic regression, naive Bayes, Complementary Naive Bayes, latent factor loglinear algorithms, stochastic gradient descent SVM, and random forests. Some of these methods are parallel, and some are sequential. Large scale SVD is currently being worked on, and still has some rough edges.

The biggest news in this talk was how well Mahout has been snapped up by industry. AOL uses Mahout’s methods for its product recommendation services. “A large scale electronics company” (name was secret) uses Mahout for music recommendations. Other uses of Mahout in industry include frequent itemset mining, and spam website detection.

Dunning mentioned that Mahout does seem to work well with sparse matrices assuming that if an element of the matrix is unspecified, it is equal to 0. If I understood his statement correctly, this means that Mahout works well with most sparse matrices. Some more technical gems I learned is that Mahout can do stochastic gradient descent (although it is sequential), and its implementation uses per-term annealing which can then be used for supervised learning with logistic regression. These implementations optimize for high dimensional sparser data, possibly with interactions. These methods are scalable and fast to train. Ted mentioned that for a particular test case, the optimization converged in less than 10,000 examples. For large datasets, it is possible that the method will converge before seeing all of the data. With that said, in the “best” case, an algorithm using stochastic gradient descent can be sublinear in the number of training examples.

Towards the end of the session, Ted answered some questions personally, and it gave me some insight into data mining methods. He is not a fan of “most common” itemset algorithms (Apriori, Eclat, etc.) because they are difficult to parallelize due to their quadratic nature. Instead, he prefers co-occurence analysis methods. He also prefers R to Weka, and he loves Python. I also prefer R to Weka, and love Python .

Large Scale Supervised Learning

The next talk I attended was rehearsed with slides etc. and was presented by Junling Hu, from eBay. Junling has a Ph.D. in Computer Science from University of Michigan, Ann Arbor. Although the talk began with (another) quick review of data mining algorithms, the meat of the talk was on how to parallelize some of these algorithms. The challenge of parallelization is that we must maintain a global function, and messages must be passed to update this global function. One basic way to do this is how map-reduce does it: split the data into subsets, perform some function on each subset and reduce the computations into one result. Each method has its own way it can be parallelized.

Decision Trees. One type of parallelization for decision trees is based on the tree nodes. One can write a map-reduce function to recursively split nodes, like the PLANET method proposed by Google.  We have some sequence of nodes in a tree and we maintain a model. With the proposed framework, we start with an empty set of nodes. We maintain a map-reduce queue for all the nodes we are going to divide and we also maintain an in-memory queue of nodes. The goal is to find the best splits based on some measure, in PLANET’s case, variance. We run some controller that controls map-reduce jobs. Each map-reduce job sends back data and we update the global variables: the model, the map-reduce queue and the in-memory queue. Then, new map-reduce jobs are constructed and the process continues. Due to time constraints, I had a difficult time following all of what was going on, but more information about the algorithm discussed can be found here.

Support Vector Machines. Hu mentioned two types of SVMs: primal SVM and parallel SVM. The idea behind parallelizing SVMs is to use block-based optimization by dividing either the data, or the features, into blocks. Stochasic gradient descent can be used for block minimization for the primal SVM. Some other ways mentioned included randomly splitting the data (bootstrapping perhaps), or using data compression (dimension reduction, perhaps). One resource for parallel SVM is the psvm project on Google Code which provides distributed gradient computation for maximum entropy, and parallel logistic regression.

Junling listed a few resources:

• ML-MapReduce, routines for machine learning using map-reduce (currently only logistic regression).
• Mahout
• IBM Parallel Machine Learning Toolbox, which is blackbox and not open-source.

Monetizing Images

The final time slot was slim pickings for me, and the second time slot (when I attended Mahout) hosted 4 sessions I wanted to attend that all conflicted with each other. The discussion about the association between tweets and stock prices sounded interesting, except for the stock prices part. So, I attended the Monetizing Images session. This session was more of a discussion about data mining with images in general.

• Apparently, Facebook can identify brands in images and use these brands for advertising. I have not seen this but I do not really doubt that it is true.
• Microsoft Photosynth crawls freely available images on the web and uses these images to create an entire scene from them, essentially allowing someone to tour Rome using just images on the web.
• The United States Postal Service uses k-nearest neighbors for intelligent character recognition (ICR) used to read addresses on envelopes.
• Google Goggles
• ZunaVision allows advertisers to embed logos and ads into a video with more flexibility than with things like green screens used on football fields etc.

We also discussed forensic photography and the ability to detect if an image has been doctored. We also discussed some techniques for measuring image similarity. David Lowe from the University of British Columbia maintains a list of uses and companies regarding computer vision on his website.

It is interesting to note that after the fact, Ken Weiner (@kweiner) from GumGum in Los Angeles indicated that monetizing images is exactly what they are doing. Sounds interesting!

At this point I was exhausted. I like meeting Twitter friends and followers, but people very quiet! It was a pleasure to meet Scott Waterman (@tswaterman) and Tommy Chheng (@tommychheng). I also got to reconnect with my friend Shaun Ahmadian (@ssahmadian) from the UCLA Department of Computer Science as well as Rob Zinkov (@zaxtax) who also made the trek from Los Angeles to San Jose.

Yahoo’s S4: Distributed Stream Computing Platform

First off, it must be said. S4 is NOT real-time map-reduce! This is the meme that has been floating around the Internets lately.

S4 is a distributed, scalable, partially fault-tolerant, pluggable platform that allows users to create applications that process unbounded streaming data. It is not a Hadoop project. A matter of fact, it is not even a form of map-reduce. S4 was developed at Yahoo for personalization of search advertising products. Map-reduce, so far, is not a great platform for dealing with streaming/non-stored data.

Pieces of data, apparently called events, are sent and consumed by a Processing Element (yes, PE, but not the kind that requires you to sweat). The PEs can do one of two things:

Streaming data is different from non-streaming data in that the user does not know how much data will be transmitted, and at what rate. Analysis on [...]]]>

This week, a few different big data processing tools were released to the open-source community. I know, I know, this is probably the 1000th blog post about this, and perhaps the train has left the station without me, but here I am.

Yahoo’s S4: Distributed Stream Computing Platform

First off, it must be said. S4 is NOT real-time map-reduce! This is the meme that has been floating around the Internets lately.

S4 is a distributed, scalable, partially fault-tolerant, pluggable platform that allows users to create applications that process unbounded streaming data. It is not a Hadoop project. A matter of fact, it is not even a form of map-reduce. S4 was developed at Yahoo for personalization of search advertising products. Map-reduce, so far, is not a great platform for dealing with streaming/non-stored data.

Pieces of data, apparently called events, are sent and consumed by a Processing Element (yes, PE, but not the kind that requires you to sweat). The PEs can do one of two things:

1. emit another event that will be consumed by another PE, or
2. publish some result

Streaming data is different from non-streaming data in that the user does not know how much data will be transmitted, and at what rate. Analysis on streaming data should not rely on storing the data, as the amount of required disk space is unknown. Additionally, the processing of the data is likely to take longer than the rate of transmission would allow. Since the data is not stored, special algorithms must be developed for aggregating and analyzing data. Neal Richter (@nealrichter) has an excellent list of resources on research on the management and analysis of streaming data.

More information can be found at the S4 Wiki and S4 main site, that contains tutorials, a manual, a cookbook as well as API documentation. The Yahoo project page, which contains very little information, can be found here. The source code is on everybody’s favorite site, GitHub.

S4 is released under the Open Source Apache 2.0 license. It must also be said that S4 is not to be confused with S3! They are two totally different technologies!

Remember, S4 is not a Hadoop. A matter of fact, Bill McColl over at Gigaom has pondered a”NoHadoop” movement…that parallels (see what I did there?) our favorite NoSQL movement.

Google’s Sawzall: Programming Language for Big Data

Google made a contribution of its own. Sawzall is an interpreted, procedural DSL for working with huge amounts of data.  Greg Linden (@greglinden) made an interesting comparison, suggesting that Yahoo’s Pig project is similar to Google’s Sawzall project. At Google, it is used on top of existing systems including Protocol Buffers, the Google File System and MapReduce. Sawzall reads one line of data at a time, and does not preserve state between reads so it is useful in the map phase of a map-reduce job. There are also routines for statistical aggregation that can be used in a reduce phase. Users compile Sawzall source using the szl compiler that can be found here.

Much more detailed information can be found on the Google Code overview site for the szl project. For programming language buffs, the language specification can be found here.

The research publication discussing this project in more detail is here.

Yelp’s mrJob: Distributed Computing for Everybody

Ok, I’ve ignored Hadoop long enough…

Every time you write a review complaining about the terrible gas the burrito at El Torasco’s gave you, or the amazing buzz you got from their margaritas, Yelp processes it and extracts some type of information from it. Yelp accumulates about 100GB of data per day! Naturally, Yelp analyzes this data using map-reduce, Amazon Elastic MapReduce to be exact.

You see, most companies are building up their Hadoop clusters but Yelp decided to tear theirs down. In May 2010, Yelp engineering moved its data processing to Amazon. mrjob is Yelp’s Python framework for writing map-reduce jobs and interacting with Amazon EMR!

Below is an example from their engineering blog. It is so simple it is beautiful!

from mrjob.job import MRJob
import re

WORD_RE = re.compile(r"[\w']+")

class MRWordFreqCount(MRJob):
    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield (word.lower(), 1)
    def reducer(self, word, counts):
        yield (word, sum(counts))

if __name__ == '__main__':
    MRWordFreqCount().run()

The mrjob code is available on GitHub as is the Python documentation.

Oh, and El Torasco’s is to be a fictional name I use in this post.

Download/view slides here. Topics include Using Python with R with an example using web mining. Web mining using pure R rather than Python.

Code for demonstration is here: offtopic_demo.py is a pure Python script that extracts data from a web forum and dumps it to disk. To actually use it, you will need to register for an account. RPy2_demo.py reads the data from the forum from disk and calls R from Python to perform some basic analysis. curljson_demo.R grabs some JSON data from the Twitter Search API using RCurl and converts it to R lists using rjson.

Video: Running the code requires some packages that you need to install. twill package for web browsing, that installs a Python package for you. Requires the mechanize package as well. twill is a wrapper [...]]]>

This past Tuesday I had the opportunity to present a short talk (a bit long) related to text mining at the Los Angeles R Users’ Group. Since I do most of my text mining in Python, I took this opportunity to discuss RPy2, an interface to R from Python. My slides are below:

• Using Python with R with an example using web mining.
• Web mining using pure R rather than Python.

Code for demonstration is here:

1. offtopic_demo.py is a pure Python script that extracts data from a web forum and dumps it to disk. To actually use it, you will need to register for an account.
2. RPy2_demo.py reads the data from the forum from disk and calls R from Python to perform some basic analysis.
3. curljson_demo.R grabs some JSON data from the Twitter Search API using RCurl and converts it to R lists using rjson.

Video:

• twill package for web browsing, that installs a Python package for you. Requires the mechanize package as well. twill is a wrapper to mechanize.
• BeautifulSoup package for Python for HTML parsing.
• R must be built to use as a shared library using --enable-R-shlib, otherwise Python cannot call it.
• RPy2, the Python interface to R.

To see the main talk of the evening, click here.

Some Recommended Books

Natural Language Processing

• Foundations of Statistical Natural Language Processing, Manning and Schuetze.
• Speech and Language Processing, Jurafsky and Martin.
• Natural Language Processing and Text Mining, Kao and Poteet.

Text Mining

• Practical Text Mining with Perl, Bilisoly. See my review of this book in the Journal of Statistical Software here which is also excerpted on Amazon!
• Text Mining: Applications and Theory, Berry and Kogan (NEW).
• The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data, Feldman and Sanger.
• Mastering Regular Expressions, Friedl.

Data Mining

• Elements of Statistical Learning: Data Mining, Inference and Prediction. Hastie, Tibshirani and Friedman.
• Data Mining: Concepts and Techniques (recommended by @nealrichter). Han, Kamber and Pei. 
• Data Mining: Practical Machine Learning Tools and Techniques [the fern book]. Witten and Frank.
• Introduction to Data Mining [the rock book]. Tan, Steinbach, Kumar.

Web Mining

• Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data, Liu.
• Mining the Web: Discovering Knowledge from Hypertext Data, Chakrabarti.
• Mining Graph Data, Cook and Holder.
• Managing and Mining Graph Data, Aggarwal and Wang.
• Social Network Analysis: Methods and Applications, Wasserman and Faust.