In this post I am goIing to summarize some of the things that I learned at Strata Santa Clara 2013. For now, I will only discuss the conference sessions as I have a much longer post about the tutorial sessions that I am still working on and will post at a later date. I will add to this post as the conference winds down.

The slides for most talks will be available here but not all speakers will share their slides.

This is/was my first trip to Strata so I was eagerly awaiting participating as an attendant. In the past, I had been put off by the cost and was also concerned that the conference would be an endless advertisement for the conference sponsors and Big Data platforms. I am happy to say that for the most part I was proven wrong. For easier reading, I am summarizing talks by topic rather than giving a laundry list schedule for a long day and also skip sessions that I did not find all that illuminating. I also do not claim 100% accuracy of this text as the days are very long and my ears and mind can only process so much data when I am context [...]]]>

In this post I am goIing to summarize some of the things that I learned at Strata Santa Clara 2013. For now, I will only discuss the conference sessions as I have a much longer post about the tutorial sessions that I am still working on and will post at a later date. I will add to this post as the conference winds down.

The slides for most talks will be available here but not all speakers will share their slides.

This is/was my first trip to Strata so I was eagerly awaiting participating as an attendant. In the past, I had been put off by the cost and was also concerned that the conference would be an endless advertisement for the conference sponsors and Big Data platforms. I am happy to say that for the most part I was proven wrong. For easier reading, I am summarizing talks by topic rather than giving a laundry list schedule for a long day and also skip sessions that I did not find all that illuminating. I also do not claim 100% accuracy of this text as the days are very long and my ears and mind can only process so much data when I am context switching between listening, tweeting, emailing etc.

In the mornings there were several short plenary talks where people throughout the industry discussed their particular views of Data Science. This was basically a warm-up for what would become very long days. I mostly used this time to catch up on email and review stuff from the previous day. The second day apparently had a lot of gimmicky sales talks, but I was not paying attention apparently. The most interest talk came from Jennifer Pahlka from Code for America. I had first learned about Code for America at the Data Scientist Summit hosted by EMC in 2011. At the time it sounded like a very good idea — we have college graduates that dedicate a couple years to teaching in inner-city schools so it makes sense that we should have some data scientists working on “projects that make a difference in the world” as Jennifer would say. The projects that these data scientists work on involve democratizing data and open data initiatives in local governments. A couple of projects that stood out to me were a project to release some 800+ datasets from the City of Santa Cruz (an amazing city) on a website. Another project involved studying bail amounts and the outcome of a criminal trial. This talk was apropos considering the recent announcement of code.org, the start of a movement to begin teaching computer programming to K-12 students. [You can sign the petition and register as a volunteer here.]

Visualization Strand

I have said in the past that visualization is not my thing. I greatly appreciate interactive graphics and cool infographics that convey strong meaning to non-data scientists but it simply is not my cup of tea yet. However, it is something that I want to invest time into. I decided to attend visualization talks up to my tolerance level (which isn’t very high)… which meant one two.

I attended Chang She‘s talk Agile Data Wrangling and Web-based Visualizations. Chang did what I usually do: pack too much into a one-hour talk… but I feel that talks like this really whet the appetite to learn more. He discussed how data science is missing a “blue button” that takes care of data management and then visualization. Using the federal election commission dataset, he showed political donations by party, candidate and state as the motivating example. Chang showed several examples of using pandas (a Python data munging library) to manipulate the data and then passing that data to d3.js using a JSON data format with a web server. I felt that this was just a basic talk on how to combine tools to munge data and then visualize it. It is far from a blue button, but shows how important such processing pipelines are.

Law, Ethics and Open Data Strand

One of the highly acclaimed talks of the day came from Joseph Turian of MetaOptimize, titled Sci vs. Sci: Attack Vectors for Black-Hat Data Scientists and Possible Countermeasures. Every skill has a good use and an evil use and Data Science is no exception. We create models to try to combat fraud, detect spam, measure influence and much more. These “good” uses of skills are called “white hat.” On the other hand, a more evil Data Scientist can circumvent these models to allow their spam to go undetected or game an influence metric such as PageRank. For example, consider a malicious web page that contains code that simply repeats a user’s 1Google query endlessly. To a very stupid search engine, such a web page would game a keyword matching algorithm and the search engine that is based on it. This crap web page would appear as the first result because it appears the most relevant. This is a very elementary example, but one can imagine how sophisticated models can produce nasty results.

Turian believes that most Data Scientists originally come from academia where the skills we learned are mainly “white hat”, but that our use in industry is mainly “grey hat” (somewhere between good and not-so-good). Such “grey hat” methods may involve some sort of data privacy issue such as with ad retargeting. A “black hat” data scientist may be useful in constructing a botnet, using Markov models or other language models to generate human-looking spam text, or to create sock puppets to sway opinion in a large social network. A sock puppet is essentially a social media account that is designed to look like a real genuine human but that has an ulterior motive, mainly to proliferate propaganda or false information. The use of these sock puppets is referred to as “astroturfing” — that is, a fake grassroots movement. One easy example I can think of are the thousands and thousands of Twitter accounts that are created simply to sway opinion about President Obama (search for #tcot and you are likely to find some examples, though many are also legitimate users). Turian cited one unsophisticated example of astroturfing: Newt Gingrinch and his huge jump of followers in a short period of time, which was determined to be fake. In this case, it is alleged that Gingrinch’s campaign paid for followers rather than create an army of sock puppets. Some methods for locating sock puppets are the presence of reply spam (@spam), manual classification, andhoneypots.

Some interesting statistics:

1. 7% of Tweeps (Twitter users) are spam bots.
2. 20% of us accept friend requests from people we do not know.
3. 30% of us have been deceived by chat bots.

Note: MetaOptimize hosts an amazing machine learning Q and A site similar in function to StackExchange/StackOverflow. You can visit it here.

Data Science Strand

IPython Notebooks

The first talk of this series I attended was The IPython Notebook: a Comprehensive Tool for Data Science by Brian Granger at Cal Poly San Luis Obispo and Chronicle Labs. One of the major problems in Data Science is that “code and data do not communicate much.” That is, code is usually placed in one file, and data in another file and an analysis involves the coupling of data and code that must be kept in sync throughout the process. Imagine if all of your work as a Data Scientist could be contained on your physical desktop as separate objects — this is a good analogy for IPython Notebooks. An IPython Notebook functions much like aMathematica notebook, or a Sage notebook. One can analyze data in pandas data frames, use some fancy models from SciPy or scikit-learn, use the general Python language as well as the niceties provided by IPython all in one place. Once the code is written, one can produce plots with matplotlib in place and then distribute the document to others. IPython Notebooks provide a living document of one’s work and allows resilience from change by keeping all of the code in one place. Additionally, the concept of cell magic allows the execution of other languages such as R, Ruby and Julia from within the IPython Notebook! Soon there may be no need to run multiple interpreters or have multiple different open-source notebook projects for each additional language!

Here is the amazing part: by using so-called cell magic, one can push a Python object, say a pandas dataframe directly into R and it is converted into an R dataframe. I do not remember the specifics of why this is possible, but this is huge. This eliminates the need for packages like RPy2 for basic computations between R and Python. [Edit: RPy2 is used under the hood for this conversion. Thanks to Dirk for pointing this out.] Brian mentioned that it also may be possible to eventually allow Python objects to interact with JavaScript libraries such as d3.js for visualization using widgets.

IPython Notebooks support narrative text, headings, graphics and also mathematical typesetting via MathJax. Executing code produces JSON strings that are portable and serializable for saving results without requiring code to be re-executed. The site nbviewer.ipython.com provides an online viewer for IPython notebooks via URL, Git repository URL or Gist URL. This viewer does not require the web service to be installed locally. One current limitation of IPython Notebooks is that they only support a single user and thus cannot be hosted for, say, multiple students to login to their own notebook session in a classroom.

Once ipython and ipython-notebook (the Ubuntu packagename) are installed, one just executes the command ipython notebook in the directory of interest to start up a webserver for working with IPython Notebooks.

Apparently entire textbooks are being written as IPython Notebooks for their beauty, scientific ease and portability.

Adversarial Learning

The final talk I attended was What To Do When Your Machine Learning Gets Attacked by Vishwanath Ramarao. The purpose of this talk was to discuss issues with the bad guys trying to circumvent machine learning models designed to prevent abuse of a system, such as a spammer learning how to get around a spam filter over time. This spammer is called an adversary, and can be a “black hat” data scientist. Some examples of adversarial situations are login fraud (spearfishing, PR embarrassment or financial information), comment/mail spam, sign up fraud, astroturfing, credit card fraud and click fraud. Adversarial learning is the set of techniques that classify data emitted from an adversary.

An adversarial situation arises when the adversary is able to observe the output of the learning system and can change some subset of the features used in that system so that their attempts go unpunished. The goal of adversarial learning is to make it costly for an adversary to change features. The approach towards a solution is labor intensive, but simple to explain. Ramarao essentially said that the best way to combat adversaries is to

1. engineer features interactively and quickly. 
2. not throw away features as we commonly do. It is possible that some features may be activated as the adversary’s methods evolve. 
3. consider the entire transmission of an adversarial transaction — that is, do not just look at the words in a spam email but also look at the HTTP headers and other communication information passed along with the text.
4. study anomalies (outliers and high leverage points) and not discard them. Usually such anomalies are adversaries.
5. permit overfitting when necessary for the reason mentioned in #3.

As a text mining enthusiast, I learned some interesting tricks on fitting machine learning models to text, neither which had anything to do with adversarial learning.

• A homoglyph is the translation of a word by replacing some characters with a character that looks similar. For example, p0rn is a homoglyph of porn – theo in porn is replaced with a character that looks similar, the zero 0. Broken words
• A broken word is a translation of an intended word with spaces added. For example, the word nigeria could be a feature for a spam detection algorithm. An adversary can bypass the filter by instead writing ni geria.
• Hash busters are cases where new words that were not in the lexicon used to train the text model are injected into content. One should use the count of the number of hash busters and use it as a feature in a model. One common hash buster for a naive profanity filter would be the word fcuk instead of the actual word f*ck.

Julia

After being enlightened by this wonderful talk, I am going to write a more substantial post focusing solely on Julia, so for now I will just briefly describe some of the more easy-to-explain content. This talk was presented by Michael Bean from Forio (developers of Julia Studio). As data scientists we love dynamic environments for interactive data munging such as R, or the Python shell (with pandas or SciPy). We typically start with a high level language such as R and then port this code to a compiled or performant language like C, C++ or Java (and maybe Python). This is a large barrier in scientific computing because it requires the data scientist to know two languages: one to experiment, and one to implement. Julia is a scientific computing language that provides the performance of a programming language like C++ and adds technical libraries and accessibility for scientific exploration. Bean cited that Julia’s performance is similar to C++. Julia allows us to complete tasks faster because we remove the need for “glue” code and Julia packages are written in Julia for performance rather than requiring C or Fortran. [R packages can be written solely in R, but for computationally intensive operations, or for packages that will sit in a bottom layer such as data structures etc. there is a huge performance hit.] Once one is familiar with Julia, it is easy to “hack the core” so to speak. 

Other features that impress me:

• the user can redefine arithmetic operations and construct new data types. Julia uses multiple dispatch which is a programming language feature that uses different implementations of functions depending on the data types passed to the function. For example, if A and B are of type matrix, then Julia will know that A * B is the matrix multiplication operation rather than elementwise multiplication. 
• common data structures found in computer science are supported natively such as BitArrays and SubArrays as well as types statisticians are already familiar with including Distribution and DataFrame.
• support for list comprehensions. For example, to square every element, use [xi^2 for xi in x] instead of a for loop.
• every package is a Git repository and thus open-source and easy to access.
• some packages support multicore natively.
• certain functions that can have a bash (!) appended which tells Julia not to make copies of the object (think in-place sort which is sort!).

Bean showed that the development process with Julia is shorter than languages such as R because production-level re-implementation is not necessary. The runtime is also faster for the few examples he showed. The following is an example of the recursive implementation of generating Fibonacci numbers in both R and Julia

fib <- function(n)
{
  if (n < 2) {
    return(n)
  } else {
    return(fib(n-1) + fib(n-2))
  }
}
 
start <- Sys.time()
fib(36)
end <- Sys.time()
end - start

fib(n) = n < 2 ? n : fib(n - 1) + fib(n - 2)
@elapsed fib(36)

Connected World Strand

Bit.ly: Deriving an Interest Graph

The first talk in this strand that I attended was by Anna Smith of bit.ly titled Deriving an Interest Graph for Social Data. It should be no surprise that a URL shortening service would have a ton of data to sift through. Anna stated that a lot of her work is one-off analysis. What I liked about Anna’s talk in particular is that the visualizations she used were very basic. There was nothing fancy about what her graphics displayed — they just displayed some insights about the data and that is it. 

Bitly extracts a lot of data from each shortened URL including keywords, topics and the probability the click was a human. One can derive a taxonomy and interest graph by analyzing click data among links. The idea is to look at other webpages a user went to from the page related to the shortened URL. It is hypothesized that the next page the user visits is related in content to the current page. On a domain level, a coclick graph uses domains as nodes and the number of clicks between them as edges. From this, we can derive a graph of keywords by using the Jaccard similarity using the number of clicks to a domain with a particular keyword for both sets. The resulting coclick graph has 4.5 million keywords and 9 million edges. By using some basic processing (removing non-English keywords and keywords with low click numbers) and then running a clustering algorithm called DBSCAN, they were able to simplify their graph to 200,000 keyword clusters and 1 million edges.

The Data Science group at bit.ly keeps an updated GitHub repository for their work here.

LinkedIn Endorsements

The last session in this strand that I attended was by Sam Shah and Pete Skomoroch from LinkedIn. This talk discussed the skills endorsement feature of LinkedIn and how they made it successful using science. Sam and Pete credit most of the success to establishing viral loops and using recommendation engines as follows:A endorses B -> B is notified -> B accepts the endorsement and endorses someone else.

Social tagging of skills also accelerated adoption. First users market their skills, and then other skills are recommended for them to add. First, a user thinks about their skills and tags them on their profile. Then, a recommendation system recommends other related skills as well as some potential people for the user to endorse. But this is not the interesting part…

How does LinkedIn maintain a skill dictionary and taxonomy? This is a high unwieldy problem due to human psychology and variations of language usage. One of the biggest issues is in phrase sense disambiguation. The motivating example was the skill angel. If I list angel as a skill on my profile, am I referring to myself as an angel investor or as a spiritual being? The speakers indicated that by using the graph of all the skills listed in addition to angel, we could use agglomerative clustering and then a distance metric to determine which meaning is most likely. This is an example of MS Office, Microsoft Office, Office. All of these concepts refer to the same thing. For this particular problem, LinkedIn used crowdsourcing with Mechanical Turk tasks. An example human interaction task was to ask a participant to find the best Wikipedia article for the particular topic, since Wikipedia tends to already have a strong army that de-duplicates content. 

This is all great for users that actively use the Skills feature, but some do not. For those users, a system passes a sliding window over the profile text (n-grams) and emits possible matches basd on the taxonomy, and tossing out words that do not fit into the inferred topics in the profile. For example, if my profile text says “I love working with data, Python, Java and Hadoop.” The words I, with and and will all be tossed as stopwords. Then, I have the following keywords left:love, working, data, Python, Java, HadoopFor all practical purposes, working is probably considered a stopword or low-impact word because it appears so much on LinkedIn profiles. data is probably not an actual skill so both of these words are removed, leavinglove, Python, Java, HadoopUsing LinkedIn’s taxonomy of skills, we would probably deduce that Python, Java and Hadoop are highly related and love is an extreme outlier (for some people love may be an actual skill, but likely not in this context). Finally, this system would tag Python, Java, and Hadoop as skills to add to the profile. For more complex (realistic) examples, LinkedIn would then apply word sense disambiguation and de-duplication. A simple Naive Bayes algorithm is used to generate the actual recommendations. In the event of a completely blank profile, recommended skills are based on title, organization and perhaps social network features.

LinkedIn can also suggest endorsements where the system asks a user to endorse another user for particular skills that the user may know about. Some features used for this recommendation engine include people-skill combinations, school overlap, group overlap, similarity in industry, title similarity, site interactions, and co-interactions. Such a recommendation engine is basically a binary classification problem for link presense.

This talk by LinkedIn was surprisingly candid. Obviously, one cannot employ the methods they discussed because we do not have access to their data or infrastructure, thus such a talk is of no risk to intellectual property. Many companies do not get this and do not allow their employees to speak about anything involving their work.

Conclusion

I am glad I forked out the money to attend Strata and I will likely attend next year. The conference was huge and there was something for every data geek including a ton of food. The conference overall was not as sales-y as I thought it would be, but there were definitely moments particularly in the morning sessions and at the expo. I mainly just collected t-shirts at the expo hall, but it was basically just a giant “my Hadoop distribution is 100x faster than the other guys’.” There was also a really cool sensor lab setup for collection of data using Arduino sensors. There were several sensors placed throughout the conference venue and the data was visualized and place here.  

During my time at Strata so far, I have finally had the chance to meet some longtime Twitter friends and reunite with others. It was great meeting Neil Kodner and discussing our common interests as well as meeting Mathieu Bastian and discussing graph processing and the future of Gephi (I need to write a blog post about Gephi soon). I had a chance to talk to Wes McKinney over lunch as well about Python and the pandas community. On the last day, I went to an event hosted by Facebook and met several Facebook engineers and other Twitter friends including Joseph Turian, Bradford Stephens, Daniel Tunkelang and Greg Rahn. Everybody I met has now relocated to the Bay Area and I think I am going to need to follow…

For development, Raspberry Pi can connect to a television (or other display) via HDMI or composite video (the “yellow” plug for those still stuck in the 1900s haha). A keyboard, mouse and other devices can be connected via two USB ports. A powered hub can provide support for even more devices. There are also various pins for connecting to a breadboard for analyzing analog signals, for a camera or for an external (or touchscreen) display. An SD Card essentially serves as the hard disk and probably a portion of the RAM. The more recent Model B ships with 256MB RAM. Raspberry Pi began shipping in February 2012 and these little guys have been very difficult to get a [...]]]>

Last week I received two Raspberry Pis in the mail from AdaFruit and just now have some time to play with them. The Raspberry Pi is a minimal computer system that is about the size of a credit card. In the embedded systems community, the excitement is for obvious reasons, but I strongly believe that such a device can help collect and use data to help us make better decisions because not only is it a computer, but it is small and portable.

For development, Raspberry Pi can connect to a television (or other display) via HDMI or composite video (the “yellow” plug for those still stuck in the 1900s haha). A keyboard, mouse and other devices can be connected via two USB ports. A powered hub can provide support for even more devices. There are also various pins for connecting to a breadboard for analyzing analog signals, for a camera or for an external (or touchscreen) display. An SD Card essentially serves as the hard disk and probably a portion of the RAM. The more recent Model B ships with 256MB RAM. Raspberry Pi began shipping in February 2012 and these little guys have been very difficult to get a hold of. I finally got tipped off as to when more became available by following the Raspberry Pi subreddit. Raspberry Pi was originally not designed with geeks in mind. In fact, they were originally designed to teach school children about computers and programming. 

The figure below shows the size of my Raspberry Pi versus the size of the credit card I purchased it with (just kidding). The price is also small, at about $35 depending on where you buy it!

So what can you do with it? I imagine almost anything a computer can do. Just remember that you are limited by lightweight CPU, power restrictions and potential heat issues. Raspberry Pi does allow outputting high definition video though. I have not done enough testing to check these though.

Here are some generic ideas:

• Realtime informational displays of data and graphics on a large display. 
  • The Raspberry Pi conforms to some standard that allows it to be mounted (with assistance) to the back of an HDMI display.
  • Use the RPi as a dedicated system for pulling data from other systems, doing some lightweight processing (or pull results from another system) and then display the results.
• Small, portable data collection and transmission devices.
  • Raspbery Pi can be connected to AC power of course, or using a MicroUSB to USB cable, similar to those used to charge Android devices.
  • Connect a small (or regular sized) wireless adapter, or 3G/4G dongle for data transfer.
  • Connect a Bluetooth dongle for communication with other data collection devices (think GPS receivers etc.).
  • Connect an IR receiver via USB for remote control.
  • Connect a USB battery backup for times where AC is not available (5V) such as in the field, or when an automobile does not provide power.
• Development of data-driven “fat” clients.
  • Use Raspberry Pi to make automated decisions using machine learning using your favorite development tools and statistical libraries including R. Obviously, mileage may vary. We are not talking about 8-core Xeon CPUs here…
• For use as a “motherboard” (pun?) for collecting and analyzing analog signals using a separate breakout board for Raspberry Pi.

It is important to understand that hardware compatibility is more hit-or-miss than it is with a standard desktop or laptop. Certain chipsets must be matched, and drivers must be compatible with the ARM architecture. To research which items to purchase, I took a look at the RPi Verified Peripherals wiki page.

A similar platform I was eyeing was the Arduino. The biggest win of the Raspberry Pi over Arduino (I believe) is that Raspberry Pi is a mini-computer that can run a standard garden-variety operating system (well, Linux), whereas Arduino is a platform for collecting and transmitting analog and digital signals using its own software.

So, what do I plan to do with my Raspberry Pis? It is kind of secret… OK, not really, but I don’t want to write about it until I have something to show! What will you do with yours?

OpenPaths is a service that allows users with mobile phones to transmit and store their location. It is an initiative by the New York Times that allows users to use their own data, or to contribute their location data for research projects and perhaps startups that wish to get into the geospatial space. OpenPaths brands itself as “a secure data locker for personal location information.” There is one aspect where OpenPaths is very different from other services like Google Latitude: Only the user has access to his/her own data and it is never shared with anybody else unless the user chooses to do so. Additionally, initiatives that wish to use a user’s location data must be asked personally via email (pictured below), and the user has the ability to deny the request.The data shared with each initiative provides only location, and not other data that may be personally identifiable such as name, email, browser, mobile type etc. In this sense, OpenPaths has provided a barebones platform for the collection and storage of location information. Google Latitude is similar, but the data stored on Google’s servers is obviously used by other Google services without explicit user permission.

The service is also opt-in, that [...]]]>

OpenPaths is a service that allows users with mobile phones to transmit and store their location. It is an initiative by the New York Times that allows users to use their own data, or to contribute their location data for research projects and perhaps startups that wish to get into the geospatial space. OpenPaths brands itself as “a secure data locker for personal location information.” There is one aspect where OpenPaths is very different from other services like Google Latitude: Only the user has access to his/her own data and it is never shared with anybody else unless the user chooses to do so. Additionally, initiatives that wish to use a user’s location data must be asked personally via email (pictured below), and the user has the ability to deny the request.The data shared with each initiative provides only location, and not other data that may be personally identifiable such as name, email, browser, mobile type etc. In this sense, OpenPaths has provided a barebones platform for the collection and storage of location information. Google Latitude is similar, but the data stored on Google’s servers is obviously used by other Google services without explicit user permission.

The service is also opt-in, that is, it does not use hidden files on the user’s phone to track location — instead, the user must launch the OpenPaths mobile application to “opt-in” to location tracking. This is a double-edged sword though. I am in the minority, but I would rather have OpenPaths transparently transmit my location data (like Google Latitude) without me having the launch an app, because then the data is more complete. Despite running the application consistently for the past month, there are some unexplainable gaps in the data. For example, somehow only a few days of data for the month of June are available. Fortunately (or unfortunately), my habits do not change that much. After logging in, the user has the ability to visualize their data, or download it in several formats: JSON, CSV or KML for Google Earth.

The user can then view his or her location history on a map. The points are colored by time of day (morning, afternoon, night), or time of week (weekday, weekend). The gradient could and should be more fine grained to provide for better understanding of the user’s location and habits. The map has an animation setting that shows your movement throughout time. Compared to other services, the data seems very coarse, even with the finest settings. Driving at 70mph, a data point seems to be transmitted every 20 miles or so which is 2-3 times per hour. All in all, the map functionality is cool, but leaves a lot to be desired currently.

OpenPaths also allows data to be collected from FourSquare if you choose to do so. Unfortunately, none of my check-ins showed up on the map. Of course, the user can choose to delete their information entirely, adding to the OpenPath’s idea of “opt-in” location sharing.

All in all, I am very excited of where OpenPaths can go. Currently, there seem to be a few technical issues with the Android app that prevents transmitting location data when the app is running in the background. I hope that OpenPaths can run more transparently and reliably on Android as development continues. The privacy policy and ownership of data are both very important and a model that others should implement. The visualization aspect is not incredibly useful or flashy right now, but the user is free to download their data and create their own visualizations or analysis.

To download an OpenPaths app for your Apple or Android phone, click here.

A: “When I was a kid, I really enjoyed looking at graphs, plots and maps. My parents and I could not make of what was behind the interest. At the same time, I was also heavily interested in education. My mother was a teacher and the first set of statistics I ever encountered were standardized test scores. I strived to understand what the scores attempted to say about me, and why such scores and tests are so trustworthy. When the stakes increased with the AP and SAT exams, I began reading articles published by the Educational Testing Service and learned a ton about how these tests are constructed to minimize bias, and how scores are comparable across forms. It fascinated me how much science goes into these tests, but in the end of the day they are still just one factor [...]]]>

Recently, I participated in an email interview about what being a Statistics major entailed, how I got interested in the field and the future of Statistics. I figured this might be of interest to those that are contemplating majoring in Statistics, or considering a career in Data Science.

A: “When I was a kid, I really enjoyed looking at graphs, plots and maps. My parents and I could not make of what was behind the interest. At the same time, I was also heavily interested in education. My mother was a teacher and the first set of statistics I ever encountered were standardized test scores. I strived to understand what the scores attempted to say about me, and why such scores and tests are so trustworthy. When the stakes increased with the AP and SAT exams, I began reading articles published by the Educational Testing Service and learned a ton about how these tests are constructed to minimize bias, and how scores are comparable across forms. It fascinated me how much science goes into these tests, but in the end of the day they are still just one factor in the whole picture of a student. This niche interest lead me to statistics, psychometrics in particular, and although I no longer study psychometrics, I found what I learned to be incredibly valuable.”

Q2: “I noticed you have bachelor’s, master’s, and doctoral degrees in statistics. How did your graduate study build on what you learned in your undergraduate program?”

A: “For me, the undergraduate and graduate programs were night and day. The undergraduate program focused more on modeling and data analysis. The graduate program focused more on thinking about data and how to develop a scientific “common sense” about how to work with, express and make automated decisions based on data. The graduate program was much more mathematically and computationally intensive than the undergraduate major. My graduate study actually built more on my mathematics major in college because many of the concepts in graduate statistics require knowledge of linear algebra, numerical analysis and real analysis. Fortunately, our statistics major requires upper division math courses.”

Q3: What was the most interesting part of majoring in statistics? What did you find most challenging?

A: “The most interesting part of majoring in statistics was seeing how many fields can grow and transform based on insights from data and statistics. In my case, I found it most interesting seeing how it integrates and interacts with computer science. Every time someone surfs through Facebook, enters a Google search, or looks at an item on Amazon, data about what you are doing are collected and algorithms process this data to enrich the experience by, say,recommending books and offering special deals on Amazon, recommending friends and showing relevant stories on Facebook, and the most groundbreaking of all: returning relevant search results.

The most challenging part for me was the mathematical theory. Although I loved math, I sometimes had trouble connecting the theory to the application, and statistics is such an applied field. I look at it as a rite of passage and once I saw enough theory relevant to my interests, the learning process became easier.”

Q4: How do you apply what you learned in your statistics education in your current line of work?

A: “It is ironic, but it is the more basic concepts of statistics and probability that I use everyday rather than the complicated models I learned. Concepts such as independence, confidence, power, accuracy etc. are important building blocks for building my own models, or for choosing an existing one from those that I learned in school.

I always start with some exploratory analysis such as computing some statistics and making plots that show relationships clearly. Then I set explicit guidelines for the input and output of the model I want to build and note any critical assumptions that are violated or that must be met. I then try several different methods and models and validate their results using common metrics taught in undergraduate statistics before settling on a final model configuration.”

Q5: What skills did you learn in the statistics major that you find useful for work and everyday life?

A: “The training in mathematics I received as part of the statistics major taught me how to think logically, and this is very important in my work in computer science. I think patience was another very important skill I learned. I love what I do, and sometimes I take for granted that others have the same mathematical training that I do because I am so entrenched in it. Through my experience teaching as well as consulting as a student, I gained a better sense of the challenges and difficulties many people face when thinking about and interpreting statistics and how to better communicate results and ideas.”

Q6: Any advice for students who are considering majoring in statistics?

A: “My advice for students majoring in statistics is to choose an additional major or minor that uses statistics and is of interest to the student. I do not consider statistics to be a “standalone” major. When interviewing for a job, employers want to know why an interviewee is passionate about their company. For example, if interviewing for a finance company, the company wants to hear about passion for finance, and see education or experience in such fields. Another way to accomplish this instead of double majoring is to do some internships, projects or research in a field of interest.”

Conclusion: Finally, could you tell me a little about yourself for an intro bio we will include before the Q&A interview? For instance, what university(ies) did you attend, what degree(s) have you earned, what is your current job title, where do you work and for how long (you can be general here, or include a link to your professional website or blog if you have one), and what are you career goals?

A: “I attended University of California, Los Angeles (UCLA) for my B.S. (Statistics, Mathematics of Computation), two M.S. (Statistics and Computer Science) and Ph.D. (Statistics). I currently work for an Internet advertising startup in Santa Monica, CA as Chief Data Scientist/Research Engineer, and have been working in the field for three years. Whenever I get a free moment, I write about statistics, data mining and computer science topics on my blog at http://www.bytemining.com. I plan on dedicating the rest of my life working with and communicating about data and turning online phenomena into knowledge that can be used to progress technology and change the world!”

Look, if I am at a casino, I am well aware that the odds are against me, so why even try to think that I can use statistics to make money in this way? Although I love numbers and math, the stuff flows through my brain all day long (and night long), every day. If the goal is to enjoy and have fun, I do not want to sit there crunching probability formulas in my head (yes that’s fun, but it is also work). So that leaves me at the video Poker machines enjoying the free drinks. Another positive about video Poker is that $20 can sometimes last a few hours. So it should be no surprise that I do not agree with using Poker to teach probability.  Poker is an extremely superficial way to introduce such a powerful tool and gives the impression that probability is a way to make a quick buck, rather than as an important tool in science and society. The only [...]]]>

Whenever I tell people in my family that I study Statistics, one of the first questions I get from laypeople is “do you count cards?” A blank look comes over their face when I say “no.”

Look, if I am at a casino, I am well aware that the odds are against me, so why even try to think that I can use statistics to make money in this way? Although I love numbers and math, the stuff flows through my brain all day long (and night long), every day. If the goal is to enjoy and have fun, I do not want to sit there crunching probability formulas in my head (yes that’s fun, but it is also work). So that leaves me at the video Poker machines enjoying the free drinks. Another positive about video Poker is that $20 can sometimes last a few hours. So it should be no surprise that I do not agree with using Poker to teach probability.  Poker is an extremely superficial way to introduce such a powerful tool and gives the impression that probability is a way to make a quick buck, rather than as an important tool in science and society. The only time that I have used Poker in teaching (besides when required), is to cover the hypergeometric distribution and sampling without replacement.

Since I took Intro Probability Theory, I have always wondered what to do in the following situation. Say a pair of cruddy low cards appear on the first draw. The game only awards money for pairs of jacks or better. If all I have in the hand is a pair of low cards and no face cards, my decision is easy: hold the pair of low cards. But what if there is at least one face card showing (no other pairs)? Pictorially this looks like

The conundrum:

1. Hold the two low cards and deal, hoping for a three of a kind, or
2. Hold the two low cards AND one of the face cards, hoping for a three of a kind, OR a pair of Jacks of Better.

Under each of these decisions, which yields the highest probability of winning something and which one yields the highest payout? This problem can be solved exactly by using combinatorics, conditional probability and expectation, but since a video poker game is basically a simulator (though likely biased), I wrote my own simulation. For the answer, scroll to the end!

Data Structure

In most card games, we would want to store the state of the game: the outstanding cards in the deck(s), and the hand(s) of each player. In standard video poker, there is one deck, and one player, so only the player hand needs to be recorded because every card in the deck is either in the hand, or it is not. One obvious way to represent the hand is as an array of denomination/suit tuples in an array. Unfortunately, this data structure requires other data structures to store the possible suits, and possible denominations. It is also more tedious to detect certain kinds of wins. For this simulation, I use a 13 x 4 matrix where each row is a different denomination, and each column is each of the four suits. This matrix allows us to easily see which cards are possible to be dealt. Additionally, this matrix, as well as vector-based languages such as R, make it easy to detect wins. Such a matrix looks like the following for the hand 2♠ 5♣ 8♥ 8♣ A♦

         

where Cij denotes a card, i is the denomination and j is the suit and H is the player’s hand in question.

Poker wins are not disjoint. A three of a kind involving Jacks is also a pair of Jacks or better, etc. When checking wins, I start with the lowest paying win, and move up to Royal Flush, only keeping track of the highest win. Thus, this algorithm detects a four-of-a-kind involving Queens as Jacks or Better, two pairs of Queens, and a three-of-a-kind of Queens, but only counts it as the highest win, the four-of-a-kind.

1. Pair of Jacks or Better: a pair of Jacks, Queens, Kings or Aces. In A, this is simply the condition that at least one row in rows 10 through 13 has a row sum greater than 1.
2. Two pair: two pairs of anything. In A, this is the condition that at least two rows have a sum greater than 1.
3. Three of a kind: three of any card. In A, this is the condition that at least one row has a sum of at least 3.
4. Straight: all 5 cards can be permuted such that they form an ascending sequence: A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A. This case is interesting and will be discussed in a bit.
5. Flush: all 5 cards are of the same suit. In A, this is the condition that at least one column has a sum of at least 5.
6. Full House: one three-of-a-kind, and a pair of anything. In A, this is the condition that a row has sum 3, and another row has sum 2.
7. Four of a Kind: 4 of any card. In A, this is the condition that a row has sum 4.
8. Straight Flush: the 5 cards can be permuted to form an ascending sequence and are all of the same suit. In A, this is simply the condition that we have a straight and a flush in the same hand.
9. Royal Flush: a straight flush with the Ace as the high card. In A, this is simply the condition that we have a straight flush AND the sum of row 13 is 1.

Bug alert: It just occurred to me that there are many more wrap-around straights such as Q, K, A, 2, 3. This will be fixed this evening.

From basic computer science and number theory, every natural number can be written as the sum of distinct powers or 2 and the representation of such an integer is unique. Furthermore, the sum of n successive powers of 2 is divisible by . After some experimentation I came up with the following rule: if all of the row sums are 0/1 and the integer representation of this binary vector is divisible by , then A is a straight. The only straight that does not fit this pattern is the wrap-around straight: J, Q, K, A, 2 which can be checked manually.

1. Randomly generate a hand containing a pair of low cards (2-10) and at least one face card.
2. Hold the pair of low cards. Under strategy 2, hold one (and only one) of the face cards.
3. Discard the unheld cards from the deck and draw 2 or 3 cards at random from the same deck.
4. Check for wins.
5. Increment a win counter.
6. Repeat steps 1-5 tons of times, recording the percentage of hands that yielded a win, of the n games/hands played.

Results: Hold the Pair of Low Cards Only

My usual strategy is to always hold the low pair and take one face card along for the ride. That way, I hopefully match one of the two denominations I hold. My parents on the other hand, always told me to hold the low pair only, because that gives one more card (degree of freedom) for a win. It turns out they were right. Each game consisted of 1,000 hands. A percentage of these hands yields a win. This percentage is a random variable, so I ran this simulation to play 1,000 games. The table below shows the distribution of the win percentages.

Note that under strategy 1 (hold low pair only), all wins are more likely than under strategy 2! Of course, the estimate in the last column is an average; the mean in this case. The plot below shows the distribution of win percentages for both strategies. 

The Code

The code for my simulation is below. Note that it can easily be modified for your own target hands of interest. In my simulation, certain functions were never used because certain winning hands were not possible. 

DISCLAIMER: I did this for fun, and it is possible that there are bugs or problems with my code, algorithm or simulation. The results seem correct because I empirically I seem to do about the same using either strategy, and in a gambling perspective, an 8% discrepancy is not likely to set off bells in the head.

The above resources are available not only for Wikipedia, but for other Wikimedia Foundation projects such as Wiktionary, Wikibooks and Wikiquotes.

As Wikipedia readers will notice, the articles are very well formatted and this formatting is generated by a somewhat unusual markup format defined by the MediaWiki project. As Dirk Riehl stated:

There was no grammar, no defined processing rules, and no defined output like a DOM tree based on a well defined document object model. This is to say, the content of Wikipedia is stored in a format that is not an open standard. The format is defined by 5000 lines of php code (the parse function of MediaWiki). That code may be open source, but it is incomprehensible to most. That’s why [...]]]>

Lately I have doing a lot of work with the Wikipedia XML dump as a corpus. Wikipedia provides a wealth information to researchers in easy to access formats including XML, SQL and HTML dumps for all language properties. Some of the data freely available from the Wikimedia Foundation include article content and template pages article content with revision history (huge files) article content including user pages and talk pages redirect graph page-to-page link lists: redirects, categories, image links, page links, interwiki etc. image metadata site statistics

The above resources are available not only for Wikipedia, but for other Wikimedia Foundation projects such as Wiktionary, Wikibooks and Wikiquotes.

As Wikipedia readers will notice, the articles are very well formatted and this formatting is generated by a somewhat unusual markup format defined by the MediaWiki project. As Dirk Riehl stated:

There was no grammar, no defined processing rules, and no defined output like a DOM tree based on a well defined document object model. This is to say, the content of Wikipedia is stored in a format that is not an open standard. The format is defined by 5000 lines of php code (the parse function of MediaWiki). That code may be open source, but it is incomprehensible to most. That’s why there are 30+ failed attempts at writing alternative parsers.

For example, below is an excert of Wiki-syntax for a page on data mining.

'''Data mining''' (the analysis step of the '''knowledge discovery in databases''' process,<ref name="Fayyad"> or KDD), 
a relatively young and interdisciplinary field of [[computer science]]<ref name="acm" />
{{cite web|url=http://www.sigkdd.org/curriculum.php |title=Data Mining Curriculum |
publisher=[[Association for Computing Machinery|ACM]] [[SIGKDD]] |date=2006-04-30 |accessdate=2011-10-28}}
</ref><ref name=brittanica>{{cite web | last = Clifton | first = Christopher | title = Encyclopedia Britannica: Definition 
of Data Mining | year = 2010 | url = http://www.britannica.com/EBchecked/topic/1056150/data-mining | 
accessdate = 2010-12-09}}</ref> is the process of discovering new patterns from large [[data set]]s 
involving methods at the intersection of [[artificial intelligence]], [[machine learning]], [[statistics]] and 
[[database system]]s.<ref name="acm"> The goal of data mining is to extract knowledge from a data set in a 
human-understandable structure<ref name="acm" /> and involves database and [[data management]], 
[[Data Pre-processing|data preprocessing]], [[statistical model|model]] and [[Statistical inference|inference]] 
considerations, interestingness metrics, [[Computational complexity theory|complexity]] considerations, post-processing 
of found structure, [[Data visualization|visualization]] and [[Online algorithm|online updating]].<ref name="acm" />

I was epicly worried that I would spend weeks writing my own parser and never complete the project I am working on at work. To my surprise, I found a fairly good parser. Since I am working on named entity extraction and ngram extraction, I wanted to only extract the plain text. If we take the above junk and extract only the plain text, we would get 

Data mining (the analysis step of the knowledge discovery in databases process, or KDD), a relatively young 
and interdisciplinary field of computer science is the process of discovering new patterns from large data sets 
involving methods at the intersection of artificial intelligence, machine learning, statistics and database systems. 
The goal of data mining is to extract knowledge from a data set in a human-understandable structure and involves 
database and data management, data preprocessing, model and inference considerations, interestingness
metrics, complexity considerations, post-processing of found structure, visualization and online updating.

and from this we can remove punctuation (except sentence terminators .?!), convert to lower case and perform other pre-processing text mining steps. There are many, many Wikipedia parsers of various qualities. Some do not work at all, some work only on certain articles, some have been abandoned as incomplete and some are slow as molasses.

I was delighted to stumble upon Wikipedia Extractor, a Python library developed by Antonio Fuschetto, Multimedia Laboratory, Dipartimento di Informatica, Università di Pisa, that extracts plain-text from the Wikipedia XML dump file. The script is heavily object-oriented, and it is very easy to modify and extend for other purposes. For me, it is the easiest parser to use and yields the best quality output although there are other options.

Pros

• Very easy to run; it’s just a Python script.
• Yields high quality output; no stray wikisyntax garbage.
• Highly object-oriented; easy to extend and embed in text mining projects.
• Object-oriented style makes it easier to parallelize with lightweight processes (written by the user).
• Allows specifying the maximum size of each produced file (good for sending to S3).
• It is written in Python.

Cons

• Far too slow. Python profilers show major overhead involved in regex search and replace, and string replacement.
• Is not perfect, but one of the best I have seen. For some reason, Wikilinks are converted to HTML links. Correcting this required modifying the source code.
• Retooling the package to work with Hadoop Streaming is not too difficult, but requires some work and grokery that should be easier.

Wikipedia Extractor is good for offline analysis, but users will probably want something that runs faster. Wikipedia Extractor parsed the entire Wikipedia dump in approximately 13 hours, on one core, which is quite painful. Add in further parsing and the processing time becomes unbearable even on multiple cores. A Hadoop Streaming job using Wikipedia Extractor as well as too much file I/O between Elastic MapReduce and S3 required 10 hours to complete on 15 c1.medium instances. 

Ken Weiner (@kweiner) recently re-introduced me to the Cloud9 package by Jimmy Lin (@lintool) of Twitter which fills in some of these gaps. I avoided it at first because Java is not the first language I like to turn to. Cloud9 is written in Java and designed for use with Hadoop MapReduce in mind. There is a method within the package that explicitly extracts the body text of each Wikipedia article. This method calls the Bliki Wikipedia parsing library. One common problem with these Wikipedia parsers is that they often leave syntax still in the output. Jimmy seems to wrap Bliki with his own code to do a better job of extracting high quality text only output. Cloud9 also has counters and functions that detect non-article content such as redirects, disambiguation pages, and more.

Developers can introduce their own analysis, text mining and NLP code to process the article text in the mapper or reducer code. An example job distributed with Cloud9 which simply counts the number of pages in the corpus took approximately 15 minutes to run on 8 cores on an EC2 instance. A job that did more substantial required 3 hours to complete, and once the corpus was refactored as sequence files, the same job took approximately 90 minutes to run.

Conclusion

I am looking forward to playing with Cloud9 some more… I will take 90 minutes over 10 hours any day! Wikipedia Extractor is an impressive Python package that does a very good job of extracting plain text from Wikipedia articles and for that I am grateful. Unfortunately, it is far too slow to be used on a pay-per-use system such as AWS or for quick processing. Cloud9 is a Java package designed with scalability and MapReduce in mind, allowing much quicker and more wallet friendly processing.

LexisNexis has open-sourced its alternative to Hadoop, called High Performance Computing Cluster. The code is available on GitHub. For years the code was restricted to LexisNexis Risk Solutions. The system contains two major components:

The protocol that drives the whole process is the Enterprise Control Language which is said to be faster and more efficient than Hadoop’s version of MapReduce. A picture is a much better way to show how the system works. Below is a diagram from the Gigaom article from which most of this information originates.

To me, Roxie seems much more exciting because it seems to complement (or replace) several technologies currently in the space. I do not know all the details, but it seems to potentially encapsulate technologies such as HBase, Hive, RabbitMQ and MemcacheDB, technologies that are common used to query and [...]]]>

A month ago, I wrote about alternatives to the Hadoop MapReduce platform and HPCC was included in that article. For more information, see here.

LexisNexis has open-sourced its alternative to Hadoop, called High Performance Computing Cluster. The code is available on GitHub. For years the code was restricted to LexisNexis Risk Solutions. The system contains two major components:

• Thor (Thor Data Refinery Cluster) is the data processing framework. It “crunches, analyzes and indexes huge amounts of data a la Hadoop.”
• Roxie (Roxy Radid Data Delivery Cluster) is more like a data warehouse and is designed with quick querying in mind for frontends.

The protocol that drives the whole process is the Enterprise Control Language which is said to be faster and more efficient than Hadoop’s version of MapReduce. A picture is a much better way to show how the system works. Below is a diagram from the Gigaom article from which most of this information originates.

To me, Roxie seems much more exciting because it seems to complement (or replace) several technologies currently in the space. I do not know all the details, but it seems to potentially encapsulate technologies such as HBase, Hive, RabbitMQ and MemcacheDB, technologies that are common used to query and speed data to a web frontend.

My opinion on HPCC is mixed. Although Hadoop has already taken off in usage, LexisNexis is a very strong institution and could potentially convince some corporate users to use their system instead — those that do not want to use Microsoft’s Dryad project. I do not see HPCC being a Hadoop killer, just as I do not see Spark or any other alternative to be a Hadoop killer. However, if HPCC does become a strong alternative, I sense this could be trouble for some of the newer players in the Hadoop field such as HortonWorks and MapR. I do not have much of an interest in studying business and competition, but Hadoop Summit 2011 showed that the Hadoop space has become crowded, and small breakthroughs such as another company developing a similar project is enough to add volatility and uncertainty for all involved.

<< My review of Day 1.

I am summarizing all of the days together since each talk was short, and I was too exhausted to write a post after each day. Due to the broken-up schedule of the KDD sessions, I group everything together instead of switching back and forth among a dozen different topics. By far the most enjoyable and interesting aspects of the conference were the breakout sessions.

Keynotes

KDD 2011 featured several keynote speeches that were spread out among three days and throughout the day. This year’s conference had a few big names.

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year. AdMeld did an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!

Mining and Learning on Graphs Workshop 2011

I had originally planned to attend the 2-day workshop Mining and Learning with Graphs (MLG2011) but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first talk I arrived for was Networks Spill the Beans by Lada Adamic from the University of Michigan. Adamic’s presented work involved inferring properties of content (the “what”) using network structure alone (using only the “who”: who shares with whom). One example she presented involved questions and answers on a Java programming language forum. The research problem was to determine things such as who is most likely to answer a Java beginner’s question: a guru, or a slightly more experienced user? Another research question asked what dynamic interactions tell us about information flow. [...]]]>

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year. AdMeld did an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!

Mining and Learning on Graphs Workshop 2011

I had originally planned to attend the 2-day workshop Mining and Learning with Graphs (MLG2011) but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first talk I arrived for was Networks Spill the Beans by Lada Adamic from the University of Michigan. Adamic’s presented work involved inferring properties of content (the “what”) using network structure alone (using only the “who”: who shares with whom). One example she presented involved questions and answers on a Java programming language forum. The research problem was to determine things such as who is most likely to answer a Java beginner’s question: a guru, or a slightly more experienced user? Another research question asked what dynamic interactions tell us about information flow. For this example, Adamic used data from the virtual world SecondLife. Certain landmarks (such as a bench) can be bookmarked by users and certain gestures (like a kiss) can be studied. This made my ears rise. SecondLife is a treasure trove of cool data. Is there a way to access it? It looks there might be a way to access some of it including monetary valuation, market purchases, and several APIs for different aspects of SecondLife.  I will have to look into that later though. Adamic concluded with a discussion of Twitter as a social network, but I was starting to fall asleep from my hectic and early morning. The gist of her talk, and many other talks in this field, was to combine semantic variables (NLP) with topological variables (SNA) to predict som other semantic variables. This talk was very digestible, and very interesting (despite my lack of sleep), but featured some of the worst visualizations I have ever seen (area plots representing correlations across multiple levels of an ordinal variable), but that was minor. Of course, Nathan might disagree .

Social network analysis, and network analysis in general, is a field that I really want to sink my teeth into. The difficulty I have is that the discussion of this field seems to involve so much vernacular that is specific to the field that everything seems so much more difficult than it really is.

At this point I took off to lunch. Just across from the Hyatt (a beautiful hotel by the way) is Seaport Village, a beautiful waterfront park containing nice landscaping, shops, restaurants, all with the ocean in the background. There is no beach there — the village backs right up to the water. Across the bay is some type of military complex and Coronado Island. I had a $7 hot dog, followed by a chocolate-covered strawberry and a peanut butter cup from the nearby candy store. It was such a nice day I walked around for a while, grabbed a strawberry shake and then headed back for the next session… the one I had been waiting for!

Afternoon Tutorial: Probabilistic Topic Models
David Blei, Princeton

My dissertation topic is related to Latent Dirichlet allocation (well, topic modeling in general), so I was definitely interested to hear what the father of LDA had to say. Since this was a 3 hour tutorial, I was expecting that Blei would start with the unigram model, and then discuss Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Indexing (pLSI) building up to LDA. Instead, Blei started with LDA and for good reason! In this post, I will not summarize the mechanics of Latent Dirichlet Allocation as that is another post entirely. For some introduction, see here. LDA and its extensions can be used to model the evolution of topics over time, to model the connections among topics, and to predict links among objects in a network. Topic modeling is a case study in machine learning rather than a field in itself; topic modeling draws on several different concepts including Bayesian statistics, time series analysis, hierarchical models, Markov chain monte carlo (MCMC), Bayesian non-parametric statistics and sparsity. In LDA, a document is represented as a mixture of topics (some hypothetical quantity that captures content clustering), and a topic is a distribution over words in a vocabulary.

Again, this is a high-level description of what was discussed. A full mechanical analysis would require dozens of pages. LDA is just a probabilistic model. As such, there are established ways for estimating the parameters of the model as well as the topic assignments. Some of these include mean field variational methods, expectation propagation, Gibbs sampling, collapsed Gibbs sampling, collapsed variational Bayes and online variational Bayes. Each of these estimation methods has its own advantages and disadvantages. Blei showed the LDA and pLSI have a lot in common. Unlike LDA, pLSI uses maximum likelihood estimations (and the EM algorithm) for parameter estimation; pLSI tends to overfit badly. The hyperparameter α adds regularization to the ϴ parameter in the LDA model. [Sorry to refer to these random parameters, but it is difficult to describe without them. See the links mentioned earlier for an overview of LDA.]

Preprocessing. A lot of preprocessing must be performed before computing a topic model. First, we should remove stopwords, which are words that provide absolutely no clues to the content of the text. If we leave stopwords in the corpus when computing the model, we may end up with meaningless topics that are described with only stopwords, due to their high probability. Second, Blei mentioned that stemming is a good idea, but modern stemming algorithms tend to be too aggressive. If resources allow, I think it would be useful to have humans manually strip words to their root words. Multiword phrases such as “black hole” are also an issue. With sufficient resources, one could ask human labelers to identify these phrases and recode them as a single word by replacing the space between words with an underscore. Hanna Wallach (U. Mass) has a paper that describes how to identify multiwork phrases by using n-grams. Blei has a similar paper that discusses an algorithm called TurboTopics. He also mentioned that a standard statistical hypothesis test such as chi-squared, permutation tests, or a nested hypothesis test would also be sufficient, though inefficient. I have not thought of how this would work however. Finally, remove rare words because they can lead to local optima in the likelihood surface probably yielding inefficient computation.

Some hairy details. One of the parameters that makes LDA useful is  α. α is a hyperparameter in the LDA model that determines the sparsity of draws from the underlying Dirichlet distribution. α is typically a small number; Blei mentioned that 0.01 is a good a priori value for α. As α gets larger, the distribution of topics tends towards the uniform (each topic equally likely) distribution and as α approaches 0, we get sparser draws, meaning more peaked topic probabilities. Setting α to be ridiculously small (i.e. 0.001) may yield a single topic dominating the model. α can be chosen, or we can fit α to the data using cross-validation or some other method. He also discussed the parameter η.

Open source software. We quickly (flash of an eye) went through a list of some open-source LDA implementations:

• LDA-C (variational EM), Blei.
• HDP (hierarchical Dirichlet processes), Blei
• LDA (R package, collapsed Gibbs), Jonathan Chang, Data Scientist, Facebook.
• Lingpipe, alias-i
• Mallet (collapsed Gibbs), UMass
• Gensim (online and batch LDA), Radim Řehůřek

To my delight, Blei seemed to favor the R package (although Gensim is a nice Python implementation). The R package not only contains LDA, but several other models including RTMs, MMSB and sLDA which will be discussed later. It is supposedly fast as well. The output from the R package can be visualized using the Topic Model Visualizer by Allison Chaney.

The beauty of LDA is that it can be embedded in many more complicated models. Some applications of these extensions include word sense, graphs and hierarchies. Before delving into specifics, there are a couple of changes to the LDA model that motivate the next topics.

1. The probability of observing word w given a set of topics β and a set of topic labels z is given by P(w|β,z) which is multinomial. The distribution of P(w|ß,z) can be changed depending on what we are modeling. For example, for count data, P(w|β,z) can be Poisson. This drastically changes the model, however. In LDA, P(w|ß,z) is multinomial which is convenient because it is the conjugate prior of the Dirichlet distribution.
2. The characteristic LDA posterior distribution can be used in more creative ways…

Correlated Topic Model. In LDA, all topics are considered independent of each other, and this is usually unrealistic. CTM allows the topics to be correlated. For example, a paper classified as about calculus is more likely to also be classified as about physics, than it is to be classified as about sewing. Blei mentioned that CTM allows for better prediction, likely because it is more realistic. CTM is also more robust to overfitting. The main distinction from LDA is that ϴ follows the logistic normal distribution instead of the Dirichlet distribution.

Dynamic Topic Model. DTM models how each individual topic changes over time. One example Blei showed involved a topic that could be labeled “technology”. In the late 1700s, this topic contained the words “coal”, “steel” (I am making it up from memory…probably badly…bear with me) and in 2011 contained the words “silicon” and “solar”. The main distinction from LDA is two-fold: assuming the topic at time t is normally distributed with the topic at time t-1 as the mean and some variance. That is,

and

instead of multinomial.

A limitation of DTM is that it does not handle the death of a topic gracefully.

Supervised LDA. In sLDA, we associate each document with an external variable. For example, a document may be a Yelp review containing text. The external variable associated with the Yelp review may be the number of stars in the associated rating. We can use sLDA to use the topics estimated by LDA as regressors to predict this external variable Y. Various types of regression can be performed from standard linear regression to the generalized linear model (GLM).The Yelp example would likely use an ordered logit model for Y.

Relational Topic Models. RTM applies sLDA to every pair of documents in a corpus and attempts to use content to predict connectedness in a graph. For example, given the content on my Facebook profile, one could use sLDA to predict what kind of reaction I would have to an ad (i.e. click or no click) and this could be used for targeted ad serving, or any other type of recommendation engine. Think collaborative filtering! RTM is also good for certains types of data that have spatial/geographic dependencies.

Ideal Point Topic Models were barely touched upon since we were running short on time (although we voted to extend the session by 30 mins and Blei happily obliged). They seem particularly useful in political science for predicting roll call votes.

Bayesian Non-Parametric Models are a hot topic but are too complicated to describe here. In LDA, the number of topics is determined a priori and remains fixed throughout the model. In real life, topics can be “born” and can “die” off and we may not know a priori how many topics to use. One can model the latter situation as a Chinese Restaurant Process where each table is associated with a topic. Furthermore, a Chinese Restaurant Franchise can be used for modeling hierarchies (hLDA). In CRF, there is a corpus level restaurant where each table is a parameter and a topic (called plates). Then, each document has its own Chinese restaurant where each table is associated with a customer in the corpus level Chinese restaurant. Blei recommended a book by Hjort.

Algorithms. The last few minutes were dedicated to discussing inference algorithms for LDA, particularly Gibbs sampling and variational Bayes. Gibbs sampling is very simple to implement, though Blei stated that it does not work for DTM or CTM because the assumptions of conjugacy (multinomial/Dirichlet) are violated. Variational Bayes is more difficult to implement, but handles non-conjugacy in CTM and DTM much better.

Plenary Sessions

The plenary sessions consisted of several thank-yous and awards. The committee provided some humor which gave some humility to the long process of writing and submitting papers. They went over paper acceptance statistics and read some of the funnier comments that reviewers gave, one of which was something like “It is clear that the author did not read this paper before submitting it.” I don’t know how many times I have said that in various situations. The committee handed out awards for best paper and best dissertation. This year’s KDD Cup competition was a contest similar to the Netflix challenge, but involved music recommendation. The winner was the National Taiwan University, for the fourth straight year in a row I am told. The innovation award went to a researcher dear to my heart, Ross Quinlan, who developed the C4.5 decision tree modeling software.

For more information about topic modeling software, see David Blei’s website at http://www.cs.princeton.edu/~blei which contains code for most if not all of these topic models. For the notes from the tutorial, see http://www.cs.princeton.edu/~blei/kdd-tutorial.pdf.

Hadoop has become the de facto standard in the research and industry uses of small and large-scale MapReduce. Since its inception, an entire ecosystem has been built around it including conferences (Hadoop World, Hadoop Summit), books, training, and commercial distributions (Cloudera, Hortonworks, MapR) with support. Several projects that integrate with Hadoop have been released from the Apache incubator and are designed for certain use cases:

Hadoop is meant to be modeled after Google MapReduce. To store and process huge amounts of data, we typically need several machines in some cluster configuration. A distributed filesystem (HDFS for Hadoop) uses space across [...]]]>

It’s been a while since I have posted… in the midst of trying to plow through this dissertation while working on papers for submission to some conferences.

Hadoop has become the de facto standard in the research and industry uses of small and large-scale MapReduce. Since its inception, an entire ecosystem has been built around it including conferences (Hadoop World, Hadoop Summit), books, training, and commercial distributions (Cloudera, Hortonworks, MapR) with support. Several projects that integrate with Hadoop have been released from the Apache incubator and are designed for certain use cases:

• Pig, developed at Yahoo, is a high-level scripting language for working with big data and Hive is a SQL-like query language for big data in a warehouse configuration.
• HBase, developed at Facebook, is a column-oriented database often used as a datastore on which MapReduce jobs can be executed.
• ZooKeeper and Chukwa
• Mahout is a library for scalable machine learning, part of which can use Hadoop.
• Cascading (Chris Wensel), Oozie (Yahoo) and Azkaban (LinkedIn) provide MapReduce job workflows and scheduling.

Hadoop is meant to be modeled after Google MapReduce. To store and process huge amounts of data, we typically need several machines in some cluster configuration. A distributed filesystem (HDFS for Hadoop) uses space across a cluster to store data so that it appears to be in a contiguous volume and provides redundancy to prevent data loss. The distributed filesystem also allows data collectors to dump data into HDFS so that it is already prime for use with MapReduce. A Data Scientist or Software Engineer then writes a Hadoop MapReduce job.

As a review, the Hadoop job consists of two main steps, a map step and a reduce step. There may optionally be other steps before the map phase or between the map and reduce phases. The map step reads in a bunch of data, does something to it, and emits a series of key-value pairs. One can think of the map phase as a partitioner. In text mining, the map phase is where most parsing and cleaning is performed. The output of the mappers is sorted and then fed into a series of reducers. The reduce step takes the key value pairs and computes some aggregate (reduced) set of data such as a sum, average, etc. The trivial word count exercise starts with a map phase where text is parsed and a key-value pair is emitted: a word, followed by the number “1″ indicating that the key-value pair represents 1 instance of the word. The user might also emit something to coerce Hadoop into passing data into different reducers. The words and 1s are sorted and passed to the reducers. The reducers take like key-value pairs and compute the number of times the word appears in the original input.

After working extensively with (Vanilla) Hadoop professional for the past 6 months, and at home for research, I have found several nagging issues with Hadoop that have convinced me to look elsewhere for everyday use and certain applications. For these applications, the though of writing a Hadoop job makes me take a deep breath. Before I continue, I will say that I still love Hadoop and the community.

• Writing Hadoop jobs in Java is very time consuming because everything must be a class, and many times these classes extend several other classes or extend multiple interfaces; the Java API is very bloated. Adding a simple counter to a Hadoop job becomes a chore of its own.
• Documentation for the bloated Java API is sufficient, but not the most helpful.
• HDFS is complicated and has plenty of issues of its own. I recently heard a story about data loss in HDFS just because the IP address block used by the cluster changed.
• Debugging a failure is a nightmare; is it the code itself? Is it a configuration parameter? Is it the cluster or one/several machines on the cluster? Is it the filesystem or disk itself? Who knows?!
• Logging is verbose to the point that finding errors is like finding a needle in a haystack. That is, if you are even lucky to have an error recorded! I’ve had plenty of instances where jobs fail and there is absolutely nothing in the stdout or stderr logs.
• Large clusters require a dedicated team to keep it running properly, but that is not surprising.
• Writing a Hadoop job becomes a software engineering task rather than a data analysis task.

Hadoop will be around for a long time, and for good reason. MapReduce cannot solve every problem (fact), and Hadoop can solve even fewer problems (opinion?). After dealing with some of the innards of Hadoop, I’ve often said to myself “there must be a better way.” For large corporations that routinely crunch large amounts of data using MapReduce, Hadoop is still a great choice. For research, experimentation, and everyday data munging, one of these other frameworks may be better if the advantages of HDFS are not necessarily imperative:

BashReduce

Unlike Hadoop, BashReduce is just a script! BashReduce implements MapReduce for standard Unix commands such as sort, awk, grep, join etc. It supports mapping/partitioning, reducing, and merging. The developers note that BashReduce “sort of” handles task coordination and a distributed file system. In my opinion, these are strengths rather than weaknesses. There is actually no task coordination as a master process simply fires off jobs and data. There is also no distributed file system at all, but BashReduce will distribute files to worker machines. Of course, without a distributed file system there is a lack of fault-tolerance among other things.

Intermachine communication is facilitated with simple passwordless SSH, but there is a large cost associated with transferring files from a master machine to its workers whereas with Hadoop, data is stored centrally in HDFS. Additionally, partition/merge in the standard unix tools is not optimized for this use case, thus the developer had to use a few additional C programs to speed up the process.

Compared to Hadoop, there is less complexity and faster development. The result is the lack of fault-tolerance, and lack of flexibility as BashReduce only works with certain Unix commands. Unlike Hadoop, BashReduce is more of a tool than a full system for MapReduce. BashReduce was developed by Erik Frey et. al. of last.fm.

Disco Project

Disco was initially developed by Nokia Research and has been around silently for a few years. Developers write MapReduce jobs in simple, beautiful Python. Disco’s backend is written in Erlang, a scalable functional language with built-in support for concurrency, fault tolerance and distribution — perfect for a MapReduce system! Similar to Hadoop, Disco distributes and replicates data, but it does not use its own file system. Disco also has efficient job scheduling features.

It seems that Disco is a pretty standard and powerful MapReduce implementation that removes some of the painful aspects of Hadoop, but it also likely removes persistent fault tolerance as it relies on a standard filesystem rather than one like HDFS, but Erlang may impose some functionality that provides a “good enough” level of fault tolerance for data.

Spark

Spark is one of the newest players in the MapReduce field. Its purpose is to make data analytics fast to write, and fast to run. Unlike many MapReduce systems, Spark allows in-memory querying of data (even distributed across machines) rather than using disk I/O. It is of no surprise then that Spark out-performs Hadoop on many iterative algorithms. Spark is implemented in Scala, a functional object-oriented language that sits on top of the JVM. Similar to other languages like Python, Ruby, and Clojure, Scala has an interactive propt and users can use Spark to query big data straight from the Scala interpreter.

One wrinkle is that Spark requires installing a cluster manager called Mesos. I had some difficulty installing it on Ubuntu, but the development team was an amazing help, and made a few changes to the source and now it runs well. On the downside, Mesos adds a layer of complexity that we are trying to avoid. On the upside, Mesos allows Spark to co-exist with Hadoop and it can read any data source that Hadoop supports, and it “feels” light, similar to Disco’s server UI.

Spark was developed by the UC Berkeley AMP Lab. Currently, its main users are UC Berkeley researchers and Conviva. Hadoop Summit 2011 featured a talk on Spark by one of the developers, which I wrote about earlier this summer.

GraphLab

GraphLab was developed at Carnegie Mellon and is designed for use in machine learning. GraphLab’s goal is to make the design and implementation of efficient and correct parallel machine learning algorithms easier. Their website states that paradigms like MapReduce lack expressiveness while lower level tools such as MPI present overhead by requiring the researcher to write code that beats a dead horse.

GraphLab has its own version of the map stage, called the update phase. Unlike MapReduce, the update phase can both read and modify overlapping sets of data. Recall that MapReduce requires data to be partitioned. GraphLab accomplishes this by allowing the user to specify data as a graph where each vertex and edge in the graph is associated memory. The update phases can be chained in such a way such that one update function can recursively trigger other update functions that operate on vertices in the graph. This graph-based approach would not only make machine learning on graphs more tractable, but it also improves dynamic iterative algorithms.

GraphLab also has its own version of the reduce stage, called the sync operation. The results of the sync operation are global and can be used by all vertices in the graph. In MapReduce, output from the reducers is local (until committed) and there is a strict data barrier among reducers. The sync operations are performed at time intervals, and there is not as strong of a tie between the update and sync phases. What I mean is that the sync intervals are not necessarily dependent on some prior update completing.

GraphLab’s website also contains the original UAI paper and presentation, a document better explaining the abstraction, and there is even a Google Group for the GraphLab API. To me, GraphLab seems like a very powerful generalization, and re-specification, of MapReduce.

Storm

Recently, Nathan Marz of BackType made waves in the Twitter big data community with a blog post titled Preview of Storm: The Hadoop of Realtime Processing. Within a day, Storm became known as “Real-time Hadoop” to the chagrin of some developers from Apache. Hadoop is a batch-processing system — that is, give it a lot of fixed data and it does something with it. Storm is real-time — it processes data in parallel as it streams.

Marz writes that with their previous system, much time was spent worrying about graphs of queues and workers: where to send and receive messages, deploying workers and queues, and a lack of fault tolerance. Storm abstracts all of these complications away. Storm is written in Clojure, but any programming language can be used to write programs on top of Storm. Storm is fault-tolerant, horizontally scalable, and reliable. Storm is also very fast, with ZeroMQ used as the underlying message passing system.

Nathan Marz is a software developer at BackType, and made waves in 2010 with Cascalog. Cascalog really took off after his presentation at the 2010 Hadoop Summit, and I am delighted I got to see him present it. Storm will be open-sourced soon and I hope to write more about it later.

I included Storm in this post based on its colloquial name “Real-time Hadoop” — it is not clear to me whether or not Storm even uses MapReduce though.

HPCC Systems (from LexisNexis)

Perhaps the project with the least flattering name comes from LexisNexis, which has developed its own framework for massive data analytics. HPCC attempts to make writing parallel-processing workflows easier by using Enterprise Control Language (ECL), a declarative, data-centric language. I should note that SQL, Datalog and Pig are also said to be declarative, data-centric languages. A matter of fact, the development team has a converter for translating Pig jobs to ECL. HPCC is written in C++. Some have commented that this will make in-memory querying much faster because there is less bloated object sizes originating from the JVM. I also prefer C++ simply because it feels closer to human though — we think in terms of objects (object-oriented) at times, and a series of steps (procedural) at other times and use both thought processes together.

HPCC already has its own jungle of technologies like Hadoop. HPCC has two “systems” for processing and serving data: the Thor Data Refinery Cluster, and the Roxy Rapid Data Delivery Cluster. Thor is a data processor, like Hadoop. Roxie is similar to a data warehouse (like HBase) and supports transactions. HPCC uses a distributed file system.

Although details are still preliminary as is the system, this certainly has a “feel” for potentially being a solid alternative for Hadoop, but only time will tell.

With all these alternatives, why use Hadoop?

One word: HDFS. For a moment, assume you could bring all of your files and data with you everywhere you go. No matter what system, or type of system, you login to, your data is intact waiting for you. Suppose you find a cool picture on the Internet. You save it directly to your file store and it goes everywhere you go. HDFS gives users the ability to dump very large datasets (usually log files) to this distributed filesystem and easily access it with tools, namely Hadoop. Not only does HDFS store a large amount of data, it is fault tolerant. Losing a disk, or a machine, typically does not spell disaster for your data. HDFS has become a reliable way to store data and share it with other open-source data analysis tools. Spark can read data from HDFS, but if you would rather stick with Hadoop, you can try to spice it up:

Hadoop Streaming is an easy way to avoid the monolith of Vanilla Hadoop without leaving HDFS, and allows the user to write map and reduce functions in any language that supports writing to stdout, and reading from stdin. Choosing a simple language such as Python for Streaming allows the user to focus more on writing code that processes data rather than software engineering. Once code is written, it is easy to test from the command line:

cat a_bunch_of_files | ./mapper.py | sort | ./reducer.py

And, running and monitoring the job is similar to Vanilla Hadoop. Hadoop Streaming was my first introduction to Hadoop and it was quite pleasant.

Or, you could use a Hadoopified project that better solves the problem. Vanilla Hadoop can do some sophisticated stuff, but it suffers the problems I mentioned at the beginning of the post. Developers have created software that works on HDFS, but is geared toward different audiences. A Data Scientist may prefer Pig or Hive for data analysis whereas a Systems and Software Engineer may prefer a workflow solution (Oozie, Cascading etc.) and a (modern) DBA may want to use HBase. Each of these achieve different goals, but still rely on HDFS.

This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.

Keynotes

We first heard a series of keynote speeches which I will summarize. The first keynote was from Jay Rossiter, SVP of the Cloud Platform Group at Yahoo. He introduced how Hadoop is used at Yahoo, which is fitting since they organized the event. The content of his presentation was very similar to last year’s. One interesting application of Hadoop at Yahoo was for “retiling” the map of the United States. I imagine this refers to the change in aerial imagery over time. When performed by hand, retiling took 6 weeks; with Hadoop, it took 5 days. Yahoo also [...]]]>

I woke up early and cheery Wednesday morning to attend the 2011 Hadoop Summit in Santa Clara, after a long drive from Los Angeles and the Big Data Camp that lasted until 10pm the night before. Having been to Hadoop Summit 2010, I was interested to see how much of the content in the conference had changed.

This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.

Keynotes

We first heard a series of keynote speeches which I will summarize. The first keynote was from Jay Rossiter, SVP of the Cloud Platform Group at Yahoo. He introduced how Hadoop is used at Yahoo, which is fitting since they organized the event. The content of his presentation was very similar to last year’s. One interesting application of Hadoop at Yahoo was for “retiling” the map of the United States. I imagine this refers to the change in aerial imagery over time. When performed by hand, retiling took 6 weeks; with Hadoop, it took 5 days. Yahoo also uses Hadoop for fraud detection, spam detection, search assist, geotagging data/local indexing, ad targeting, predicting supply and demand and the aggregation and categorization of news stories. Jay also mentioned that Dapper runs models on data with Hadoop for ad personalization. Jay also mentioned that Big Data conferences all over the country are selling out.

Eric Baldeschwieler, the CEO of Hortonworks was next. Hortonworks seems to be a new company that spun off from Yahoo. Their goal is to provide commercial support and a full Apache Hadoop platform for users. Yes, they are very similar to Cloudera, and yes, they are competition. (Hortonworks and MapR both did a good job of not stepping on everyone’s toes in terms of how they presented themselves.) Cloudera provides its own distribution of Hadoop, which is of course similar to the Apache version. Hortonworks’ goal is to provide similar services, but with more transparency by using the Apache Hadoop distribution rather than wrapping its own. Paraphrasing Eric, Hortonworks is open-source from the ground up. A bit later, Sanjay Radia also of Hortonworks discussed Hadoop for the enterprise. Hortonworks has contributed, or is working on security (preventing users from deleting others’ data), service level agreements (SLAs), predictability and a Fair-Share scheduler.

Anant Jhingran, CTO of IBM discussed how Hadoop was used in IBM Watson. It seemed pretty obvious that Hadoop or some form of map-reduce was used in the system, but it did not seem to be highly publicized. Watson learned from 200 million pages of data, about 2-5TB and required between 3000 and 4000 Watts. Anant went quickly through a cool user interface representing a Jeopardy board and stated that the user interface to an artificial intelligence application is just important as the application itself. He also prefers the term IA (intelligence augmentation) over AI, and apparently this is a common distinction. To me, I interpret AI vs. IA to be artificial intelligence vs. knowledge discovery (data mining).

Karthic Ranganathan from Facebook discussed Facebook’s messaging system which was built on HBase, HDFS and MapReduce. Facebook sees 15 billion messages per month, excluding SMS and email, approximately 14TB of data! There are also 120 billion chat messages (25TB), for a grand total of almost 300TB per month. (I may have missed something as these numbers do not add up). Facebook uses HBase for the bodies of small messages, metadata, and for the search index. Facebook uses HBase because of its high write throughput and easy horizontal scalability. Facebook uses another system called Haystack for photos, bodies of large messages and attachments. Of course, HDFS is used for fault tolerance, scalability, checksums for data integrity and its MapReduce abilities. Profiles and services are partitioned by user. Each machine has 16 cores, with 12 1TB hard disks, and 48GB RAM (24GB used for HBase). Some things that Facebook would like contribute and improve: NameNode high availability and a second NameNode, better performance overall, and using flash memory to improve performance. Facebook often adds several columns to a table so that DevOps does not need to take the server offline to add new columns.

Big Data conferences all over the country are selling out.

Breakout Sessions

There were so many great sessions and I can only summarize the ones I attended. Check out the event agenda for abstracts on all sessions.

First I attended Web Crawl Cache – Using HBase to Manage a Copy of the Web. In this talk, we learned about Yahoo’s Web Crawl Cache (WCC) that collects and organizes data from Microsoft as a result of a search deal. These snapshots of the web are not only useful for search, but also for drilling into other avenues such as local assets, influence and language corpora. WCC uses HBase for several reasons: bulk load, MapReduce jobs are efficient, random access reads, a usable consistency model, and it is easy to dynamically add columns (this seems to contradict Karthick’s claim).

It was very difficult to pick a session for the 1:45 to 2:15 time slot. Options included Next Generation Hadoop, Scaling out Realtime Data (Facebook) and Building Kafka (LinkedIn). I admire the work and clout that LinkedIn has built over the past year or two, so I attended Jay Kreps session. LinkedIn’s data pipeline includes a lot of tracking, logging, metrics, messages and queuing. LinkedIn attempted to use messaging systems such as JMS and RabbitMQ. Streaming data is prevalent at LinkedIn such as search trends, click trends, invitation social networks etc. Kafka is LinkedIn’s solution for a distributed message queue; rather than polling for data, users subscribe to a data stream and data sources publish data to it. Kafka is 7000 lines of Scala, a functional and object-oriented language on top of the Java Virtual Machine (JVM). Kafka can produce about 250,000 messages per second (50 MB) and consume 550,000 messages per second (110 MB).

Next I attended another talk by Hortonworks, this time on HCatalog. HCatalog changes the way we think about data in HDFS. No longer do we need to worry about files and directories. Instead, HCatalog seems to add a layer of abstraction on top of HDFS that treats data as a set of tables. Tools such as Pig and Hive use this layer of abstraction, and currently Hive is tightly integrated with HCatalog. Hortonworks intends to add support for HBase and Streaming later this year.

I waited all day to see Matei Zaharia‘s talk on Spark. Zaharia is a graduate student at UC Berkeley and it was a nice change of pace to see a student present some work. Spark is a data processing platform that sits on top of the Mesos cluster management project (also produced by Berkeley). Mesos can handle 10,000s nodes, 100s of concurrent jobs and can be isolated in Linux containers (i.e. OpenVZ). Spark aims to extend MapReduce for iterative algorithms, and interactive low latency data mining. One major difference between MapReduce and Spark is that MapReduce is acyclic. That is, data flows in from a stable source, is processed, and flows out to a stable filesystem. Spark allows iterative computation on the same data, which would form a cycle if jobs were visualized. Resilient Distributed Dataset (RDD) serves as an abstraction to raw data, and some data is kept in memory and cached for later use. This last point is very important; Spark allows data to be committed in RAM for an approximate 20x speedup over MapReduce based on disks. RDDs are immutable and created through parallel transformations such as map, filter, groupBy and reduce. RDD immutability is similar to immutable types in functional programming languages. It does not mean that the dataset cannot change. Instead, it means that a new copy of the dataset is created, with the change included. The user can also perform actions on RDDs such as count, collect, etc. Some applications using Spark are traffic prediction (Berkeley), spam classification (Twitter), kmeans, alternating least squares matrix factorization, and network simulation.

The main takeaway from Hadoop Summit 2010 was Cascalog. I predict the main takeaway from Hadoop Summit 2011 is Spark.

One time at work I had a bizarre issue with corrupted data in HDFS. After that, I began blaming everything on HDFS. The next session Data Integrity and Availability of HDFS was englightening. HDFS takes good care of Yahoo’s data. We can trust Yahoo because if HDFS breaks, Yahoo begins losing money so they know what they are talking about! Yahoo’s goal is to have 60 PB online all the time. The key to HDFS reliability is replication. A replication factor of 3 (3 copies of every file? block?) is appropriate. A replication factor of 2 is also quite robust, but should only be used when there is a backup of the data because the probability of data loss is much higher. Yahoo has had issues with losing blocks (blocks are pieces of data, so lost blocks = data loss). There are a variety of reasons and most of them had nothing to do with HDFS. One cause of lost blocks is a bug in a Hadoop component like Pig, particularly a new version. In one incident, a new version of Pig opened a lot of files without closing them, and created a lot of abandoned files. In the speaker’s anecdotal case study, none of the incidents of data loss were caused by HDFS proper. Other causes of data loss encoutered were exhausting disk space, users hammering HDFS, and “other.” The speaker noted that NameNode high availability (a hot topic) would have only helped in 8 of the 36 incidents studied. Some ways of preventing data loss include resource allocation, selecting good tenants of a cluster, and fixing hardware errors quickly.

If your job isn’t running, it’s not likely caused by HDFS.

Bill Graham of CBS Interactive gave an interesting talk about using Hadoop to build a graph of users a content. Surprisingly, CBSi has quite a large arsenal of MapReduce enabled technologies: Chukwa, Pig, Hive, HBase, Cascading, Sqoop and Oozie. CBSi uses only 100 nodes with 500 TB of disk space for processing data associated with 235 million uniques (individuals, roughly). Mapping users to content should be easy, right? Well, some users have multiple identities, including anonymous identities. The goal is to create a holistic graph that “matches” all of the identities efficiently for uses such as ad targeting. CBSi’s needs in a Hadoop platform: rapid experimentation and data mining, and to power new site features and ad optimization. The main vehicle for representing data is a Pig RDF that allows for a kind of graph based join so to speak. CBSi hopes to add Oozie, Azkaban, HCatalog and Hama (graph processing) to its arsenal.

MapR was a very prominent sponsor of Hadoop Summit. M. C. Srivas presented a technical discussion of MapR’s capabilities and how it differs from Apache Hadoop. MapR is a full distribution of Hadoop and is 100% compatible with the Apache distribution and projects such as Pig and Oozie. MapR is fast and boasts high availability by rethinking the NameNode. The NameNode is a bottleneck since 60% of file operations are metadata. The NameNode and its limitations limit the size of a cluster. To resolve some problems with the NameNode, MapR turns every server into a metadata server. Since metadata is seldom retrieved, it is paged to disk so more RAM can be used for MapReduce proper. MapR distributes NameNode functionality and provides full random read and write semantics as well as export to NFS. With the distributed NameNode, runaway tasks no longer take down the NameNode. MapR has some lofty performance goals. While HDFS can handle 10-50PB, MapR can handle 1010 EB (exabytes). While HDFS can handle 2000 nodes in a cluster, MapR can handle 10,000 or more. It was mentioned at BigDataCamp that MapR does not rely on HDFS at all.

The final session I attended was Avery Ching’s talk on Giraph: Large-scale Graph Processing on Hadoop. Unfortunately, Avery jumped right into the technical details of Giraph without giving a high level overview of the problem Giraph solves. Also, his slides were in 10 point font and I could not read them. Combine this with the fact that my brain was exhausted, so I wanted to head to the bar. Vanilla Hadoop incurs too much overhead for graph data processing. Yahoo used MPI in the past for graph data but it had no fault tolerance and was too generic. Giraph is a library for iterative graph processing. Giraph is fault-tolerant and dynamic. Giraph takes a vertex centric approach to graph data. I found this interesting because most of my work is edge centric. Overall, Giraph is similar in goal to Pregel, but available to non-Googlers and has no single point of failure (except those incurred by Hadoop).

Now I have to catch my breath with some wine, beer and cheese at the nice happy hour reception afterwards. It was a long day, and a great day at Hadoop Summit 2011 and I will of course be back next year. I have no clue what is in store for me next year. Will the NameNode be removed as the single point of failure? Will other open-source software start integrating Hadoop? We shall see…

And now it is time to head back to Los Angeles.

Have a happy and safe Fourth of July!

The food/drinks/networking event must have been amazing because it was very difficult to get everyone to come to the main room to start the event! We started with a series of lightning talks from some familiar names and some unfamiliar ones.

Chris Wensel, the developer of Cascading, is also the founder of Concurrent, Inc. Cascading is an alternate API for Map-Reduce written in Java. With Cascading, developers can chain multiple map-reduce jobs to form an ad hoc workflow. Cascading adds a built-in planner to manage jobs. Cascading usually infers Hadoop, but Cascading can run on other platforms including EMC Greenplum and the new MapR project. RazorFish and BestBuy use Cascading for behavioral targeting. Flightcaster uses a domain specific language (DSL) [...]]]>

It has been a while since I have been to Silicon Valley, but Hadoop Summit gave me the opportunity to go. To make the most of the long trip, I also decided to check out BigDataCamp held the night before from 5:30 to 10pm. Although the weather was as predicted, I was not prepared for the deluge of pouring rain in the end of June. The weather is one of the things that is preventing me from moving up to Silicon Valley.

The food/drinks/networking event must have been amazing because it was very difficult to get everyone to come to the main room to start the event! We started with a series of lightning talks from some familiar names and some unfamiliar ones.

Chris Wensel, the developer of Cascading, is also the founder of Concurrent, Inc. Cascading is an alternate API for Map-Reduce written in Java. With Cascading, developers can chain multiple map-reduce jobs to form an ad hoc workflow. Cascading adds a built-in planner to manage jobs. Cascading usually infers Hadoop, but Cascading can run on other platforms including EMC Greenplum and the new MapR project. RazorFish and BestBuy use Cascading for behavioral targeting. Flightcaster uses a domain specific language (DSL) written in Clojure on top of Cascading for large data processing jobs. Etsy uses a DSL written in JRuby as a layer on top of Cascading. Of course, the big player is BackType. Cascalog combines Cascading with the Datalog language to provide a declarative language for working with data and map-reduce. Wensel noted that one disadvantage of Pig and Hive that Cascading addresses is that Pig and Hive lack a physical planner. Workflow managers such as Oozie and Azaband can run Cascading jobs as part of a workflow. Version 2.0 of Cascading removes Hadoop as a dependency and will allow users to run Cascading jobs on data that is in RAM rather than on disk.

James Falgout from Pervasive DataRush presented the second lightning talk. Pervasive’s products seem to use this “dataflow” paradigm that attempts to fill in features that are missing in map-reduce. The basic description compared dataflow to the Unix shell pipeline with message passing. James showed an example dataflow that a user could configure visually. Pervasive is working on integrating dataflow with Hive.

Guy Harrison from Quest Software introduced their system Toad for Cloud Databases. Toad attempts to merge data from several different data sources for analysis such as Hive, MongoDB, and Cassandra. Unfortunately, Guy’s thick Australian accent made his humorous talk unintelligible to me (hearing loss?).

Steve Wooledge from AsterData (now part of Teradata) discussed the company’s product goal of taking a standard relational database system and integrating map-reduce on top of it. Such a system is flexible and allows both SQL-like access as well as programmatic access to data. This hybrid row-oriented and column-oriented datastore can be used for path and pattern matching, text processing and graph traversal among the usual tasks. nPath is a product that enhances a system with transactional data analytics (click analysis, sessionalization).

Andrew Yu from EMC presented some of EMC’s data analytics products. I wrote about EMC in an earlier blog post so I will spare the details. EMC offers a data warehouse product as well as a hybrid, pre-configured system containing its Greenplum warehouse and map-reduce built-in.

Ben Lee from Foursquare discussed how big data is used at Foursquare and gave some statistics about its service. This was by far the most interesting talk to me. Foursqaure offers realtime suggestions of places to visit based on the user’s history, and the user’s friends’ histories based on day of week and time of day. Foursquare has 10 million users, 50 million venues, and 750 million check-ins. There are over 3 million check-ins per day. 10,000 developers use Foursquare’s API. MongoDB is the main datastore and Scala is used for the front end. Back end data processing uses Hadoop (both vanilla, and Streaming) as well as Flume, Elastic MapReduce, and S3. Ben displayed an awesome visualization of check-in data; researchers took check-ins from New York City and performed sentiment analysis on the text attached to the check-in. The visualization suggested that people were the “happiest” in Manhattan.

Paul Baclace introduced some software called Phatvis that allows developers to visualize map-reduce jobs. It is his hope that the visualization can be used to fine tune Hadoop parameters based on evidence from prior jobs. The source can be found here

Of course, the fun in every “unconference” is the circus known as scheduling the sessions. Some of the proposed sessions:

• Big Data 101 / Intro to Hadoop
• Extract MapReduce Data into Relational Database High Performance Database
• “ETL was Yesterday” What’s next?
• Operations of Hadoop Clusters
• SQL / NoSQL Why not Both? (Aster)
• Geodata
• Big Data Retention / Compression
• Business Intelligence and Hadoop
• Data Management Lifecycle
• Distributions of Hadoop
• Hadoop for Bioinformatics and Healthcare

The topics did not seem exciting this time, and seemed to have a lot of overlap with presentations at Hadoop Summit, but I found two (we could only attend two) that stood out.

Session 1: Operating a Hadoop Cluster

Thank goodness managing a Hadoop cluster is not in my job description (only small clusters I use for research). Charles Wimmer, the lead of the Operations track for Hadoop Summit, lead this discussion and much of the discussion dovetailed off of incidents that occurred at Yahoo. A popular topic of discussion was backup. There is no such thing as “backing up” a Hadoop cluster we agreed. Any data that is important should be replicated, preferrably 3 times, or transmitted in parallel over a pipe to multiple data centers. One strict limitation of replication is that if some new release of Hadoop, or some new Hadoop distribution contains a bug that corrupts the data, all replicates may also be corrupted.

Discussion then turned to hardware. Yahoo uses high-density storage nodes with 6 drives each containing 2-3TB of space. Charles mentioned that a common problem with Hadoop is that it is difficult to keep the CPUs busy especially in a server with 8 Nehalem processors (8 CPUs or 8 cores?). The major reason for this is that the main bottleneck in map-reduce jobs is the network I/O required in the shuffle phase as data comes out of the mappers. The map phase is the most CPU bound phase. Wimmer, and several others, made one thing clear: use SATA, not SAS. Apparently SATA and SAS drives have similar read performance (I believe I misheard that) for practical purposes. The original Google map-reduce was based on commodity hardware and quantity is more important than quality (within reason). For this reason, SATA provides a lot more space for your data. The same amount of space is an order of magnitude more expensive for SAS drives.

The next topic of discussion was the NameNode as the single point of failure. Apparently the MapR system does not use the HDFS, and recovering from a lost NameNode is not as severe as it is for Hadoop. Hadoop 0.20.2 also supposedly introduces sharding, called NameNode federation, where the namespace is divided over several NameNodes.

Hadoop has some issues with certain types of scalability, particularly with the JobTracker. When a large job with a large number of mappers and reducers finish quickly, the TaskTrackers send an influx of messages to the JobTracker and it gets overwhelmed. To prevent users from thrashing a cluster, use a capacity scheduler to put hard caps on queues. There was also some high level discussion of QoS-like functionality among users and sophisticated monitoring of jobs. Map-Reduce NextGen improves scalability by allocating a JobTracker to each individual job whose purpose is solely to monitor resource allocation. The biggest feature Charles would like to see is high availability NameNodes.

Yahoo boasts an impressive 22 clusters each containing between 400 and 4200 nodes. A fellow from AOL indicated that AOL has a cluster of size close to 1000. Is AOL coming back from the dead?

Thank goodness managing a Hadoop cluster is not in my job description…

Session 2: Geodata

I do not get the opportunity to work with geographical data often, so I was curious to see what these folks had to say. The discussion was lead by a fellow named Brian from OSGeo. The largest point that I took away from this talk was that not an incredible amount of thought has been dedicated to Big Geodata, particularly how to store and process it. PostgreSQL and PostGIS are a few ways to store and analyze manageable amounts of data, but not large data. MongoDB is one solution but has its issues. A fellow from Foursquare mentioned that MongoDB cannot shard across geographic data, but I could not hear precisely what he said to that effect. The biggest challenge seems to be a lack of an indexer capable of indexing a large amount of geospatial data aside from the standard RTree implementation. I believe that geodata as well as streaming data and multimedia are some of the biggest unsolved problems in Big Data.

Anyways, on to Hadoop Summit!

As our motivating example, consider we find some really cool graphic on the web and we want to know where it likely originated (i.e. art, a meme). In such a search engine, we could upload the graphic and get results containing the exact image, or images that are very similar such as variations of the image (crop, resize, borders, different effects), modifications of the image (consider Obama-izing [...]]]>

Recently, I have been thinking about alternate ways of specifying search queries other than with text. A couple of weeks ago I came across a piece of music that I could not identify. I thought it would be a huge win for a search engine to allow me to upload this piece, and it would present me with matches, or near matches to other pieces that sound similar, or have similar characteristics. Some services already exist. Shazam allows a user to place a microphone near playing music and it will identify the artist and song. Some uses of search-by-sound:

• Music identification (“solved” – Shazam)
• Music personalizaton and recommendation (“solved” – Pandora)
• Identification of the source of a sound (i.e. a species of bird, a musical instrument, an inanimate object)
• MP3 and media file search
• Finding material that violates copyright

As our motivating example, consider we find some really cool graphic on the web and we want to know where it likely originated (i.e. art, a meme). In such a search engine, we could upload the graphic and get results containing the exact image, or images that are very similar such as variations of the image (crop, resize, borders, different effects), modifications of the image (consider Obama-izing [the campaign logo] someone’s picture), and semantically similar images (different photos of the same object or person). Wouldn’t this be cool? A billion-dollar idea, right?

Well, Google apparently beat me (and millions of others I’m sure) to it with its search-by-image feature on Google Images. I uploaded a photo of myself to see what I would get. We see my school website (where the image originated), as well as several other sites that use my Gravatar. Not too bad.

On the results page, users can also provide some type of labeled data to Google. I am not exactly sure what it is used for yet, but note the text in the search bar: “Describe this image.” Upon entering my name, Google found another photo that looks almost identical to the first one — a variation.

Below are the “visually related” images that were presented to me (before I labeled my photo in the search bar):

I see Steve Jobs (I am honored), but 7 out of 16 images are women, and of the men, we look nothing alike. I know, I know, “visually related” refers to similarity in pixels between images, but I expected more. In these images, we see a lot of red and blue hues.

Let’s try something that will generate many more hits: a popular meme…

The image I uploaded was originally posted on Amazon S3, and is linked to by the above two web pages. Google does a much better job when using a URL rather than uploading an image for obvious reasons. More interestingly, the “visually similar” images show variations and modifications of the same image, based on pixel similarity.

And we get also see web pages containing a copy of the image (not linked to the original S3 file):

But this Isn’t Good Enough Yet!

Google “Search-by-Image” is an awesome first step, and I look forward to seeing more as it is undoubtedly coming. For search-by-image (or search-by-multimedia) to be useful, it must also take “semantic” or conceptual knowledge into account, just like with text search. That is, if I upload a photo of myself, I should get back other photos of myself from various (hopefully authorized) sources. Or, if we upload a photo of the Eiffel Tower, we should get back integrated search results containing other images of the Eiffel Tower as well as text results with information about the Eiffel Tower, and perhaps a tourist’s video or documentary.

One may at first believe that the O RLY search used some semantic knowledge; however, all of the images share a large number of pixels and these images are likely just “visually similar” as stated. Using semantic knowledge, one may see results of other famous owls used in memes in addition to the variations and modifications of the O RLY owl.

All of the data collected by such a system would also provide a hell of a corpus for image and multimedia classification. Researchers could construct classifiers for detecting spammy multimedia, knockoff multimedia (second, third generation grain in images, waveform distortion in audio), pornographic content, as well as augmenting labeled and unlabeled multimedia with metadata. For example, suppose we take a picture of what I think is a rhodendron (inside joke for readers). With such a large corpus, I can upload the photo and have Google (or some other AwesomeSearch) retag the image as that of a hydrangea instead.

Uses of search-by-multimedia with semantic knowledge:

• Cross referencing objects or people on various different sites.
• Product search when textual information (or QR code) is not known
• Catching criminals
• Cataloging media
• Methods for multimedia spam detection
• Geolocation without use of GPS or WiFi, and location search
• Augmentation of metadata and tagging of objects, people, etc.
• Detecting adult, inappropriate or illegal content.
• Identification of actions from images, video or audio and retrieval of related information

Of course, search-by-multimedia poses the same challenges that we face in big data today:

• choosing and boosting the proper features
• collecting a significant and correctly labeled corpus
• fast processing of large datasets with new and existing machine learning algorithms
• efficient indexing and retrieval algorithms to match queries with probably results
• these things are easier said than done, but a lot of fun.

Search-by-multimedia is a very interesting concept and is exciting to think about. In this age of big data and technology, anything will possible. I look forward to the day where anything on the Internet can be found, no matter its content or medium.

To check out Google’s search-by-image, click here and then click on the camera icon.

The CPU/Processor

If you are reading this article, I probably do not need to explain what the CPU/processor does. For high performance computing, you will want to get a CPU that is very “fast” and also has multiple cores. The definition of the word “fast” is in the eyes of the beholder and typically refers to more than just clock speed (GHz). In the constant war between AMD and Intel, I stick with Intel. AMD processors are powerful, but they seem to have [...]]]>

I am usually pretty reserved with cash, but after working full-time for six months, I finally decided to spend some of my money on building a new research development server. This process was long overdue and the reason it took me so long to commit to this project was all of the new technology developed since building my last server. This “new technology” can be pretty confusing unless one specializes in computer architecture. I want to share what I have learned throughout this process, while giving some background. These are only my opinions, and I may be wrong on some things as I am not a hardware expert. I encourage you to read and learn more on your own.

The CPU/Processor

If you are reading this article, I probably do not need to explain what the CPU/processor does. For high performance computing, you will want to get a CPU that is very “fast” and also has multiple cores. The definition of the word “fast” is in the eyes of the beholder and typically refers to more than just clock speed (GHz). In the constant war between AMD and Intel, I stick with Intel. AMD processors are powerful, but they seem to have more of a market with gamers. Intel is my preference, but I have not yet run into anyone that feels strongly towards AMD for high-performance computing (HPC). There are two main processor lines under Intel: standard, and Xeon. Standard processors are your run of the mill CPUs that are found in consumer desktop machines. Xeon processors are designed for non-consumer server, workstation and embedded systems use. I do not consider researchers as “consumers,” we are producers, so the Xeon family is better suited to our needs. On the other hand, you may find that a standard CPU will fit your needs for your particular research or use case. Xeon processors typically have more cache and more multiprocessing capabilities…and they are a lot more expensive. For high-performance computing, I strongly suggest Intel Xeon.

After months of research, I have concluded that multiple Intel Xeon processors are better than one Intel Core i7. As of the time of this writing, it seems that i7 processors cannot be doubled (or tripled etc.) up like Xeons can. Like the AMD, the i7 seems to be favored by gamers and those needing a richer multimedia experience. 

In 2011, most CPUs in new systems have multiple cores. Each core can essentially run one process each. A system with n cores can run n processes simultaneously. Many CPUs are hyperthreading enabled, meaning that each core can actually run 2 threads simultaneously, bringing the total number of threads to 2n. But can’t the system already run multiple processes concurrently? We can run Firefox, TweetDeck, Thunderbird etc. concurrently, right? In practice, it seems that the CPU is processing multiple threads simultaneously. If we could slow down time to the micro level, one would see that the CPU works on one process at a time, then does a context switch to another process. Theoretically, this gives the illusion that the CPU is running multiple processes simultaneously.

While Intel makes great products, its inventory is a nightmare to navigate. There are several things that you must know to ballpark a particular CPU model.

• the model number (the most reliable!)
• the brand name specifies a group of CPU models satisfying similar use cases (Core [i3/i5/i7/i9], Core 2 Duo, Quad Core, Pentium, Xeon).
• the architecture/subarchitecture — specifies a type of processor within a brand, each containing many series (Nehalem, Westmere, Sandy Bridge are common ones these days)
• the chipset (not commonly referred to, examples: Tylersburg, Cougar Point, Panther Point)
• the platform which refers to a set of models (e.g. Harpertown, Jasper Forest, Gainestown, Prescott, Gulftown). Models within a series are typically only differentiated by clock speed (GHz).
• the socket type specifies the shape and size of the CPU. The CPU and the motherboard must have the same socket type (i.e. LGA1366, Socket 775)

As if this is not confusing enough, each Intel Xeon model number is prefixed with a letter for different use cases. The letter distinguishes CPUs with differing thermal dissipation power (TDP). (source)

• W stands for “Workstation” and is meant to be installed in pairs. This designation does not seem very common anymore. They typically run the fastest (clock speed) and the hottest. They require significant cooling.
• E is “mainstream (rack mount)” and the standard model of CPU. Although it is “standard,” there is nothing wrong with it performancewise, but will run hot even when idle.
• X stands for “performance” and are similar to E but provide for extra overclocking capabilities and have lower idle power draw.
• L stands for “power optimized” and are low voltage CPUs (60W or less) that are typically only used for data centers or rack servers. They typically do not come in the higher clock speeds etc.

For the Intel Xeon, model numbers indicate what configuration it is compatible with on the motherboard (source):

• 3xxx Xeons are designed to be used by themselves, as the only CPU on the motherboard.
• 5xxx Xeons are designed to be used in pairs; two CPUs on the motherboard.
• 7xxx Xeons are designed to be used in pairs, or in larger groups.

The 2 CPUs that I purchased are model Intel Xeon E5645. The Intel Xeon E5645 is part of the Gulftown platform of the Xeon family. It uses the Westmere subarchiture which is the 32 nm shrink of the Nehalem architecture spec and connects to the system bus using socket LGA1366. (This is the same architecture used for the i7-9xx series to make it more confusing) The E means that it is a “mainstream” CPU. Since it is a 5000 model, it is installed with another identical CPU on the same board.

The number of cores is important. Most chips in current desktops contain 2 or 4 cores. Higher end systems and servers may have 6, 8 or 10 cores per chip. Xeons with 8 and 10 cores per unit debuted in Q2 of 2011 and are very expensive (about $2000 for 8 cores). They also require a brand new socket type (LGA1367), which means a new, expensive motherboard. A CPU with more cores allows an application to perform several units of work per task; these processors allow higher bandwidth.

The clock speed (GHz) used to be the deciding factor for most people, until Moore’s Law broke down. Higher clock speed possibly allows a single process to complete faster. Since games typically use a limited number of threads and require quick performance, a single i7 is a good choice. The i7 has multiple cores, and also has a very high clock speed.

The cache size and speed is also important. The cache allows very high speed access to memory locations that are frequently accessed by copying the data from RAM into the CPU cache. Modern systems typically have three levels of cache: L1, L2 and L3. L1 cache is said to be the “closest” to the CPU, meaning the CPU queries the L1 cache first when performing a memory access. The L1 cache is the smallest. The L2 and L3 caches are accessed next in order, and L3 cache is larger than L2 cache. Very simply put, CPUs with larger caches (especially L1) are better.

Newer processors report CPU throughput as gigatransfers per second (GT/sec) which, like GHz, quantifies some measure of “speed.” Using GT/s, one can compute the number of bits the CPU can transfer per second as

Think of the cores vs. clock speed decision as a highway. Suppose the clock speed indicates the maximum speed limit on a single lane highway. A faster CPU corresponds to a single lane highway with a high speed limit. You will get to your destination faster. On the other hand, consider a one-lane vs. a two-lane highway, both with identical speed limits. If one lane is too busy for you, take the other lane. An increase in the number of cores increases the number of choices of lanes you can transition to. On the single-lane highway, you would need to slow down and wait for the cars in front you to move forward. By switching lanes, you may get to your destination faster, or you may not, but more driving is completed overall. 

Some time over the past 6 weeks I randomly saw a tweet announcing the “Data Scientist Summit” and shortly below it I saw that it would be held in Las Vegas at the Venetian. Being a Data Scientist myself is reason enough to not pass up this opportunity, but Vegas definitely sweetens the deal! On Wednesday I woke up at 6am to partake on the 5.5 hour voyage to Las Vegas.

The Pre-Party

The Venetian and all close hotels were booked, so I ended up at the Aria; a new experience. The hotel is beautiful and very ritzy. I had heard that the rooms were very technologically advanced but I wasn’t prepared for the recorded welcome message, music and automatic shades opening upon entry to the room. The Aria is a geek’s paradise. Everything is computerized. Key cards are “waved” rather than swiped, lights are turned on/off and dimmed by use case (“sleep”, “read” etc.), rather than manually. There are no paper “Do Not Disturb” signs; rather, a switch on the wall (or via TV) toggles an indicator light outside the door. And the best part… Internet is FREE!

Elastic Compute Cloud (EC2) is a service provided a Amazon Web Services that allows users to leverage computing power without the need to build and maintain servers, or spend money on special hardware. The idea is simple, the user “boots” up one or more machines and then accesses those machines as if they were logged into any other machine remotely. I used EC2 and Elastic MapReduce extensively for my M.S. thesis last spring, but mainly used its large memory capabilities rather than its potential for explicit parallelism.

Recently, I ran a crawling job on EC2 using a parellel crawler I wrote in Python with twill. Using EC2 poses its own challenges. Using parallel code poses more challenges. Combining these two facts with the fact that crawling is I/O bound can create some more interesting challenges. If you have taken a course in operating systems, you have heard this stuff over and over again. So have I, but I am stubborn. I tend to learn lessons from experience, and this was no exception. Through this series of posts, I want to point out difficulties and “gotchas” that are important to keep in mind when using EC2, and in this post, with [...]]]>

Elastic Compute Cloud (EC2) is a service provided a Amazon Web Services that allows users to leverage computing power without the need to build and maintain servers, or spend money on special hardware. The idea is simple, the user “boots” up one or more machines and then accesses those machines as if they were logged into any other machine remotely. I used EC2 and Elastic MapReduce extensively for my M.S. thesis last spring, but mainly used its large memory capabilities rather than its potential for explicit parallelism.

Recently, I ran a crawling job on EC2 using a parellel crawler I wrote in Python with twill. Using EC2 poses its own challenges. Using parallel code poses more challenges. Combining these two facts with the fact that crawling is I/O bound can create some more interesting challenges. If you have taken a course in operating systems, you have heard this stuff over and over again. So have I, but I am stubborn. I tend to learn lessons from experience, and this was no exception. Through this series of posts, I want to point out difficulties and “gotchas” that are important to keep in mind when using EC2, and in this post, with using parallelism in your code to accomplish large tasks.

Monitor your Instances

Monitoring your instances has two important benefits. First, to make sure that you are not maxing out resources on the machine. EC2 is “elastic.” With some clever programming, you can boot up more machines if you notice resources becoming scarce on your current machines, and then decommission them later when they are not needed.  I did not do this at first, and I ran into several issues.

Disk Space. The concept of a “disk” is very confusing in EC2. The AMI forms a disk, sort of. Above and beyond the OS and any other software and packages you may install as part of the AMI, you can use whatever free space is remaining to store output files. The total disk space used by the AMI seems to be configured at the moment the AMI is constructed. Thus, it is not a good idea to store files in the instance. I did this. Fortunately, I found out before it was too late that my “disk” was filling up. I wrote a cron job to copy all of my output files to /mnt every five minutes. Use /mnt to store your files as it has lots and lots of space; HOWEVER if you terminate your instance, the files are gone. This is still true if you use space within the instance. Once your job completes, upload your files to S3. s3cmd allows access to S3 from the command line, and with the modification here, you can upload and download files in parallel (a life saver for big batches). Another option is to create an EBS volume, mount it, and write files directly to the EBS volume. EBS space is much more expensive than S3 space.

Memory. On my first attempt, I maxed out memory to the point that the OS killed 6 of my 8 processes. This caused a huge blow in the performance of my crawler and rendered the extra money I spent on an extra large instance wasted. Monitor your job’s memory using top. If memory usage seems to grow too fast to your liking, consider using a memory profiler to make sure that there are no memory leaks in your code. I have found that long running Python processes eat up a lot of RAM, even if there are no explicit growths of data structures.

Additionally, maxing out RAM means that the disk will begin to swap. This is devastating to performance because this extra grinding of the disk decreases the total I/O throughput your job can handle. This is crucial for crawlers as files need to be written to disk quickly.

If after profiling you find that your job is still using too much RAM, consider caching, or using a high memory EC2 instance.

I/O Throughput. How fast your job consumes and produces data is a good way to determine if something is going amiss in your job, or with the other resources you are using. When I started my crawling job, I was crawling n pages per hour, but after twelve hours, the throughout decreased exponentially until it got so slow that I had to add more instances. One way to monitor throughput is to save the results of ls -latr –full-time to disk and extract the date/time of each file. Using a tool like R, you can quickly plot your I/O throughput over time using an aggregate(). A decrease in I/O throughput can be the result of many things: 1) swapping from exhausting RAM, 2) low disk space, 3) network congestion within AWS, 4) poor resource performance (if crawling, the resource would be the website being crawled), 5) hammering an external resource and/or HTTP throttling, 6) congestion in the Internet. For crawling, you may want to consider using several smaller instances rather than fewer larger instances. This way, you will be accessing the resource from many IPs and the result of being throttled should be lessened. Additionally, use instances that have “High” I/O performance; some are rated “Moderate” or “Low.” 

CPU. A general rule of thumb is that you can run n processes in parallel, for n cores. Additionally, if each core supports hyperthreading then the number of processes you can run is approximately 2n. If you run more than the suggested number, the price of context switching can slow down your performance. If you find the need to routinely exceed this guideline, use an instance with more cores.

When running parallel code, routinely do a ps aux | grep processName to make sure the correct number of processes is running. If any were killed, this will be noted in /var/syslog with a reason.

Financial metrics. Are you getting your money’s worth? Are you really using all of the cores you are paying for? Are you really using all of the memory you are paying for? This is up to you and your budget to dictate. But do not get carried away and assume that you must stay with the same instance size. Most AMIs can run on different instance types (except 64bit AMIs are restricted to m1.large and bigger).

Quarantine Essential Services

My crawler used Redis as a work queue. Each process could easily write new thread IDs and page numbers to the queue as they are discovered, and read thread IDs and page numbers from the queue as each process is ready to crawl a page. One problem that I faced was that I coupled the crawling operation with queue management into the same script, and ran the Redis server on a server where a crawler was running. This coupling posed two challenges. First, it can sustain nasty bugs. Whenever a process was created on the master Redis node, my code would wipe the Redis queue clean to prepare it for crawling (bug!). Flushing the queue, and the initial population of the queue should have taken place in two separate scripts. Due to my major bug, I wiped the entire queue clean in the middle of the crawl. Fortunately, I followed the advice in the next section.

Second, I had to be careful that my processes did not exceed RAM limitations. Because Redis is mainly an in-memory key-value store, it itself can hog up most of the RAM in the instance. For this reason, it is best to quarantine essential services such as queues to their own instances.

Document Everything

Log everything. Log every resource you are going to use (URLs for a crawl) and log everything that was done and any problems that arise. Using the directory structure (ls) as well as a log of what work was already performed, I was able to reconstruct and repopulate the work queue and essentially start where I left off. For my crawling operation, I wrote the following events to logs, each with a timestamp.

• Starting the crawl.
• Logging in to the site being crawled.
• Clearing and populating the queue.
• Visiting a thread’s first page.
• Discovering the number of pages of posts in the thread/inserting to the queue.
• HTTP redirects, when a thread has been moved.
• Visiting a thread ID that does not exist.
• Inadvertent logouts, marking work to be redone.
• Queuing inconsistencies.

An activity log verbosely documented everything that occurred without logging actual data. An inventory manifest indicated which URLs/forum posts had valid content and how many pages of content were associated with them. A standard directory listing indicated what work had been done. By cross referencing the manifest and the directory listing, it is easy to see which posts had not yet been processed. A system log prepared by the operating system also documents critical failures for you, such as lack of disk space or processes being killed.

When writing your logs, use the advice in the next section!

Take Care to not Clobber Files and Objects

It’s been said over and over again. Each process should hold as much of its own real estate as possible. When two or more processes write to the same object, corruption can occur unless there is a locking mechanism in place. If two processes write to the same file at the same time, you will notice garbled entries in your logs. This did not affect my crawled data because each file was written by a single process. The same can be true for reading data as well. When spawning multiple processes, I shared the same Redis connection with all of the processes. If two processes read from the queue at the same time, one process would get the correct data (a thread ID and page number) and the other would get “OK”, which was the result of the first process’ fetch operation. This is mostly my fault, but partially redis-py‘s fault for filling some buffer between Python and Redis with meaningless information (“OK”).

Each process should write is own log files. When opening a file, you can use the following:

import os
 OUT = open("mylogfile-%s.log" % str(os.getpid()), "w")
 ...
 OUT.close()

Crawler Specific: Set an Upper Bound

Crawling is fun, but you must practice moderation or it is easy to attempt to boil the ocean. When I first started, I would run a crawl, have it crash, and then deem the data out of date and start over from the beginning and crawl until there was nothing possible left to crawl. It is good to set an upper bound: “I will crawl 10 days worth of data”, or “I will only use threads created prior to May 1, 2011.”

One of the keys to success with EC2 is to get over the penny pinching. If you have a project, just take the plunge and do it on EC2 (if required). The amount you spend on the first few projects will save you more on future projects.

I ditched my iPhone this past December (good riddance) in favor of the Droid X (Android). I figured, on such an open source OS, Google must be doing the same thing. After surfing through Hacker News, it turns out I was right.

Compared to the iPhone though, getting the data on an Android phone is not simple.

One wrinkle with NumPy/SciPy that needs to be ironed out is the difficulty of installation on certain OSes, and particularly, architectures.The SciPy SuperPack has done a good job of taking care of this issue, but it has not yet been updated for 2.7.1 and manually hacking away at its script has not worked for me.

I cannot take credit for the instructions in this article. A brave warrior, Jeremy Conlin, somehow managed to figure out how to install 64-bit NumPy and SciPy, with 64-bit Python 2.7.1 on Snow Leopard; he posted the directions to the SciPy User mailing list on February 24. I followed the directions, and miraculously they worked. I am reproducing them here for Google bait.

Install Python 2.7.1

1. Download the universal Mac 2.7.1 installer here (Python 2.7.1 Mac OS X 64-bit/32-bit x86-64/i386 Installer). Typically, Python will be installed to /Library/Frameworks/Python.framework/Versions/2.7/, but may be in other locations.

2. Verify that your new version of Python is 64-bit enabled. Note: Python installations typically do not get toggled as the default Python, so find the location of the 2.7.1 Python executable. On my machine, it is /Library/Frameworks/Python.framework/Versions/2.7/bin/python. python2.7 should also work.

Load [...]]]>

Numpy and SciPy are packages for numerical computation and scientific computing, for Python.

One wrinkle with NumPy/SciPy that needs to be ironed out is the difficulty of installation on certain OSes, and particularly, architectures.The SciPy SuperPack has done a good job of taking care of this issue, but it has not yet been updated for 2.7.1 and manually hacking away at its script has not worked for me.

I cannot take credit for the instructions in this article. A brave warrior, Jeremy Conlin, somehow managed to figure out how to install 64-bit NumPy and SciPy, with 64-bit Python 2.7.1 on Snow Leopard; he posted the directions to the SciPy User mailing list on February 24. I followed the directions, and miraculously they worked. I am reproducing them here for Google bait.

Install Python 2.7.1

1. Download the universal Mac 2.7.1 installer here (Python 2.7.1 Mac OS X 64-bit/32-bit x86-64/i386 Installer). Typically, Python will be installed to /Library/Frameworks/Python.framework/Versions/2.7/, but may be in other locations.

2. Verify that your new version of Python is 64-bit enabled. Note: Python installations typically do not get toggled as the default Python, so find the location of the 2.7.1 Python executable. On my machine, it is /Library/Frameworks/Python.framework/Versions/2.7/bin/python. python2.7 should also work.

Load Python 2.7.1 and execute the code below. If you get 64, then you are ready to proceed.

 import sys
 from math import log
 log(sys.maxsize, 2) + 1
 

Another way is to execute the following. If you get 1099511627776 (and NOT 1099511627776L) you are in good shape.

 2**40
 

(This tip comes from Aaron Meurer’s SymPy blog)

Install gfortran

1. Download gfortran-4.2.3 here and double-click the file to mount the disk image. 

2. Create a temporary directory (I call it tmp here) and run the following commands.

 cd tmp
 mkdir gfortran
 cd gfortran
 pax -zrvf /Volumes/GNU\ Fortran\ 4.2.3/gfortran.pkg/Contents/Archive.pax.gz .
 cp -r usr/local/* /usr/local
 

Drop Down to the Root Shell

NOTE: From this point forward, I dropped to the root command prompt (sudo su -) so that I had full control over the environment. This is mainly due to the way to the shell deals with PYTHONPATH and the complications that can stem from it when installing with root privileges.

Locate the Python 2.7.1 distribution and find a directory that ends in site_packages. For example, for me this is /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/.

Assuming Bash is your shell of choice, enter the following

export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/

Or, for tcsh,

setenv PYTHONPATH /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/

Install distribute-0.6.14

The next few steps follow the typical Python package installation method. I use python2.7 rather than python just to make sure that my system is using 2.7.1 and not built in 2.6.1.

Distribute adds tools for installing Python modules including pip and easy_install.

 curl -O http://pypi.python.org/packages/source/d/distribute/distribute-0.6.15.tar.gz
 tar -xzvf distribute-0.6.15.tar.gz
 cd distribute-0.6.15
 python2.7 setup.py install
 

Install nose

Nose is al alternate test discovery and running process for unittest, similar to py.test.

Installation requires easy_install. I am not a fan of easy_install in the slightest. There are two important things to remember here:

1. PYTHONPATH must be set correctly, or easy_install will complain.
2. If you have easy_install for another version of Python, make sure you use the version for 2.7.1 and not some previous version or easy_install will complain.

On my system, easy_install for 2.7.1 is located at /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install so I use:

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install nose
 

Install NumPy

Finally, we can install NumPy. Version 1.5.1 is the first version that apparently works with Python 3.

 curl -O http://downloads.sourceforge.net/project/numpy/NumPy/1.5.1/numpy-1.5.1.tar.gz
 tar xzvf numpy-1.5.1.tar.gz
 cd numpy-1.5.1
 sudo python2.7 setup.py install
 

Then, open Python and execute the following to import the library, and test it. On my system, all tests passed.

 import numpy
 numpy.test()
 

Install SciPy

Next, install SciPy. Version 0.9 is the first version that is compatible with Python 3.

 curl -O http://downloads.sourceforge.net/project/scipy/scipy/0.9.0/scipy-0.9.0.tar.gz
 tar xzvf scipy-0.9.0.tar.gz
 cd scipy-0.9.0
 sudo python2.7 setup.py install
 

Then, open Python and execute the following to import the library, and test it. On my system, all but 14 tests passed.

 import scipy
 scipy.test()
 

Install readline

readline provides functions for completion and reading/writing of history files from the Python interpreter (up/down arrow).

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install readline
 

Install IPython

IPython provides an enhanced command line interface to the Python interpreter. It is much more pleasant to work with than the standard command line interface.

 /Library/Frameworks/Python.framework/Versions/2.7/bin/easy_install ipython
 

Install wxPython

wxPython is a graphical user interface toolkit for Python. Up until recently, this was the blocker in the process — there was no 64-bit version of wxPython for Mac. Download the DMG here and double click the installer. Versions for Cocoa and Carbon are provided, but Python 2.7.1 apparently requires the Cocoa version.

Install matplotlib

matplotlib provides the plotting interface which is crucial to SciPy. Installing matplotlib is always the most complicated part of the process.

 wget http://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1.tar.gz
 tar xzvf matplotlib-1.0.1.tar.gz
 cd matplotlib-1.0.1
 

Then, open the file make.osx, find lines that look like below, and make the appropriate changes

 PYVERSION=2.7
 ZLIBVERSION=1.2.5
 PNGVERSION=1.4.5
 FREETYPEVERSION=2.4.4
 

Finally, delete line 63. Line 63 looks like:

 cp .libs/libpng.a . &&\
 

Within the matplotlib-1.0.1 directory, download the following archives. Do not extract them!

1. zlib 1.2.5
2. libpng 1.4.5
3. freetype 2.4.4

Or execute the following commands within the matplotlib-1.0.1 directory.

 wget http://zlib.net/zlib-1.2.5.tar.gz
 wget http://downloads.sourceforge.net/project/libpng/libpng14/older-releases/1.4.5/libpng-1.4.5.tar.gz
 wget http://download.savannah.gnu.org/releases/freetype/freetype-2.4.4.tar.bz2
 

Finally, install matplotlib-1.0.1 using the following command

 PREFIX=$PYTHONROOT make -f make.osx deps mpl_install
 

where PYTHONROOT is the directory containing the pacakage tree. On my system, it is /Library/Frameworks/Python.framework/Versions/2.7/.

Finally, enter Python and test matplotlib.

 import matplotlib.pyplot as pyplot
 pyplot.plot([1,2,3])
 

If installation was successful, a plot like the following should appear.

Hopefully this is helpful to someone, and here’s to hoping a SciPy Superpack for Python 2.7.1 will be released soon!

RStudio plays to this analogy very well, and makes modifications where appropriate. RStudio provides many features that are lacking in the standard R GUI, and improves on features that do not work properly in the Windows R GUI. Over the past few days, I have been doing all of my R analysis within RStudio, shortly with the Desktop version, and mostly with the Server version. I will discuss mostly the server version [...]]]>

As most readers are probably aware, the free IDE for R, called RStudio, was recently released for general use and it immediately made huge waves within the R community. IDE stands for Integrated Development Environment. IDEs typically provides a rich set tools developing in some target language. For standard programming languages like C++ (VisualStudio) and Java (Eclipse or NetBeans), IDEs contain:

• an editor tailored to the target language. The editor typically has tab/auto-complete for variable names, functions and class methods and properties and also features syntax highlighting.
• a multiple document interface (MDI) where there may be several documents opened in different tabs.
• a window that interacts with the compiler, or a panel containing the console to the language, a la MATLAB, and even vanilla R’s GUI.
• a debugger
• a file browser and language reference.

RStudio plays to this analogy very well, and makes modifications where appropriate. RStudio provides many features that are lacking in the standard R GUI, and improves on features that do not work properly in the Windows R GUI. Over the past few days, I have been doing all of my R analysis within RStudio, shortly with the Desktop version, and mostly with the Server version. I will discuss mostly the server version since that is what I have been using. It is identical (AFAIK) to the desktop version, so you are not missing anything by using either version.

RStudio Server

First, installation is miraculously easy. I only had a few very minor glitches to deal with. Armed with sudo access to a machine on a research cluster at work, I was able to simply download the RPM and install it using the instructions provided on the web site. Then, all I had to do was fire up a browser and go to

http://servername.com:8787

and I was asked for my login credentials. But I couldn’t get in. This server authenticates using LDAP, but all I had to do was replace the contents of /etc/pam.d/rstudio with the contents or /etc/pam.d/login and I was able to login. But then there was a “unknown error.” Oh, the version of R that was installed was too old (2.8). I just did a yum upgrade R, and RStudio logged me in with no problems. What showed up on my browser screen was beautiful! It looked identical to the desktop version of RStudio.

Once logged in, I somehow have access to ALL of my files on the remote server. I can load my data (typically produced by Hadoop) already residing on the server, and I can save output, graphs, data and even the R session itself on the remote server! All while just clicking buttons. No commands to remember, no screwed up PDF files, and most importantly…. no scping files back and forth from the server just to create a plot (X worked well, but had limitations)!

Things I Love about R Studio

I will have to go panel by panel, but even then I will have missed cool features. I also will not discuss features that are already present in the MacOS X R GUI and are repeated and beautified in RStudio:
The R command prompt still looks the same. At first, my reaction was “Damn, what am I supposed to do?” But when the GUI finished loading, the familiar R command prompt appeared in all is 1970ish glory. I immediately started typing commands and seeing fields in the other panes populate and change to display different usages. It left me with a “oh, I see” feeling.

Saves R sessions correctly, and when I return to RStudio, ALL of my work is there! I could never get the save session/image function to work in R GUI. I gave up several years ago. In RStudio, it works properly, but you don’t even need it because… when you leave RStudio and then return, everything is there! The workspace (variables, functions, data, etc), the scripts you were working on, the plots, even the last dang help screen you looked at!

The Stop Execution button in the console actually works. When executing a long running computation in R GUI (that’s the first mistake), it is sometimes necessary to cancel the computation either because I made an error, or because the computation is killing my system’s performance. In R GUI, particularly on MacOS X, the Stop Execution button did absolutely nothing, because there was typically a spinning beachball preventing me from clicking it. Hitting ESC also did not work. In RStudio, clicking Stop actually seems to break out of the madness.

Workspace panel. The workspace panel displays the variables, functions, data frames and other objects that reside in the current workspace, a la MATLAB. From this panel, one can also switch or save workspaces. The user can also import a dataset from a text file using a trivial wizard (a la SPSS, etc.), or from a web URL. The user can also clear the workspace. A frequently overlooked command to do the same from the command line is rm(list=ls()), but that is no longer necessary to remember!

Clicking on a data frame object in the workspace pane, causes it to be displayed in a nice tabular format. It can also be printed to a local printer, or opened in a new window.

Clicking on a numerical value allows the user to change it by opening an in-place edit box. Clicking on other objects like lists, vectors and functions opens an edit window displaying the definition that created it.

Files panel. There is nothing really exciting to see here, except that by clicking the Upload button, I can upload files directly to the remote server just by selecting the file, without having to SCP!

Scripting panel. This is the second best feature of R studio and has the same feeling as the stock script editor that ships with R. The largest difference is that the editor in RStudio is stable. On MacOS X, the editor tends to garble 2-3 rows of code together on every single scroll. This editor does a better job of indentation than R GUI. When opening a function, R GUI tends to indent the body properly, but insert a closing } prematurely. RStudio’s editor also features auto-completion, a feature present in the command-line of R GUI and R, but not in the editor of R GUI. The user can also save their script on the remote server, print code to a local printer and search. Similar to MATLAB, the user can select one or more lines of code and run them by clicking the “Run Line(s)” button, rather than having to copy and paste lines. “Run All” is a point-and-click replacement for source.

The “Source on Save” function is interesting. If enabled, RStudio will run/source the script each time the script is saved. Honestly, I do not find this feature to be all that useful unless in the middle of debugging, and dangerous if not debugging. Suppose after a long 10-fold-cross-validation computation there is an error that we want to fix. We fix the error and save the script. Do we really want to run the computation again? If R were a compiled language, then yes. Since R is not a compiled language, this feature is not entirely useful in concept.

The “magic wand” icon contains what I suspect to be a growing collection of coding tools. Currently, the user can comment and uncomment a bunch of lines at once. This is particularly useful since, for some reason, there is no multiline comment flag in R. The user can also select a series of lines and wrap a function around them. This feature could be dangerous for those not familiar with coding but provides a very nice way to put a bunch of code into a function as an afterthought.

Plot panel. By far my favorite part of RStudio is the plot panel! All plots are saved in this panel, and the user move back and forth among plots that were already constructed. The Export button allows exporting a plot to user defined dimensions and save to the local machine as a PNG, or even copy it to the local machine’s clipboard! Of course, the PDF button produces a PDF file of the plot that can be saved on the local machine. If the plots are all too much, we can click “Clear All” and start again with a clean slate.

But, is it possible to create plots of larger size? I am sure there is, but I did not spend much time looking.

LaTeX and Sweave documents. From the File menu the user can create new documents including LaTeX and Sweave. Unfortunately, I cannot experiment more with these features because there is something amiss in my configuration. For students and researchers, having Sweave and LaTeX integrated with RStudio is a huge, huge, huge advantage. No longer must we copy/paste among different programs. To make the integration complete, BibTeX, Asymptote/TikZ/gnuplot whatever should be easily included by the user.

At any point if the user interface shows stale data, there is a Reload button to help you out by refreshing the entire RStudio interface.

Things that Need Improvement

I do not really have any complaints about RStudio, quite the opposite actually. However, there are some things that do not seem to work. I should note however, that I have not spent much (well, any) time debugging them. The developers are probably already working on some of them. Some of them are probably problems in my configuration and others are probably settings that I need to tweak.

No auto-completion of parentheses or quotation marks. This is a bummer, but not a deal breaker. On the other hand, as you type closing marks, RStudio highlights the matching mark.

The dataset view needs work. Columns can’t be resized. Other natural functionalities that seem to be missing are: column renaming (a call to names), cannot sort or order values by a column, and data manipulation (I didn’t say that). These missing features are a tad disappointing, but a hell of a lot better than displaying in the terminal.

Install packages in the packages panel does not work on our server’s configuration.

LaTeX cannot be found. Upon attempting to create a new LaTeX or Sweave document, I got a friendly notice (instead of a bizarre error message) saying that LaTeX is not installed. The problem is, it is installed and there does not seem to be anywhere in the GUI to configure its location. Additionally, some LaTeX templates would be useful.

In Conclusion…

My Workflow Before and After RStudio

Before RStudio



After RStudio

The developers of this open source project seemed to get it right on the first try. How the hell is that possible??? So has anyone switched from the big R to the big blue ball?

Web mining requires a lot more sophistication. The programmer in many cases does not know the full formatting of the data published on a web site. Additionally, this format may change over time. There are certain standards that do apply to certain types of data on the web, but one cannot rely on web developers to follow these standards. For example, the RSS Advisory Board developed a convention for the formatting of web pages so that browsers can automatically discover the links to the site’s RSS feeds. I have found in my research that approximately 95% of my sample actually implemented this convention. Not bad, but not perfect.

Always Have a Plan B, C, D, …

One might say that 95% is good enough. I am a bit obsessive when it comes to data quality, so I wanted to extract a feed for 99% of the sites I had on my list. Also, I am always leery [...]]]>

Programming defensively requires knowing the input that your code should be able to handle. Typically, the programmer may be intimately familiar with the type of data that his/her code will encounter and can perform checks and catch exceptions with respect to the format of the data.

Web mining requires a lot more sophistication. The programmer in many cases does not know the full formatting of the data published on a web site. Additionally, this format may change over time. There are certain standards that do apply to certain types of data on the web, but one cannot rely on web developers to follow these standards. For example, the RSS Advisory Board developed a convention for the formatting of web pages so that browsers can automatically discover the links to the site’s RSS feeds. I have found in my research that approximately 95% of my sample actually implemented this convention. Not bad, but not perfect.

Always Have a Plan B, C, D, …

One might say that 95% is good enough. I am a bit obsessive when it comes to data quality, so I wanted to extract a feed for 99% of the sites I had on my list. Also, I am always leery of bias. Could there be something special about these sites that do not implement RSS autodiscovery? Clearly, there are exceptions to my Plan A. So time to move to Plan B. I found that some of the sites in this 5% used FeedBurner to index their feeds, so Plan B was to use a regular expression to extract FeedBurner URLs. This only added another 1% (actually less than that) to my coverage.

((?:http|feed)://feeds.feedburner.com/.*?)

Next, Plan C took the domain name and simply slapped /feed to it and hope it sticks. I called this process “feed probing” and it added the remaining 3% that I was looking for. If Plans A, B and C all failed to find a suitable RSS feed, all hope is lost and we just skip this site (1% error).

On the other hand, there are times when it is the HTTP client or server cannot be trusted…

Common Python Exceptions in Web Mining

It is all too common to encounter an exception while web mining or crawling. Code must handle these errors gracefully by catching exceptions or failing without aborting. One method that works well is to provide a resume mechanism that restarts execution where the code left off, rather than having to start a multiple hour/day/week job over again! Below is a taxonomy of common problems (and their Python exceptions):

HTTP Errors. These occur frequently. Some are recoverable, and others are worth just throwing out a record over. The most common ones are below, but for more information, refer to RFC 2616.

• 404 Not Found: the web page you tried to download could not be found. Skip the record.
• 400 Bad Request: the server has deemed the client’s HTTP request as malformed. Either retry, or double check your code!
• 401 Unauthorized: authentication is required before proceeding. Either skip the record, or add authentication to your code.
• 403 Forbidden: you are trying to access something you are not allowed to access. This is a common HTTP error thrown when your program is being rate limited.
• 500 Internal Server Error: something on the server end is wrong. Either try again immediately, or wait a while and retry the request.
• 3xx Redirect: the web page you are trying to access has moved somewhere else. In my research, these are rare but common in practice. I choose to skip these sites. You may wish not to.

In Python, these can be caught as urllib2.HTTPError. It is also possible to specify actions based on the specific HTTP error code returned:

try:
    content = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
    if e.code == 404:
        print "Not Found"
    elif e.code == 500:
        print "Internal Server Error"

Server Errors “URLError”. These occur frequently as well and seem to denote some sort of server or connection trouble, such as “Connection refused” or site does not exist. Usually, these are resolved by retrying the fetch. In Python, it is very important to note that HTTPError is a subclass of URLError, so when handling both exceptions distinctly, HTTPError must be caught first.

try:
    content = urllib2.urlopen(url).read()
except urllib2.HTTPError, e:
    ...
except urllib2.URLError, f:
    print f.reason

Other Bizarreness. The web is very chaotic. Sometimes weird stuff happens. The rare, elusive httplib.BadStatusLine exception technically means that the server returned an error code that the client does not understand, but it can also be thrown when the page being fetched is blank. On a recent project, I ran into a new one: httplib.IncompleteRead which has little documentation. Both of these issues can usually be resolved by retrying the fetch. Both of these pesky errors (and more) can be handled by simply catching their parent exception: httplib.HTTPException.

try:
    content = urllib2.urlopen(url).read()
except httplib.HTTPException:
    #you've encountered a rare beast. You win a prize!

Everything Deserves a Second Chance

One common reaction to any error is to just throw the record out. URLErrors errors are so common, that it is probably unwise to do that if you are using the data for something. Typically, these errors go away if you try again. I use the following loop to catch errors and react appropriately.

attempt = 0
while attempt < HTTP_RETRIES:
    attempt += 1
    try: 
        temp = urllib2.urlopen(url).read()
        break 
    except urllib2.HTTPError:
        break
    except urllib2.URLError:
        continue
    except httplib.HTTPException:
        continue
else:
    continue

This code attempts to fetch URL a maximum of HTTP_RETRIES times. If the fetch is successful, Python breaks out of the loop. If a URLError or HTTPException occurs, we move on to another attempt of the fetch. If we encounter an HTTP error (not found, restricted etc), give up. Depending on the error, we can modify the code to retry on certain errors, and abort on others, but for my purposes, I do not care.

The Comatose Crawler

If you have ever done a large scale crawl on a web site, you are bound to encounter a state where your crawler becomes comatose – it is running, maybe using system resources, but is not outputting anything or reporting progress. It looks like an infinite no-op loop. I have encountered this problem since I started doing web mining in 2006 and did not, until just this past weekend, realize exactly why it was happening and how to prevent it.

Your crawler has sunk in a swamp, and is essentially trapped. For whatever reason, the HTTP server your code is communicating with maintains an open connection, but sends no data. I suppose this could be a deadlock-type situation where the HTTP server is waiting for an additional request (?), and the crawler is waiting for output from the HTTP server. It was my misunderstanding that the HTTP protocol had a built-in timeout, and I was relying on it. This is apparently not the case. There is a simple way to avoid this swamp, by setting a timeout on the socket sending the HTTP request:

import socket
...
HTTP_TIMEOUT = 5
socket.setdefaulttimeout(HTTP_TIMEOUT)
...
handle = urllib2.urlopen("http://www.google.com")
content = handle.read()
...

If a request to the socket goes unanswered after HTTP_TIMEOUT seconds, Python throws a urllib2.URLError exception that can be caught. In my code, I just skip these troublemakers.

Traceback (most recent call last):
  File "", line 1, in ?
  File "/usr/lib64/python2.4/urllib2.py", line 130, in urlopen
    return _opener.open(url, data)
  File "/usr/lib64/python2.4/urllib2.py", line 358, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.4/urllib2.py", line 376, in _open
    '_open', req)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 1021, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.4/urllib2.py", line 996, in do_open
    raise URLError(err)
urllib2.URLError: urlopen error="" timed="" out=""

With enough experience, dedication, blood, sweat, tears, and caffeine, data mining the jungle known as the World Wide Web becomes both simple and fun. Happy web mining!

Some of the ones that I frequently read, or are written by Twitter friends/followers (in no particular order):