Comments on: Parsing Web Pages

By: Jon

Jon — Mon, 26 Oct 2009 06:00:59 +0000

I work for a company that is trying to scrape several sites to get their pricing competitive with everyone else online. I spent a good month learning the programs out there and using them until I found one that actually worked. I am very new when it comes to web scraping but not only did I find something that worked, it blew me way. These guys have a great vision making the system open enough to have the data harvested and then sent back to the user via REST, FTP, or email. Try mozenda.com for your web scraper I think you will be very happy. I wish I had read this before spending so much time trying the other ones.

In full disclosure I have to say that I Invested in this company (Mozenda Inc.) after trying their tool. The down side is that they charged a monthly fee.

Hope this helps,
Jon

By: rsmith

rsmith — Sun, 18 Oct 2009 19:15:35 +0000

You can do Python Parsing. Parsing web pages using python, which is a newer programming language than php and perl, can give you great functional control of the output you are looking for.

There has been a number of Python tools out there that can help parsing out data structures from a web page, log file or general text files. Just to name a few of these tools:

PyGgy
PyGgy is a python package for generating parsers and lexers in python. The PyGgy distribution contains two tools:

PyLly - (Pronounced "pile-ey") A lexer generator that generates DFA tables for lexing tokens.
PyGgy - (Pronounced "piggy") A parser generator that generates SLR tables for a GLR parsing engine.

Pyparsing
pyparsing is a general parsing module for Python. Grammars are implemented directly in the client code using parsing objects, instead of externally, as with lex/yacc-type tools. Includes simple examples for parsing SQL, CORBA IDL, and 4-function math.

picoparse
Picoparse is a very small parser / scanner library for Python. It is built to make constructing parsers straight forward, and without the complications regular expressions bring to the table.

Parsing web pages with Python does require a pretty good understanding of the Python programming language but if you already know it then getting the results should not be a big hassle.

By: frankg

frankg — Sun, 18 Oct 2009 18:32:05 +0000

I looked at that web parsing program you mentioned and it is a huge time saver. In the past I have used the java based web page parser library from sourceforge. Worked very well for me but you must know or understand java. Here is a quote from their webpage: “HTML Parser is a Java library used to parse HTML in either a linear or nested fashion. Primarily used for transformation or extraction, it features filters, visitors, custom tags and easy to use JavaBeans. It is a fast, robust and well tested package.”

For your readers to review the HTML Parser program, here’s the link.
http://htmlparser.sourceforge.net/