Web Scraping data from different sites

I am looking for a few ideas on how can I solve a design problem I'm going to be faced with building a web scraper to scrape multiple sites. Writing the scraper(s) is not the problem, matching the data from different sites (which may have small differences) is.

For the sake of being generic assume that I am scraping something like this from two or more different sites:

    public class Data {
        public int id;
        public String firstname;
        public String surname;
        ....
    }

If i scrape this from two different sites, I will encounter the situation where I could have the following:

Site A: id=100, firstname=William, surname=Doe

Site B: id=1974, firstname=Bill, surname=Doe

Essentially, I would like to consider these two sets of data the same (they are the same person but with their name slightly different on each site). I am looking for possible design solutions that can handle this.

The only idea I've come up with is scraping the data from a third location and using it as a reference list. Then when I scrape site A or B I can, over time, build up a list of failures and store them in a list for each scraper so that it can know (if i find id=100 then i know that the firstname will be William etc). I can't help but feel this is a rubbish idea!

If you need any more info, or if you think my description is a bit naff, let me know!

Thanks,

DMcB


Source: http://stackoverflow.com/questions/23970057/web-scraping-data-from-different-sites

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

How Your Online Information is Stolen - The Art of Web Scraping and Data Harvesting

Web scraping, also known as web/internet harvesting involves the use of a computer program which is able to extract data from another program's display output. The main difference between standard parsing and web scraping is that in it, the output being scraped is meant for display to its human viewers instead of simply input to another program.

Therefore, it isn't generally document or structured for practical parsing. Generally web scraping will require that binary data be ignored - this usually means multimedia data or images - and then formatting the pieces that will confuse the desired goal - the text data. This means that in actually, optical character recognition software is a form of visual web scraper.

Usually a transfer of data occurring between two programs would utilize data structures designed to be processed automatically by computers, saving people from having to do this tedious job themselves. This usually involves formats and protocols with rigid structures that are therefore easy to parse, well documented, compact, and function to minimize duplication and ambiguity. In fact, they are so "computer-based" that they are generally not even readable by humans.

If human readability is desired, then the only automated way to accomplish this kind of a data transfer is by way of web scraping. At first, this was practiced in order to read the text data from the display screen of a computer. It was usually accomplished by reading the memory of the terminal via its auxiliary port, or through a connection between one computer's output port and another computer's input port.

It has therefore become a kind of way to parse the HTML text of web pages. The web scraping program is designed to process the text data that is of interest to the human reader, while identifying and removing any unwanted data, images, and formatting for the web design.

Though web scraping is often done for ethical reasons, it is frequently performed in order to swipe the data of "value" from another person or organization's website in order to apply it to someone else's - or to sabotage the original text altogether. Many efforts are now being put into place by webmasters in order to prevent this form of theft and vandalism.

Source:http://ezinearticles.com/?How-Your-Online-Information-is-Stolen---The-Art-of-Web-Scraping-and-Data-Harvesting&id=923976

Data Extraction - A Guideline to Use Scrapping Tools Effectively

So many people around the world do not have much knowledge about these scrapping tools. In their views, mining means extracting resources from the earth. In these internet technology days, the new mined resource is data. There are so many data mining software tools are available in the internet to extract specific data from the web. Every company in the world has been dealing with tons of data, managing and converting this data into a useful form is a real hectic work for them. If this right information is not available at the right time a company will lose valuable time to making strategic decisions on this accurate information.

This type of situation will break opportunities in the present competitive market. However, in these situations, the data extraction and data mining tools will help you to take the strategic decisions in right time to reach your goals in this competitive business. There are so many advantages with these tools that you can store customer information in a sequential manner, you can know the operations of your competitors, and also you can figure out your company performance. And it is a critical job to every company to have this information at fingertips when they need this information.

To survive in this competitive business world, this data extraction and data mining are critical in operations of the company. There is a powerful tool called Website scraper used in online digital mining. With this toll, you can filter the data in internet and retrieves the information for specific needs. This scrapping tool is used in various fields and types are numerous. Research, surveillance, and the harvesting of direct marketing leads is just a few ways the website scraper assists professionals in the workplace.

Screen scrapping tool is another tool which useful to extract the data from the web. This is much helpful when you work on the internet to mine data to your local hard disks. It provides a graphical interface allowing you to designate Universal Resource Locator, data elements to be extracted, and scripting logic to traverse pages and work with mined data. You can use this tool as periodical intervals. By using this tool, you can download the database in internet to you spread sheets. The important one in scrapping tools is Data mining software, it will extract the large amount of information from the web, and it will compare that date into a useful format. This tool is used in various sectors of business, especially, for those who are creating leads, budget establishing seeing the competitors charges and analysis the trends in online. With this tool, the information is gathered and immediately uses for your business needs.

Another best scrapping tool is e mailing scrapping tool, this tool crawls the public email addresses from various web sites. You can easily from a large mailing list with this tool. You can use these mailing lists to promote your product through online and proposals sending an offer for related business and many more to do. With this toll, you can find the targeted customers towards your product or potential business parents. This will allows you to expand your business in the online market.

There are so many well established and esteemed organizations are providing these features free of cost as the trial offer to customers. If you want permanent services, you need to pay nominal fees. You can download these services from their valuable web sites also.

Source:http://ezinearticles.com/?Data-Extraction---A-Guideline-to-Use-Scrapping-Tools-Effectively&id=3600918

Online Data Entry - How Online Data Entry is Useful in Business?

In last article about "Online Data Entry Projects - Grab An Online Audience by Data Entry", I mentioned some newly provoked ideas which are currently outsourced by various companies around the world such as United States, United Kingdom, United Arab Emirates, Canada and Others. In this article, I emphasize on some symbolized and basic online data entry techniques that most of the businesses require. Here we go:

Online Compilation from Websites: Company requires a huge amount of information to run business smoothly. You require details of raw material suppliers, Machine suppliers, maintenance service vendor, product dispenser and many more. If company executive have compiled information, they can act promptly and complete the task quickly. Online websites are great source to search for particular details. By outsourcing online compilation from website task to some reputed data entry company, you can get highly accurate information that helps you in taking powerful decisions.

Online Business Card Entry: Business Cards are much helpful not only in getting a better idea about business of someone but also getting the contact information easily. Sometimes it happens that you misplace the business card when you require it urgently. If you have entered business card information in your PC, you can easily search for such. You can quickly contact the needed person and solve your queries promptly.

Online Catalog Data Entry: Catalog is the most powerful tool to sell your products. If you don't have informative and attractive catalog, you can not convert your viewers into customers. It is also possible that you are avoiding online potential customer by not uploading your catalog online. However, online catalog data entry can be the solution for such need. Insert good amount of information in your catalog and attract more visitors. You can get not only good business from this but also provoke your brand.

Online Survey Form Entry: Survey is very important tool to check the mindset of customers. The data mention in survey forms are very important while upgrading product, emerging into new field, changing strategy, branding and marketing products. The information is only useful if it is precise and quickly available. Online survey form entry can help you in making surveyed information organized so that you can clearly divert your focus on right direction.

These are various online data typing projects which are helpful for your business to generate more efficient environment and increase the productivity and profitability. You can meet high goals with efficient environment and increased productivity.

Source: http://ezinearticles.com/?Online-Data-Entry---How-Online-Data-Entry-is-Useful-in-Business?&id=4505450

Complete SEO Services Introduces Affordable Content Writing Services for Effective SEO

Complete SEO Services, one of the UK’s leading search engine optimization companies has introduced an article-writing service for effective SEO. The UK based SEO company who has a vast amount of knowledge of how professional content can help with the improvement of traffic and search engine rankings have put that knowledge together with their professional writing team and launched a content writing service for effective SEO.

The new content writing service is aimed at helping marketing companies and website owners achieve their SEO goals. Content is king is a popular saying with professional website marketing companies and the saying has become even truer with all the changes that popular search engines have made to the way they rank websites. Offering quality articles that are well written and engage the target audience will not only help drive more traffic to the website and keep people on their for longer, it will also encourage search engines to rank the site higher.

A spokesman for Complete SEO Services said: “For SEO to be effective the website needs to have quality content. There are a lot of companies out there who offer spinning articles that can damage a website. A site needs to have up to date content that has been professionally written to keep visitors returning and to help improve search engine rankings.”

The SEO Services Company writes articles for companies all around the world from small businesses to large companies, but providing the same quality service no matter what size business they are.

Website owners have seen how all the recent changes with popular search engines have changed the way their sites have been ranked. With search engines now concentrating on sites that offer quality content, it is important to make sure websites have up to date content that has been written by professionals to help with positive SEO.

Source: http://digitaljournal.com/pr/1973692

Collecting and Managing Data for Small Businesses

Is your company doing well? Are you profitable? How much longer will it be until you are? Where can you cut expenses? What are your most profitable products or services? Which are the least profitable? What the shelf-life of a particular product? How effective is your marketing strategy?

As business leaders, this type of information is the type of information that we need to have on hand in order to make decisions about the company. Planning solely on what is in the bank, or focusing only on one aspect of what makes your company a) successful or b) keeps it out of closing is a very poor way of operating. This is pure tunnel vision.

If you consider, for a moment, two tools that are widely used in the business world - Porter's Five Forces and the SWOT analysis, you'll notice that part of the analysis is based on things that impact the business - are outside of the business's control. As business leaders, you know that strategically, this cannot occur just in exercise, but must exist in the way that you do business. Monitoring, making adjustments and acting must be an ongoing mentality if your goal is to build an extraordinary business. Several recommendations we have made to clients include:

    Understand what questions you want to answer. Here are some samples:

    How do we know when we can purchase a new building or expand capacity

    How do we know how effective our sales people are

    How do we know how effective our marketing and other business development activities are

    Understand what kind of data you need to collect in order to make your decisions. Typically, these are going to be things such as your financials - sales, cost of goods sold, expenses, profit, investments, interest and taxes, your business development activities, manufacturing costs and rates, etc.

    Determine how to collect the data - including what is feasible. Take into consideration how you operate - does it need to be mobile? does the information need to be housed in a cloud?

    Determine how the data needs to be delivered. If you have a ton of data and like to drill down from "high level" analysis down to the details, perhaps you want something more visual. If you like to play with the numbers yourself and run scenarios, perhaps you like to play with the raw data.

•    Decide how much your level of investment. Consider this:

•    No Investment - If you don't get the data to make decisions, the likelihood of success is minimalized.

•    Your Time - If you collect and mine the data yourself, what else could or should you be doing to build or grow the business.

•    Your Resources - You could have a qualified employee collect and mine the data for you.

•    Your Money - You could invest in a software solution - be it customized, off the shelf, or a combination of the two - that could collect the data. We have used and recommend a product called Work, Etc., to centrally house most transaction that occur in the business in order to give us a single data-collection source.

    Research solutions that will work best for your company, considering the factors we pointed out above. You may consider other factors that are specific to your company, such as the ability to sync with certain existing solutions or the ability to be housed on a centrally located or remote server, etc.. May companies look at Open Source solutions such as xTurple or SugarCRM. Despite your choice, consider the total cost of ownership of the solution before investing in it.

    Determine how the solution will be implemented and how training on the software will roll out. This may mean hiring a consultant, going to a class or classes, investing in an online training solution or spending time with tech support for a self-install. Consider the different types of investment.

    Plan to re-enforce the need to use whatever solution is recommended. Change takes time. Habits take time. By providing some structure, you will increase the likelihood of success.

Your company's ability to collect and decipher the data from the activities in and around your company can be the determine factor between a series of successful ventures and a series of hit-or-miss activities. Even the simplest data-collection activities should help you determine your company's path.

Source:http://ezinearticles.com/?Collecting-and-Managing-Data-for-Small-Businesses&id=6923386