Visual Web Ripper: Using External Input Data Sources

Sometimes it is necessary to use external data sources to provide parameters for the scraping process. For example, you have a database with a bunch of ASINs and you need to scrape all product information for each one of them. As far as Visual Web Ripper is concerned, an input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values.

An input data source is normally used in one of these scenarios:

    To provide a list of input values for a web form
    To provide a list of start URLs
    To provide input values for Fixed Value elements
    To provide input values for scripts

Visual Web Ripper supports the following input data sources:

    SQL Server Database
    MySQL Database
    OleDB Database
    CSV File
    Script (A script can be used to provide data from almost any data source)

To see it in action you can download a sample project that uses an input CSV file with Amazon ASIN codes to generate Amazon start URLs and extract some product data. Place both the project file and the input CSV file in the default Visual Web Ripper project folder (My Documents\Visual Web Ripper\Projects).

For further information please look at the manual topic, explaining how to use an input data source to generate start URLs.


Source: http://extract-web-data.com/visual-web-ripper-using-external-input-data-sources/

Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

Microsys A1 Website Scraper Review

The A1 scraper by Microsys is a program that is mainly used to scrape websites to extract data in large quantities for later use in webservices. The scraper works to extract text, URLs etc., using multiple Regexes and saving the output into a CSV file. This tool is can be compared with other web harvesting and web scraping services.
How it works
This scraper program works as follows:
Scan mode

    Go to the ScanWebsite tab and enter the site’s URL into the Path subtab.
    Press the ‘Start scan‘ button to cause the crawler to find text, links and other data on this website and cache them.


Important: URLs that you scrape data from have to pass filters defined in both analysis filters and output filters. The defining of those filters can be set at the Analysis filters and Output filters subtabs respectively. They must be set at the website analysis stage (mode).
Extract mode

    Go to the Scraper Options tab
    Enter the Regex(es) into the Regex input area.
    Define the name and path of the output CSV file.
    The scraper automatically finds and extracts the data according to Regex patterns.

The result will be stored in one CSV file for all the given URLs.

There is a need to mention that the set of regular expressions will be run against all the pages scraped.
Some more scraper features

Using the scraper as a website crawler also affords:

    URL filtering.
    Adjustment of the speed of crawling according to service needs rather than server load.

If  you need to extract data from a complex website, just disable Easy mode: out press the  button. A1 Scraper’s full tutorial is available here.
Conclusion

The A1 Scraper is good for mass gathering of URLs, text, etc., with multiple conditions set. However this scraping tool is designed for using only Regex expressions, which can increase the parsing process time greatly.



Source: http://extract-web-data.com/microsys-a1-website-scraper-review/

EffeTech HTTP Sniffer Review

EffeTech Sniffer is an HTTP standalone sniffing application for the Windows operating system. The other HTTP traffic analyzers are available here.

The sniffer captures IP packets containing HTTP protocol, rebuilds the HTTP sessions, and reassembles files sent through HTTP protocol. It works to catch all the inner TCP packets, and IP addresses display.

As an ordinary sniffer, EffeTech shows the request/response headers, cookie and other details. However it does not work with HTTPS connections. The HTTP content and TCP packets are available for overview at the HTTP details page: in the toolbar menu go to Sniffer-> View details. No timeline is available.

As the sniffer captures, analyzes, parses and decodes HTTP, there is a slight delay in viewing the request lines on the application screen.

Exporting logs into HTML or CSV formats is also possible with this tool.




Source: http://extract-web-data.com/effetech-http-sniffer-review/

Outsource Data Entry Services to Reduce Labour Cost

India has become a hub for outsourcing services. One of the services being outsourced highly to India is data entry. Managing data is a tough task, especially for growing organizations. So, many of these organizations outsource their services to India.

Outsourcing data entry services to India helps companies in maintaining everyday records and details in proper order. India has many reputed firms dealing with our services. These companies have highly qualified and experienced experts that assist you in managing business related data aptly. Experts keep themselves updated with the latest technologies to live up to your expectations.

These can be availed both offline and online. Online services include managing data from e books, image files, and web browsers. Offline services include managing data from documents, papers and directories. Outsourcing data entry services to India will help you get an appropriate solution for data processing. You can also ask for a free trial of the data entering services before outsourcing your work to them. Other services that you can avail are word and data conversion, OCR clean up and PDF conversion.

By outsourcing your services to India, you can enjoy the benefit of saving up to 70% cost incurred on business operations. Thus, outsourcing will help you drive down the cost that can be further invested in the expansion of the business.

Data entry services can be outsourced regardless of the industry type. Be it retail, lodging, finance or real estate, this type of companies cover every field you can think of. By outsourcing tasks you can minimize the workload of your employees, thereby improving their efficiency and competence. This too puts you on the forefront as it reduces the labor cost up to a great extent without any effect on quality.

This article is written by Abhinav Singh and provided in courtesy of data entry outsourcing offering affordable data entry services.




Source: http://ezinearticles.com/?Outsource-Data-Entry-Services-to-Reduce-Labour-Cost&id=5805062

Outsource Data Entry - A Wise Business Decision

Getting the benefits of outsourcing data entry services for your business will be a wise choice. Many offshore companies guarantee quick and accurate data entry services. These companies offer data entry services from industry expert professionals and flexibility as per user requirements. All recent reports say, trend of outsourcing low priority work will continue to grow gradually.

In earlier days, outsourcing was thought as a temporary option of meeting particular objective, is now becoming the best industry option. Viewed as a temporary business solution, outsourcing is now a strategically important business decision. Outsourcing your services will reduce your costs with improved services.

Advantages of Data Entry Outsourcing

Data entry outsourcing gives you many business advantages include:

- By outsourcing one can easily concentrate on core business competencies and goals.
- In these cut throat competitive time, outsourcing is a cautious way of controlling expensive staffing cost. Person can get outsourcing services on per transaction basis, which ease the hurdles of having the possibility of firing staff members.
- By outsourcing you can get the advantage of economies of scale. If you work with an outsourcing company you will save your valuable money, probably boost your operational efficiency.
- By outsourcing your data-entry work your cost will be on per transaction basis which will allow you to easily predict your budget and give you the best budget planning.
- By outsourcing organizations do not have to worry about meeting time lines. As many outsourcing companies guarantee of in-time delivery which was already specified in user agreement and will not be longer concern to worry.
- Most of the outsourcing companies located in cheap offshore countries like India, Indonesia etc and having expertise of handling data entry operations.

Thus by outsourcing data-entry work organizations can get advantage in terms of time, money and efficiency which will obviously increase business productivity.

Author is related with data entry services providing firm ServicesDataEntry.co.uk. Firm outsource data entry services like online data entry services and many more.




Source: http://ezinearticles.com/?Outsource-Data-Entry---A-Wise-Business-Decision&id=2694032

Data Recovery Services - When You're Facing A Wipeout

Your computer files are the foundation of your business. What if one day you awaken to find that your computer has crashed, and the foundation of you business appears to have crumbled? Are those files nothing but dust on the winds of cyberspace? Or is there a way to gather up their bits and bytes, reassemble them, and lay the bricks of a new foundation?

There very well may be, but it requires the skilled handling of one of the many data recovery services which have come to the rescue of more computer-driven businesses than you might believe. And they have not retrieved data only for small business proprietors; data recovery services have been the saving of many a multi-million dollar operation or project. Data recovery services have also practiced good citizenship in recovering data erased from the hard drives of undesirables.

Finding Data Recovery Services

If you're someone who neglected, or never learned how, to back up your hard drive, it's time to call for help from one of the data recover service by doing an online search and finding one, if possible, nearby. If you have to settle for one of the data recovery services I another area, so be it. You're not in a position to quibble, are you?

You'll need to extract your non-functioning hard drive from your PC and send it out to have data recovery services administered. Whichever of the data recovery services company you have chosen will examine you hard drive's memory to determine how much of the data on it can be restored, and give you an estimate of the job's cost.

Only you are the expert on the importance of that data to your future, and only you can decide whether or not the price quoted by the data recovery services company is acceptable. If you think you can find a way to work around the lost data, simply tell the data recovery services company to return your hard drive.

What You'll Get For Your Money

But before you do that, consider exactly what the data recovery services will entail, and why they are not cheap. Your mangled hard drive will be taken to a clean room absolutely free of dust, and operated on with tools of surgical precision so that even the tiniest bits of functional data can be retrieved.

If their price still seems too high, ask the data recovery services company what their policy is if they find that they are unable to retrieve a meaningful amount of data. Many of them will not charge you if they cannot help your situation.

Data recovery services companies offer high-tech, high-cost solutions, but you won't find anyone else who can do what they do. So next time, backup your hard drive, but if your future is really at stake, then data recovery services are the best chance you have of getting it back.

You can also find more info on Data Recovery Program and Data Recovery Service Pcdatarecoveryhelp.com is a comprehensive resource to know about Data Recovery.



Source: http://ezinearticles.com/?Data-Recovery-Services---When-Youre-Facing-A-Wipeout&id=615548