=================================================================
=================================================================
LinkCrawler
By Carlos Alberto Umanzor Arguedas
Current Version: 3.0.0
=================================================================
=================================================================


TABLE OF CONTENTS

1. System Requirements
2. OS Supported
3. Changes in Version 3.0.0
4. Expected Features in Version 3.x Branch
5. Instructions
6. Known issues
7. Special Message

_________________________________________________________________

=================================================================
1. System Requirements
=================================================================

Minimum:

Broadband Internet Connection
Major browsers installed (IE, Firefox 8 or Superior)
Oracle JRE 7 or Superior (JRE 6 and 5 are not fully tested)


=================================================================
2. OS Supported
=================================================================

Tested:
Windows 7 32/64 bits
CentOS 6
Ubuntu 13 (Known issue is present, the GTK look and feel makes the UI ugly, a fix is coming)

Theorically, it should work on any OS that supports Java.


=================================================================
3. Changes in Version 3.0.0
=================================================================

- Refactoring the Linkcrawler source from Netbeans to a Maven project.
- New report available: JSON format, useful if you want to parse those file in other projects.
- HTML Report 2.0 with better visuals and user experience.
- Smaller app size vs previous version: 11.2 MB on 3.0.0v vs 30.2 MB on 2.3.0v.
- Fixed a bug that prevents crawling on sites that are behind a CDN (tested with Cloudflare and Amazon CDN services).

=================================================================
4. Expected Features in Version 3.x Branch
=================================================================

- Command Line (still working on it)
- A maven plugin
- Migrating UI into JavaFX


=================================================================
5. Instructions
=================================================================


----------------- How to crawl -------

On website to check field enter an Absolute url (including http:// protocol), for example:

http://carlosumanzor.com
http://mysite.com

Note: Please use the main site URL in order to crawl the entire site from a good central point.

Then click start and the application will perform the "crawl" job


----------------- How to view and save log -------

On version 3.0.0, the log is automatically generated. The log is available at the Logs folder located in the same position as the Linkcrawler jar file.
Make sure you are executing the application with administrator privileges in order to create folders and files.



----------------- How to generate a report -------

Once a "crawl" job has finished, click on Reports then "Save in format...", finally choose HTML, the report will be generated
in the same folder as the LinkCrawler application. make sure you are executing the application with administrator privileges in order to create folders


----------------- How to Use Exclusion list -------

Simple, just type a full URL to exclude a webpage, for example:

http://mysite.com/page.something

Or, Type a partial url to ignore a lot of webpages, for example:


http://mysite.com/calendar/

In this case, LinkCrawler will ignore anything that starts with "http://mysite.com/calendar/"


----------------- How to verify a Sitemap -------

In order to verify if your sitemap is valid for google, click on the XML Sitemap Verification tab, then type the url of the sitemap then click on Check Sitemap. Also you can copy the site used when crawling as well by using the copy button.

Note: Linkcrawler will attempt to use sitemap.xml in case you enter the main site url only, for example http://carlosumanzor.com.

In case of any errors, a button will be enable showing how many errors were found in the execution.

==================================================
6. Known issues
==================================================

Crawling Big websites with hundreds of pages is still slow. On 3.0.0, after a huge refactoring, the performance was improved but still, it's not what I expect to achieve
GTK Look and feel in some Linux distro makes the UI ugly. I'm planning to migrate the UI to JavaFX in order to solve this, but it will take time
Unable to detect links dynamically added by jquery. Tricky, but possible.
Unable to detect infinite link generators such a Calendar widgets (the bad ones), Use exclusion list feature as a Workaround


==================================================
7. Special Message
==================================================

Thank you for giving a chance to this application. Please, let me know what do you think at https://sourceforge.net/projects/linkcrawler
any feedback is appreciated.

Found a bug?

You can report it here at https://sourceforge.net/p/linkcrawler/tickets (Registration may be required);
otherwise send me an email to cumanzor@carlosumanzor.com with the word BUG in the subject and I will open the ticket for you.

Visit my site http://carlosumanzor.com and check any other project that I may have.
