com.gisgraphy.domain.geoloc.importer
Class AbstractImporterProcessor

java.lang.Object
  extended by com.gisgraphy.domain.geoloc.importer.AbstractImporterProcessor
All Implemented Interfaces:
IImporterProcessor
Direct Known Subclasses:
AdmExtracter, GeonamesAdm1Importer, GeonamesAdm2Importer, GeonamesAdm3Importer, GeonamesAdm4Importer, GeonamesAlternateNamesExtracter, GeonamesAlternateNamesImporter, GeonamesCountryImporter, GeonamesFeatureImporter, GeonamesLanguageImporter, OpenStreetMapImporter

public abstract class AbstractImporterProcessor
extends java.lang.Object
implements IImporterProcessor

Base class for all geonames processor. it provides session management and the ability to process one or more CSV file

Author:
David Masclet

Field Summary
protected  java.lang.String COMMENT_START
          Lines starting with this prefix are considered as comments
protected  java.io.File currentFile
          The current processed file
protected  ImporterConfig importerConfig
           
protected  java.io.BufferedReader in
          The bufferReader for the current read Geonames file
protected  IInternationalisationService internationalisationService
           
protected static org.slf4j.Logger logger
          The logger
protected  int readFileLine
           
protected  ImporterStatus status
           
protected  java.lang.String statusMessage
           
protected  int totalReadLine
           
protected  org.springframework.transaction.PlatformTransactionManager transactionManager
          The transaction manager
 
Constructor Summary
AbstractImporterProcessor()
          Default constructor
 
Method Summary
protected  void checkNumberOfColumn(java.lang.String[] fields)
          Utility method which throw an exception if the number of fields is not the one expected (retrieved by getNumberOfColumns())
protected  int countLines()
           
protected static java.lang.String dumpFields(java.lang.String[] fields)
           
protected abstract  void flushAndClear()
          Should flush and clear all the Daos that are used by the processor.
 java.lang.String getCurrentFileName()
           
protected abstract  java.io.File[] getFiles()
           
protected  int getMaxInsertsBeforeFlush()
           
protected abstract  int getNumberOfColumns()
           
 long getNumberOfLinesToProcess()
           
 long getReadFileLine()
          The number of read line for the current processed file
 ImporterStatus getStatus()
           
 java.lang.String getStatusMessage()
           
 long getTotalReadLine()
          The number of read line for all the processed file
protected static boolean isEmptyField(java.lang.String[] fields, int position, boolean required)
          Check that the array is not null, and the fields of the specified position is not empty (after been trimed)
protected  boolean isEndOfDocument()
           
 void process()
          Manage the transaction, flush Daos, and process all files to be processed
protected abstract  void processData(java.lang.String line)
          Process a read line of the geonames file, must be implemented by the concrete class
 int readLineAndProcessData()
          Process the line if needed (is not a comment, should ignore first line, is end of document,...)
 void resetStatus()
          Reset status fields, it should be done when the import has been canceled
protected abstract  void setCommitFlushMode()
          Will flush after every commit
 void setImporterConfig(ImporterConfig importerConfig)
           
 void setInternationalisationService(IInternationalisationService internationalisationService)
           
 void setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)
           
protected  void setup()
          Template method that can be override.
 boolean shouldBeSkipped()
           
protected abstract  boolean shouldIgnoreComments()
          Whether the filter should ignore the comments (i.e. lines starting with #)
protected abstract  boolean shouldIgnoreFirstLine()
          Template Method : Whether the processor should ignore the first line of the input
protected  void tearDown()
          Template method that can be override.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface com.gisgraphy.domain.geoloc.importer.IImporterProcessor
rollback
 

Field Detail

totalReadLine

protected int totalReadLine

readFileLine

protected int readFileLine

statusMessage

protected java.lang.String statusMessage

status

protected ImporterStatus status

internationalisationService

protected IInternationalisationService internationalisationService

importerConfig

protected ImporterConfig importerConfig

logger

protected static final org.slf4j.Logger logger
The logger


COMMENT_START

protected java.lang.String COMMENT_START
Lines starting with this prefix are considered as comments


in

protected java.io.BufferedReader in
The bufferReader for the current read Geonames file


transactionManager

protected org.springframework.transaction.PlatformTransactionManager transactionManager
The transaction manager


currentFile

protected java.io.File currentFile
The current processed file

Constructor Detail

AbstractImporterProcessor

public AbstractImporterProcessor()
Default constructor

Method Detail

shouldIgnoreFirstLine

protected abstract boolean shouldIgnoreFirstLine()
Template Method : Whether the processor should ignore the first line of the input

Returns:
true if the processor should ignore first line

flushAndClear

protected abstract void flushAndClear()
Should flush and clear all the Daos that are used by the processor. This avoid memory leak


setCommitFlushMode

protected abstract void setCommitFlushMode()
Will flush after every commit

See Also:
flushAndClear()

getNumberOfColumns

protected abstract int getNumberOfColumns()
Returns:
the number of fields the processed Geonames file should have

shouldIgnoreComments

protected abstract boolean shouldIgnoreComments()
Whether the filter should ignore the comments (i.e. lines starting with #)

See Also:
COMMENT_START

setup

protected void setup()
Template method that can be override. This method is called before the process start. it is not called for each file processed.


getFiles

protected abstract java.io.File[] getFiles()
Returns:
The files to be process
See Also:
ImporterHelper

getCurrentFileName

public java.lang.String getCurrentFileName()
Specified by:
getCurrentFileName in interface IImporterProcessor
Returns:
The name of the file currently processed or null if no file is processed

readLineAndProcessData

public int readLineAndProcessData()
                           throws ImporterException
Process the line if needed (is not a comment, should ignore first line, is end of document,...)

Returns:
The number of lines that have been processed for the current processed file
Throws:
ImporterException - if an error occurred

processData

protected abstract void processData(java.lang.String line)
                             throws ImporterException
Process a read line of the geonames file, must be implemented by the concrete class

Parameters:
line - the line to process
Throws:
ImporterException

process

public void process()
Manage the transaction, flush Daos, and process all files to be processed

Specified by:
process in interface IImporterProcessor

shouldBeSkipped

public boolean shouldBeSkipped()
Specified by:
shouldBeSkipped in interface IImporterProcessor
Returns:
true if the processor should Not be executed

tearDown

protected void tearDown()
Template method that can be override. This method is called after the end of the process. it is not called for each file processed. You should always call super.tearDown() when you overide this method


isEmptyField

protected static boolean isEmptyField(java.lang.String[] fields,
                                      int position,
                                      boolean required)
Check that the array is not null, and the fields of the specified position is not empty (after been trimed)

Parameters:
fields - The array to test
position - the position of the field to test in the array
required - if an exception should be thrown if the field is empty
Returns:
true is the field of the specifed position is empty
Throws:
MissingRequiredFieldException - if the fields is empty and required is true

dumpFields

protected static java.lang.String dumpFields(java.lang.String[] fields)
Parameters:
fields - The array to process
Returns:
a string which represent a human readable string of the Array

checkNumberOfColumn

protected void checkNumberOfColumn(java.lang.String[] fields)
Utility method which throw an exception if the number of fields is not the one expected (retrieved by getNumberOfColumns())

Parameters:
fields - The array to check
See Also:
getNumberOfColumns()

isEndOfDocument

protected boolean isEndOfDocument()
Returns:
true if the end of the document for the current processed file is reached

getReadFileLine

public long getReadFileLine()
Description copied from interface: IImporterProcessor
The number of read line for the current processed file

Specified by:
getReadFileLine in interface IImporterProcessor
See Also:
IImporterProcessor.getTotalReadLine()

getTotalReadLine

public long getTotalReadLine()
Description copied from interface: IImporterProcessor
The number of read line for all the processed file

Specified by:
getTotalReadLine in interface IImporterProcessor
See Also:
IImporterProcessor.getReadFileLine()

setTransactionManager

public void setTransactionManager(org.springframework.transaction.PlatformTransactionManager transactionManager)

setImporterConfig

public void setImporterConfig(ImporterConfig importerConfig)

countLines

protected int countLines()
Returns:
the number of line to process

getNumberOfLinesToProcess

public long getNumberOfLinesToProcess()
Specified by:
getNumberOfLinesToProcess in interface IImporterProcessor
Returns:
The number of line the processor will process. (it is not the number of lines left!)

getStatus

public ImporterStatus getStatus()
Specified by:
getStatus in interface IImporterProcessor
Returns:
The current status of the importer

getMaxInsertsBeforeFlush

protected int getMaxInsertsBeforeFlush()
Returns:
The option
See Also:
ImporterConfig.setMaxInsertsBeforeFlush(int)

resetStatus

public void resetStatus()
Description copied from interface: IImporterProcessor
Reset status fields, it should be done when the import has been canceled

Specified by:
resetStatus in interface IImporterProcessor

getStatusMessage

public java.lang.String getStatusMessage()
Specified by:
getStatusMessage in interface IImporterProcessor
Returns:
A text Message for the importer

setInternationalisationService

public void setInternationalisationService(IInternationalisationService internationalisationService)


Copyright © 2010. All Rights Reserved.