com.gisgraphy.importer
Class AbstractAdvancedImporterProcessor

java.lang.Object
  extended by com.gisgraphy.importer.AbstractSimpleImporterProcessor
      extended by com.gisgraphy.importer.AbstractAdvancedImporterProcessor
All Implemented Interfaces:
IImporterProcessor

public abstract class AbstractAdvancedImporterProcessor
extends AbstractSimpleImporterProcessor

Import the street from an (pre-processed) openStreet map data file in with all the processor capacity.

Author:
David Masclet

Field Summary
protected  ImporterException exception
           
protected  FileSpliter fileSpliter
           
protected  IIdGenerator idGenerator
           
protected static org.slf4j.Logger logger
          The logger
protected  List<File> splitedFiles
           
protected  org.springframework.scheduling.backportconcurrent.ThreadPoolTaskExecutor threadPoolTaskExecutor
           
protected  AtomicInteger totalReadLineMultithreaded
           
 
Fields inherited from class com.gisgraphy.importer.AbstractSimpleImporterProcessor
COMMENT_START, currentFile, importerConfig, in, internationalisationService, readFileLine, status, statusMessage, totalReadLine, transactionManager, txDefinition, txStatus
 
Constructor Summary
AbstractAdvancedImporterProcessor()
           
 
Method Summary
abstract  Runnable createTask(File file)
           
protected  void flushAndClear()
          Should flush and clear all the Daos that are used by the processor.
 String getCurrentFileName()
           
protected  int getNumberOfColumns()
           
 long getNumberOfLinesToProcess()
           
 int getNumberOfProcessors()
           
 int getNumberOfTaskExecutors()
           
 long getReadFileLine()
          The number of read line for the current processed file
protected abstract  List<File> getSplitedFiles()
           
protected  int getSplitLength(int nblines)
           
 long getTotalReadLine()
          The number of read line for all the processed file
 int incrementReadedFileLine(int increment)
           
 void process()
          Manage the transaction, flush Daos, and process all files to be processed
protected  void processData(String line)
          Process a read line of the geonames file, must be implemented by the concrete class
 void setIdGenerator(IIdGenerator idGenerator)
           
protected  void setup()
          Template method that can be override.
protected  boolean shouldIgnoreComments()
          Whether the filter should ignore the comments (i.e. lines starting with #)
protected  boolean shouldIgnoreFirstLine()
          Template Method : Whether the processor should ignore the first line of the input
protected  void splitFiles()
           
 
Methods inherited from class com.gisgraphy.importer.AbstractSimpleImporterProcessor
checkNumberOfColumn, commit, countLines, decrementReadedFileLine, dumpFields, getFiles, getMaxInsertsBeforeFlush, getStatus, getStatusMessage, isEmptyField, isEndOfDocument, needCommit, onFileProcessed, processError, readLineAndProcessData, resetStatus, rollbackTransaction, setCommitFlushMode, setImporterConfig, setInternationalisationService, setTransactionManager, shouldBeSkipped, startTransaction, tearDown
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface com.gisgraphy.importer.IImporterProcessor
rollback
 

Field Detail

idGenerator

protected IIdGenerator idGenerator

totalReadLineMultithreaded

protected AtomicInteger totalReadLineMultithreaded

threadPoolTaskExecutor

protected org.springframework.scheduling.backportconcurrent.ThreadPoolTaskExecutor threadPoolTaskExecutor

fileSpliter

protected FileSpliter fileSpliter

splitedFiles

protected List<File> splitedFiles

logger

protected static final org.slf4j.Logger logger
The logger


exception

protected ImporterException exception
Constructor Detail

AbstractAdvancedImporterProcessor

public AbstractAdvancedImporterProcessor()
Method Detail

setup

protected void setup()
Description copied from class: AbstractSimpleImporterProcessor
Template method that can be override. This method is called before the process start. it is not called for each file processed.

Overrides:
setup in class AbstractSimpleImporterProcessor

getSplitedFiles

protected abstract List<File> getSplitedFiles()

splitFiles

protected void splitFiles()

getSplitLength

protected int getSplitLength(int nblines)

process

public void process()
Description copied from class: AbstractSimpleImporterProcessor
Manage the transaction, flush Daos, and process all files to be processed

Specified by:
process in interface IImporterProcessor
Overrides:
process in class AbstractSimpleImporterProcessor

createTask

public abstract Runnable createTask(File file)

incrementReadedFileLine

public int incrementReadedFileLine(int increment)
Overrides:
incrementReadedFileLine in class AbstractSimpleImporterProcessor
Parameters:
increment -
Returns:
the incremented value

processData

protected void processData(String line)
                    throws ImporterException
Description copied from class: AbstractSimpleImporterProcessor
Process a read line of the geonames file, must be implemented by the concrete class

Specified by:
processData in class AbstractSimpleImporterProcessor
Parameters:
line - the line to process
Throws:
ImporterException

getCurrentFileName

public String getCurrentFileName()
Specified by:
getCurrentFileName in interface IImporterProcessor
Overrides:
getCurrentFileName in class AbstractSimpleImporterProcessor
Returns:
The name of the file currently processed or null if no file is processed

getReadFileLine

public long getReadFileLine()
Description copied from interface: IImporterProcessor
The number of read line for the current processed file

Specified by:
getReadFileLine in interface IImporterProcessor
Overrides:
getReadFileLine in class AbstractSimpleImporterProcessor
See Also:
IImporterProcessor.getTotalReadLine()

getTotalReadLine

public long getTotalReadLine()
Description copied from interface: IImporterProcessor
The number of read line for all the processed file

Specified by:
getTotalReadLine in interface IImporterProcessor
Overrides:
getTotalReadLine in class AbstractSimpleImporterProcessor
See Also:
IImporterProcessor.getReadFileLine()

shouldIgnoreComments

protected boolean shouldIgnoreComments()
Description copied from class: AbstractSimpleImporterProcessor
Whether the filter should ignore the comments (i.e. lines starting with #)

Specified by:
shouldIgnoreComments in class AbstractSimpleImporterProcessor
See Also:
AbstractSimpleImporterProcessor.COMMENT_START

shouldIgnoreFirstLine

protected boolean shouldIgnoreFirstLine()
Description copied from class: AbstractSimpleImporterProcessor
Template Method : Whether the processor should ignore the first line of the input

Specified by:
shouldIgnoreFirstLine in class AbstractSimpleImporterProcessor
Returns:
true if the processor should ignore first line

getNumberOfColumns

protected int getNumberOfColumns()
Specified by:
getNumberOfColumns in class AbstractSimpleImporterProcessor
Returns:
the number of fields the processed Geonames file should have

getNumberOfProcessors

public int getNumberOfProcessors()

getNumberOfTaskExecutors

public int getNumberOfTaskExecutors()

setIdGenerator

public void setIdGenerator(IIdGenerator idGenerator)

flushAndClear

protected void flushAndClear()
Description copied from class: AbstractSimpleImporterProcessor
Should flush and clear all the Daos that are used by the processor. This avoid memory leak

Specified by:
flushAndClear in class AbstractSimpleImporterProcessor

getNumberOfLinesToProcess

public long getNumberOfLinesToProcess()
Specified by:
getNumberOfLinesToProcess in interface IImporterProcessor
Overrides:
getNumberOfLinesToProcess in class AbstractSimpleImporterProcessor
Returns:
The number of line the processor will process. (it is not the number of lines left!)


Copyright © 2012. All Rights Reserved.