common::DataReaderIterator< T > Class Template Reference

A multi-threaded iterator that performs decerealization of objects and returns data in batches. More...

#include <datareader.h>

Public Member Functions
	DataReaderIterator (std::vector< std::string > paths, size_t numThreads, size_t batchSize, std::string prefix=std::string(), DataReaderThreadInitF init=DataReader_NoopF)
	~DataReaderIterator ()
bool	hasNext ()
std::vector< T >	next ()

Detailed Description

template<typename T>
class common::DataReaderIterator< T >

A multi-threaded iterator that performs decerealization of objects and returns data in batches.

Here, batches means instances of std::vector<T>.

The iterator will read data from all files specified by the list of paths passed to the constructor once. The order of files is retained and will be reflected in the resulting batches.

Call next() to retrieve the next batch of data. It will block until enough data is available to form a batch with the requested size. If the iterator cannot advance anymore (hasNext() returns false), the call with throw an exception. The last batch returned by this function may have less that batchSize elements or be empty, depending on the actual data.

For data files that cannot be decerealized (e.g. because the file cannot be accessed, or because decerialization failed), the iterator will print a message via glog but otherwise resume operation as usual.

You will probably want to use this via DataReader<T>.

Constructor & Destructor Documentation

template<typename T>

common::DataReaderIterator< T >::DataReaderIterator ( std::vector< std::string >  paths, size_t  numThreads, size_t  batchSize, std::string  prefix = std::string(), DataReaderThreadInitF  init = DataReader_NoopF  )

inline

template<typename T >

common::DataReaderIterator< T >::~DataReaderIterator

(

)

Member Function Documentation

template<typename T >

bool common::DataReaderIterator< T >::hasNext

(

)

template<typename T >

std::vector< T > common::DataReaderIterator< T >::next

(

)

---

The documentation for this class was generated from the following file:

• common/datareader.h