CSV-based database?

#1

I’m building a long CSV file with ~10k rows of filenames of images with many labels (from image recognition services). With this I plan to do image compositions based on these labels. I was wondering if I could use plain CSV files for the queries (high speed is not a concern, usually is just to choose and preload a bunch of images from these queries). The question are: what could be the best approach in this case? (I would need to do queries like contains/begin with/equals/etc and compare or match different fields) any addon that could be helpful? I should consider some light DB?

Thanks for any light.

#2

For ~10k rows, you should be able create a simple data structure like:

struct CSVRow {
  std::string column2;
  int column1;
};

// read CSV using ofBuffer
// parse rows using ofSplitString, 
// convert column tokens from string to number using ofToInt(), etc

std::vector<CSVRow> rows;

Once this is loaded into memory, searches should be very quick. You can optimize searches by sorting the vector in various ways. Then use for loops or std::search, std::find, etc to quickly search the rows vector.

1 Like
#3

CSV is just comma separated lines, so accessing each without loading the entire file should be possible with istream - I think ofBuffer (getLines) does this, and for more complicated structures there is serialisation - via something like ofxCereal