Friday, 21 April 2017

Loading in various amount of files into database

I'm having a various amount(0-10) of .dem file that i'm manually receiving each day on about 180 mb each. i'm at the moment having a script on an application server with limited amount of space being able to read the .dem file and add it into a database, however i'm not sure what the best approach in terms of storing the .dem and then automatically run cron jobs to retrieve these files and run the script to put in the database. the .dem consist of match data, that is connected to a specific tournament, however this is not present in the .dem. so far i'm thinking of creating a seperate ftp server where i daily just add the .dem files into in a tournament seperate folder and inside that folder a config.json file with tournament specific information, like below. however not sure what is the best approach?

- ftp server
    - tournaments
       - tournament 1
           - config.json
           - match1.dem
           - match2.dem
       - tournament 2
           - config.json
           - match1.dem



via Peter Pik

No comments:

Post a Comment