4: Import data into your development environment

You need to import Postgres data and Elasticsearch data. You will do these in separate steps.

Postgres

Prerequisites

• You will use the oma image for this task.

• You can access the production data, meaning you need VPN access to production (or to be supplied a data.yml file).

• You have created your oma_development database and loaded the database schema.

Dumping data

Skip this step if a data.yml has been supplied to you to by a colleague.

• Login to production_local: $>dcplr oma bash --login

• Type $>bundle installto bundle your local gems.

• Type $>bundle exec rake db:data:dumpwhich will not give any logging, but is dumping the production database into a data.yml file that resides in your db folder.

• Once complete, check that the data.yml file is there.

• Exit production_local by typing: $>exit

Loading data

• If a data.yml file has been supplied to you by a colleague, place this in the ./db folder. Otherwise, continue.

• Login to development: $>dcdr oma bash --login

• Type $>bundle installto bundle your local gems

• Type $>bundle exec rake db:data:loadto load your data. This will take a while.

• Type $>bundle exec rails cand then typeProject.countto ensure you have data in your development environment.

You are done with the Postgres portion of this task.

ElasticSearch

Importing data for ElasticSearch is different than for Postgres, in that we open a connection to the production servers and copy the data in real-time (no intermediate files). This action is performed in the oma-models repo, so you need to have that mapped into your environment by your docker-compose.override.yml file (the default). You will also choose a single project to import data for.

Loading data

• Type $dcdr oma bash --login

• Type $>cd ../oma-modelsto change to the oma-models repo.

• Ensure you have an config/application_production_local.yml in your config folder, and that its connecting to your production ES database.

• Ensure you have a config/application_development.yml file in your config folder, and that it's connecting to your development ES database.

• Type $>bundle installto bundle gems.

• Type $>bundle exec rake consoleto load the console. Ensure you are connected to the right database for writing by typing: $>Influencer13.gateway.client.transport.connections.connections.first.host. Make sure this is your local (docker) Elasticsearch and NOT your production database.

• Type $>exitto exit the console.

• Type $>rake es:dev_import:project[XXX]where XXX is the project id (1244).

Other Notes

• Check out oma-models/db/elasticsearch/migrations/es_dev_import.rake for other commands. You can import single tables as well.

• If you want a clean slate, you need to delete your indexes first, and the loader will only import new data.