OMAlab Guide
  • Introduction
  • Guiding Principles
    • Mission Statement
    • Conflict Resolution Process
  • Operating Model
    • Working Together
    • Holacracy
      • Meetings
      • Specific Roles
      • Terms and Definitions
      • Finer Points
      • Holacracy-Asana Key
    • Getting Things Done
      • Daily, Weekly, Monthly, and Annual Reviews
      • GTD-Asana Key
    • Transparency
    • Language
    • Budgeting
    • By Department
      • Engineering Operations
  • General Guidelines
  • Employment Policies
    • Equal Opportunity Employment
    • At-Will Employment
    • Code of Conduct in the Community
    • Complaint Policy
    • Drug and Alcohol Policy
    • Vacation, Holiday, and Paid Time Off (PTO) Policy
    • Supplemental Policies for Remote Employees and Contractors
    • Supplemental Policy for Bonus, Commissions, and other Performance-based Payments
    • Supplemental Policies for Hourly International Contractors or Workers
    • Supplemental Policies for Hourly International Contractors or Workers
    • Disputes and Arbitration
  • Benefits and Perks
    • Health Care
    • Vacation, Holiday and Paid Time Off (PTO) Policy
    • Holiday List
  • Hiring Documents
    • Acknowledgement of Receipt
    • Partner Proprietary Information and Inventions Agreement
  • Engineering Wiki
    • Code Snippets
      • Front End Code Snippets
    • Setup
      • 1: Overview of development using Audienti
      • 2: How to setup your dev environment on Docker
      • 2a: Setting up on our cloud your dev server
      • 3: Connect to Production using the VPN
      • 4: Import data into your development environment
    • Deployment
      • Docker based deployment of back end (manual)
    • Culture
      • How our development team works
      • Code Best Practices
    • Tips
      • Setting up a new development machine
      • Importing data to Development environment
      • GIT workflow and work tracking
      • Using Slack
      • Using Rubocop
      • Our Code Standards
      • General suggested best practices
      • Tracking your time
      • Naming Iterations
    • Migrations
      • Postgres
      • ElasticSearch
      • Redis
    • Database and System Maintenance
      • Redis Howtos
      • Elasticsearch HowTos
      • Postgres HowTos
      • Administration recipes
      • App maintenance crash course notes
    • Front End
      • 2016 Plan
      • Deploy
      • Assets
      • SearchLogic
      • How to create UI components
      • OMA Standard Tables
    • Monitoring and Alerting
      • Monitoring Systems
      • Monitoring individual controller actions
      • Get notified when a metric reaches a certain threshold
      • Instrumenting your models using Oma Stats
      • Configuring Graphite Charts
      • Tracking your results with StatsD
      • Logging Fields
      • Updating Kibana Filtering
    • Testing
      • Coverage
      • Elasticsearch mapping config synchronization
      • Testing Gotchas
      • Rspec Preloader
      • Test Best Practices
    • Models
      • Backlinks
    • Queueing and Worker System
      • Queueing and Job Overview
    • Processors
      • Rebuilding Spot Instances
      • Deploying processors
      • Running processors in development
      • Reverting to the previous build on a failed deployment
    • Processors / Opportunity Pipeline
      • Opportunity Pipeline
      • Diagram
    • Processors / Enrichment Pipeline
      • Diagram
      • Clustering
    • Processors / Backlink Pipeline
      • Diagram
      • Backlink Pipeline external APIs
      • Backlink pipeline logic
    • Processors / Automation Pipeline
      • Diagram
      • Automation Pipeline Overview
      • Agents
      • Running in development
    • Messaging and Social Accounts
      • Overview
    • API
      • Audienti API
    • Algorithms
    • Troubleshooting
      • Elasticsearch
    • Big Data Pipeline Stuff
      • Spark
    • Our Product
      • Feature synopsis of our product
    • Research
      • Backend framework comparison
      • Internet marketing Saas companies
    • Code snippets
      • Commonly Used
      • Not Used
    • Miscellaneous
      • Proxies and Bax
    • Legacy & Deprecated
      • Search criteria component
      • Classes list
      • Target Timeline
      • Twitter processor
      • Asset compilation
      • Test related information
      • Interface to EMR Hadoop jobs
      • Mongo Dex Indexes to be Built
      • Mongodb errors
      • Opportunity pipeline scoring
      • Graph Page
      • Lead scoring
      • Insights
      • Shard keys
      • Setting up OMA on local
      • Clone project to local machine
      • Getting around our servers in AWS
  • Acknowledgements
  • Documents That Receiving Your First Payment Triggers Acknowledgement and Acceptanace
Powered by GitBook
On this page
  • Docker-based development
  • How your docker-compose.override.yml works
  • Run versus CMD versus bash.
  • How image building works
  1. Engineering Wiki
  2. Setup

1: Overview of development using Audienti

In ongoing development, we use Docker for our development environment. This requires a 1 time setup using the Audienti repo (improvements are always welcome!).

The CLI that is used as part of the tool is the big thing to understand. There are a few key things it does.

  • Sets up your ENV to match what's expected by other scripts (by typing cli config)

  • Builds images and pushes them to dockerhub (cli build oma-processors)

  • Builds the rancher-compose and docker-compose files using templates to push to Rancher (cli write_rancher_stacks docker dc1)

  • Automates a rancher deploy of all microservices, or individual microservices (cli deploy_allorcli deploy opportunity-pipeline)

Docker-based development

One big thing the setup above does, is setup your environment with shortcuts that help you with development. Specifically, there are commands that are used EXTENSIVELY for development. They are called "dcdx commands".

DCDX commands are built with the following syntax:

  • dc = docker compose

  • x = your target environment (development = d, staging = s, production = p, production_local = pl, test = t)

  • r = your command (r = run, b = bash with mapped service-ports for running a web server and doing development)

You use these commands like this:

  • dcdr oma bash --login= login to the oma container at a bash prompt, run the login script for bash to give you a nicer prompt that shows your github branch etc.

  • dcplr oma bash --login= login to the oma container at a bash prompt, run the long script, in the production_local enviornment. Service ports will NOT be mapped.

  • dcplb oma= login to the oma repo container in production_local mode. Map the port 3000 to the host, so you can run the web server and see local changes, connect to the production databases.

How your docker-compose.override.yml works

When you build a container, it builds a production version of it. This will copy the code at it current state, and generate a docker image that matches it. This image is immutable (not changing). Every time you boot it, it will be the same image in the exact same state.

So, for development, we use a docker-compose.override.yml, this will essentially "map" into the production image the local folders you have on your site. It also has definitions for you postgres, elasticsearch, redis, and other databases so that you have a local copy of them in the same "network" as your app container.

If you don't have the docker-compose.override.yml, then you won't be able to connect to development databases.

Run versus CMD versus bash.

When you run a docker image with the run command (our default way to get a bash prompt), you will NOT map the service-ports, which means your ports (like port 3000 for web development) will not be exposed to the host. This is good, because you can have 2 or 3 command lines running at the same time.

If you need to get service-ports mapped, you must run it with the --service-ports flag. As indcplr --service-ports oma bash --loginto get the service-ports mapped.

When you use thedcplb omacommand, that is all this is doing. It's a run command with service ports mapped automatically.

How image building works

When you build a new image, by launching it and one doesn't exist, or by typing the commandcli build oma-processors, the command will rename your docker-compose.overrride.yml to remove the mapping before the image is built. If you build an image and try to launch it that is built with the override file in place, it will error.

PreviousSetupNext2: How to setup your dev environment on Docker

Last updated 7 years ago