Get help with morph.io and scraping

Installing local copy of morph with docker on OSX


#1

So I’m having trouble getting a local copy on OSX running. We’ll ultimately be deploying to linux, but we’d like to be able to have a local copy running for development purposes.

I’m following the github instructions from https://github.com/openaustralia/morph, but I’m not sure how docker ties into the instructions? I have docker installed, as per the OSX docker installation instructions, but there’s no call to docker run/build/compose anywhere in the instructions, so it’s not clear to me how much of it is dockerized and how much I need to do locally.

Following the instructions, I’m able to get the site running on localhost with
bundle exec dotenv foreman start
but I’m having trouble authenticating with github. I have the app registered with github with the settings in the instructions, but it throws:

execution expired```
Seeing as all of these commands are running on my machine and not in a docker container, I'm assuming the instructions aren't for a docker installation?

I also just tried running
```docker-compose up```
which seems to build everything, and morph_db and morph_elasticsearch run just fine, but morph_proxy and morph_web don’t. morph_proxy exits with the following error:
```pkg_resources.DistributionNotFound: The 'pyasn1>0.1.2' distribution was not found and is required by mitmproxy```
and morph_web isn’t able to connect to the db:
```Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (Mysql2::Error)```
I’m not sure whether that’s supposed to be the mysql db in morph_db, or if I’m supposed to have a separate mysql on localhost/in the docker VM? So it'd be nice to have some clarification about how much of the system is dockerized and how much we need to do locally. Thanks!

Here's the full log from ```docker-compose up```

    Starting morph_db_1
    Starting morph_elasticsearch_1
    Starting morph_web_1
    Starting morph_proxy_1
    Attaching to morph_db_1, morph_elasticsearch_1, morph_web_1, morph_proxy_1
    db_1             | 2017-05-17T17:31:22.097189Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    db_1             | 2017-05-17T17:31:22.108896Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 1 ...
    db_1             | 2017-05-17T17:31:22.134039Z 0 [Note] InnoDB: PUNCH HOLE support available
    db_1             | 2017-05-17T17:31:22.134139Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    db_1             | 2017-05-17T17:31:22.134195Z 0 [Note] InnoDB: Uses event mutexes
    db_1             | 2017-05-17T17:31:22.134212Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
    db_1             | 2017-05-17T17:31:22.134222Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
    db_1             | 2017-05-17T17:31:22.134232Z 0 [Note] InnoDB: Using Linux native AIO
    db_1             | 2017-05-17T17:31:22.147147Z 0 [Note] InnoDB: Number of pools: 1
    db_1             | 2017-05-17T17:31:22.160260Z 0 [Note] InnoDB: Using CPU crc32 instructions
    db_1             | 2017-05-17T17:31:22.164713Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
    db_1             | 2017-05-17T17:31:22.235981Z 0 [Note] InnoDB: Completed initialization of buffer pool
    db_1             | 2017-05-17T17:31:22.265523Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
    db_1             | 2017-05-17T17:31:22.309855Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
    db_1             | 2017-05-17T17:31:22.399676Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
    db_1             | 2017-05-17T17:31:22.399831Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
    db_1             | 2017-05-17T17:31:22.520380Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
    db_1             | 2017-05-17T17:31:22.528875Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
    db_1             | 2017-05-17T17:31:22.529330Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
    db_1             | 2017-05-17T17:31:22.555903Z 0 [Note] InnoDB: Waiting for purge to start
    db_1             | 2017-05-17T17:31:22.606878Z 0 [Note] InnoDB: 5.7.18 started; log sequence number 12139872
    db_1             | 2017-05-17T17:31:22.612516Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
    db_1             | 2017-05-17T17:31:22.613626Z 0 [Note] Plugin 'FEDERATED' is disabled.
    db_1             | 2017-05-17T17:31:22.750675Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
    db_1             | 2017-05-17T17:31:22.754121Z 0 [Warning] CA certificate ca.pem is self signed.
    db_1             | 2017-05-17T17:31:22.756126Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170517 17:31:22
    db_1             | 2017-05-17T17:31:22.761595Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
    proxy_1          | Traceback (most recent call last):
    db_1             | 2017-05-17T17:31:22.763349Z 0 [Note] IPv6 is available.
    db_1             | 2017-05-17T17:31:22.764069Z 0 [Note]   - '::' resolves to '::';
    db_1             | 2017-05-17T17:31:22.764117Z 0 [Note] Server socket created on IP: '::'.
    db_1             | 2017-05-17T17:31:22.781439Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
    db_1             | 2017-05-17T17:31:22.781534Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
    db_1             | 2017-05-17T17:31:22.782603Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
    proxy_1          |   File "/usr/local/bin/mitmdump", line 6, in <module>
    proxy_1          |     from pkg_resources import load_entry_point
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3036, in <module>
    proxy_1          |     @_call_aside
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3020, in _call_aside
    proxy_1          |     f(*args, **kwargs)
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3049, in _initialize_master_working_set
    proxy_1          |     working_set = WorkingSet._build_master()
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 654, in _build_master
    proxy_1          |     ws.require(__requires__)
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 968, in require
    proxy_1          |     needed = self.resolve(parse_requirements(requirements))
    proxy_1          |   File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 854, in resolve
    db_1             | 2017-05-17T17:31:22.784456Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
    proxy_1          |     raise DistributionNotFound(req, requirers)
    proxy_1          | pkg_resources.DistributionNotFound: The 'pyasn1>0.1.2' distribution was not found and is required by mitmproxy
    db_1             | 2017-05-17T17:31:22.837686Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
    db_1             | 2017-05-17T17:31:22.997778Z 0 [Note] Event Scheduler: Loaded 0 events
    morph_proxy_1 exited with code 1
    db_1             | 2017-05-17T17:31:22.999152Z 0 [Note] mysqld: ready for connections.
    db_1             | Version: '5.7.18'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
    db_1             | 2017-05-17T17:31:22.999272Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check.
    db_1             | 2017-05-17T17:31:22.999291Z 0 [Note] Beginning of list of non-natively partitioned tables
    elasticsearch_1  | [2017-05-17 17:31:24,501][INFO ][node                     ] [Maelstrom] version[1.6.0], pid[1], build[cdd3ac4/2015-06-09T13:36:34Z]
    elasticsearch_1  | [2017-05-17 17:31:24,502][INFO ][node                     ] [Maelstrom] initializing ...
    elasticsearch_1  | [2017-05-17 17:31:24,521][INFO ][plugins                  ] [Maelstrom] loaded [], sites []
    elasticsearch_1  | [2017-05-17 17:31:24,682][INFO ][env                      ] [Maelstrom] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [179.4gb], net total_space [192.6gb], types [ext4]
    elasticsearch_1  | [2017-05-17 17:31:32,508][INFO ][node                     ] [Maelstrom] initialized
    elasticsearch_1  | [2017-05-17 17:31:32,508][INFO ][node                     ] [Maelstrom] starting ...
    elasticsearch_1  | [2017-05-17 17:31:32,891][INFO ][transport                ] [Maelstrom] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.17.0.3:9300]}
    elasticsearch_1  | [2017-05-17 17:31:32,980][INFO ][discovery                ] [Maelstrom] elasticsearch/09R7SUgnS4CSHCD9DxgTpQ
    elasticsearch_1  | [2017-05-17 17:31:36,853][INFO ][cluster.service          ] [Maelstrom] new_master [Maelstrom][09R7SUgnS4CSHCD9DxgTpQ][49d5d361c4b2][inet[/172.17.0.3:9300]], reason: zen-disco-join (elected_as_master)
    elasticsearch_1  | [2017-05-17 17:31:36,930][INFO ][http                     ] [Maelstrom] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.0.3:9200]}
    elasticsearch_1  | [2017-05-17 17:31:36,950][INFO ][node                     ] [Maelstrom] started
    web_1            | Array values in the parameter to `Gem.paths=` are deprecated.
    web_1            | Please use a String or nil.
    web_1            | An Array ({"GEM_PATH"=>["/usr/local/bundle"]}) was passed in from bin/rails:3:in `load'
    web_1            | => Booting Puma
    web_1            | => Rails 4.2.7.1 application starting in development on http://0.0.0.0:3000
    web_1            | => Run `rails server -h` for more startup options
    web_1            | => Ctrl-C to shutdown server
    web_1            | Exiting
    web_1            | /usr/local/bundle/gems/mysql2-0.4.5/lib/mysql2/client.rb:89:in `connect': Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) (Mysql2::Error)
    web_1            | 	from /usr/local/bundle/gems/mysql2-0.4.5/lib/mysql2/client.rb:89:in `initialize'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `new'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:87:in `connection'
    web_1            | 	from /usr/local/bundle/gems/activerecord-4.2.7.1/lib/active_record/model_schema.rb:230:in `table_exists?'
    web_1            | 	from /morph/config/initializers/sidekiq.rb:3:in `<top (required)>'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:268:in `load'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:268:in `block in load'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:268:in `load'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/engine.rb:652:in `block in load_config_initializer'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:166:in `instrument'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/engine.rb:651:in `load_config_initializer'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/engine.rb:615:in `each'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/engine.rb:615:in `block in <class:Engine>'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:30:in `instance_exec'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:30:in `run'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:55:in `block in run_initializers'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:44:in `each'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:44:in `tsort_each_child'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:415:in `call'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:415:in `each_strongly_connected_component_from'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:347:in `each'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:347:in `call'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
    web_1            | 	from /usr/local/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/initializable.rb:54:in `run_initializers'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/application.rb:352:in `initialize!'
    web_1            | 	from /morph/config/environment.rb:5:in `<top (required)>'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
    web_1            | 	from /usr/local/bundle/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
    web_1            | 	from /morph/config.ru:3:in `block in <main>'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/builder.rb:55:in `instance_eval'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/builder.rb:55:in `initialize'
    web_1            | 	from /morph/config.ru:in `new'
    web_1            | 	from /morph/config.ru:in `<main>'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/builder.rb:49:in `eval'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/builder.rb:49:in `new_from_string'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/builder.rb:40:in `parse_file'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/server.rb:299:in `build_app_and_options_from_config'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/server.rb:208:in `app'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/server.rb:61:in `app'
    web_1            | 	from /usr/local/bundle/gems/rack-1.6.5/lib/rack/server.rb:336:in `wrapped_app'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/server.rb:139:in `log_to_stdout'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/server.rb:78:in `start'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:80:in `block in server'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:75:in `tap'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:75:in `server'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    web_1            | 	from /usr/local/bundle/gems/railties-4.2.7.1/lib/rails/commands.rb:17:in `<top (required)>'
    web_1            | 	from /morph/bin/rails:8:in `require'
    web_1            | 	from /morph/bin/rails:8:in `<top (required)>'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/client/rails.rb:28:in `load'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/client/rails.rb:28:in `call'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/client/command.rb:7:in `call'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/client.rb:30:in `run'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/bin/spring:49:in `<top (required)>'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/binstub.rb:31:in `load'
    web_1            | 	from /usr/local/bundle/gems/spring-2.0.0/lib/spring/binstub.rb:31:in `<top (required)>'
    web_1            | 	from /morph/bin/spring:13:in `require'
    web_1            | 	from /morph/bin/spring:13:in `<top (required)>'
    web_1            | 	from bin/rails:3:in `load'
    web_1            | 	from bin/rails:3:in `<main>'
    morph_web_1 exited with code 1