All Projects → vuestorefront → Mage2vuestorefront

vuestorefront / Mage2vuestorefront

Licence: mit
Magento to Vue-storefront datapump - synchronizes Products, Categories and Product-to-category links between your Magento2 API and NoSQL database of vue-storefront

Programming Languages

javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to Mage2vuestorefront

Nodbi
Document DBI connector for R
Stars: ✭ 56 (-69.4%)
Mutual labels:  redis, mongodb, elasticsearch, nosql
Vue Storefront Api
Vue.js storefront for Magento2 (and not only) - data backend
Stars: ✭ 328 (+79.23%)
Mutual labels:  magento, redis, elasticsearch, backend
Phalcon Vm
Vagrant configuration for PHP7, Phalcon 3.x and Zephir development.
Stars: ✭ 43 (-76.5%)
Mutual labels:  redis, mongodb, elasticsearch
Mall Learning
mall学习教程,架构、业务、技术要点全方位解析。mall项目(40k+star)是一套电商系统,使用现阶段主流技术实现。涵盖了SpringBoot 2.3.0、MyBatis 3.4.6、Elasticsearch 7.6.2、RabbitMQ 3.7.15、Redis 5.0、MongoDB 4.2.5、Mysql5.7等技术,采用Docker容器化部署。
Stars: ✭ 10,236 (+5493.44%)
Mutual labels:  redis, mongodb, elasticsearch
Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (-55.19%)
Mutual labels:  redis, mongodb, elasticsearch
Pifpaf
Python fixtures and daemon managing tools for functional testing
Stars: ✭ 161 (-12.02%)
Mutual labels:  redis, mongodb, elasticsearch
Kuzzle
Open-source Back-end, self-hostable & ready to use - Real-time, storage, advanced search - Web, Apps, Mobile, IoT -
Stars: ✭ 991 (+441.53%)
Mutual labels:  redis, elasticsearch, backend
Docker Magento Mutagen
Mage2click Docker-based development environment for Magento with mutagen.io sync for files on macOS
Stars: ✭ 64 (-65.03%)
Mutual labels:  magento2, redis, elasticsearch
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-43.17%)
Mutual labels:  redis, mongodb, elasticsearch
Mall
mall项目是一套电商系统,包括前台商城系统及后台管理系统,基于SpringBoot+MyBatis实现,采用Docker容器化部署。 前台商城系统包含首页门户、商品推荐、商品搜索、商品展示、购物车、订单流程、会员中心、客户服务、帮助中心等模块。 后台管理系统包含商品管理、订单管理、会员管理、促销管理、运营管理、内容管理、统计报表、财务管理、权限管理、设置等模块。
Stars: ✭ 54,797 (+29843.72%)
Mutual labels:  redis, mongodb, elasticsearch
Haproxy Configs
80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.
Stars: ✭ 106 (-42.08%)
Mutual labels:  redis, elasticsearch, nosql
Szt Bigdata
深圳地铁大数据客流分析系统🚇🚄🌟
Stars: ✭ 826 (+351.37%)
Mutual labels:  redis, mongodb, elasticsearch
Springbootexamples
Spring Boot 学习教程
Stars: ✭ 794 (+333.88%)
Mutual labels:  redis, mongodb, elasticsearch
Spring Boot Leaning
Spring Boot 2.X 最全课程代码
Stars: ✭ 2,008 (+997.27%)
Mutual labels:  redis, mongodb, elasticsearch
Goodskill
🐂基于springcloud +dubbo构建的模拟秒杀项目,模块化设计,集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件
Stars: ✭ 786 (+329.51%)
Mutual labels:  redis, mongodb, elasticsearch
Mall Swarm
mall-swarm是一套微服务商城系统,采用了 Spring Cloud Hoxton & Alibaba、Spring Boot 2.3、Oauth2、MyBatis、Docker、Elasticsearch、Kubernetes等核心技术,同时提供了基于Vue的管理后台方便快速搭建系统。mall-swarm在电商业务的基础集成了注册中心、配置中心、监控中心、网关等系统功能。文档齐全,附带全套Spring Cloud教程。
Stars: ✭ 7,874 (+4202.73%)
Mutual labels:  redis, mongodb, elasticsearch
Vagrant Elastic Stack
Giving the Elastic Stack a try in Vagrant
Stars: ✭ 131 (-28.42%)
Mutual labels:  redis, mongodb, elasticsearch
Newsblur
NewsBlur is a personal news reader that brings people together to talk about the world. A new sound of an old instrument.
Stars: ✭ 5,862 (+3103.28%)
Mutual labels:  redis, mongodb, elasticsearch
Stream Reactor
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Stars: ✭ 753 (+311.48%)
Mutual labels:  redis, mongodb, elasticsearch
Springboot Templates
springboot和dubbo、netty的集成,redis mongodb的nosql模板, kafka rocketmq rabbit的MQ模板, solr solrcloud elasticsearch查询引擎
Stars: ✭ 100 (-45.36%)
Mutual labels:  redis, mongodb, elasticsearch

mage2vuestorefront

For those who would love to work with Magento on backend but use NoSQL power on the frontend. Two way / real time data synchronizer.

It's part of vue-storefront project - first Progressive Web App for eCommerce with Magento2 support. Some details about the rationale and our goals here

It synchronizes all the products, attributes, taxrules, categories and links between products and categories.

This is multi-process data synchronizer between Magento to Vue Storefront ElasticSearch database.

At this point synchronization works with following entities:

  • Products
  • Categories
  • Taxrules
  • Attributes
  • Product-to-categories
  • Reviews (require custom module Divante/ReviewApi to work)
  • Cms Blocks & Pages (require custom module SnowdogApps/magento2-cms-api)

Categories and Product-to-categories links are additionaly stored in Redis cache for rapid-requests (for example from your WebAPI). Our other project vue-storefront-api exposes this database to be used in PWA/JS webapps.

Datasync uses oauth + magento2 rest API to get the data. KUE is used for job queueing and multi-process/multi-tenant processing is enabled by default ElasticSearch is used for NoSQL database Redis is used for KUE queue backend

By default all services are used without authorization and on default ports (check out config.js or ENV variables for change of this behavior).

Tutorial on installation / integration manual for Vue Storefront connectivity

How to perform full / initial import for Vue Storefront

To get started with VS we must start with some very basics about the architecture; the project is backed by three separate Node.js applications

Vue Storefront Architecture

vue-storefront (Github) — is the main project where you can also find most of the documentation, issues mapped to further releases and other resources to start with — Vue.js on webpack.

vue-storefront-api (Github) — is the API layer which provides the data to vue-storefront app — Node.js, Express; This project consist of docker instances for Redis and ElasticSearch required by mage2vuestorefront and pimcore2vuestorefront

mage2vuestorefront — THIS project -data bridges which are in charge of moving data back from Magento2 to Vue Storefront data store.

You must install vue-storefront-api locally. You may install it using the Vue Storefront installer - see details. Or manually by executing the sequence of commands:

git clone https://github.com/DivanteLtd/vue-storefront-api
cd vue-storefront-api
npm install
npm run migrate
docker-compose up -d
npm run dev

The key command is docker-compose up -d which runs the ElasticSearch and Redis instances - both required by mage2vuestorefront

Elastic 7 Support

By default, Vue Storefront API docker files and config are based on Elastic 5.6. We plan to change the default Elastic version to 7 with the 1.11 stable release. As for now, the Elastic 7 support is marked as experimental.

In order to index data to Elastic 7 please make sure you set the proper apiVersion in the config.js:

  elasticsearch: {
    apiVersion: process.env.ELASTICSEARCH_API_VERSION || '7.1'
  },

or just use the env variable:

export ELASTICSEARCH_API_VERSION=7.1

Starting from Elasitc 6 and 7 we can have just single document type per single index. Vue Storefront used to have product, category ... types defined in the vue_storefront_catalog.

From now on, we're using the separate indexes per each entity type. The convention is: ${indexName}_${entityType}. If your' logical index name is vue_storefront_catalog then it will be mapped to the physical indexes of: vue_storefront_catalog_product, vue_storefront_catalog_category ...

Initial Vue Storefront import

Now, You're ready to run the importer. Please check the config file. You may setup the Magento access data and URLs by config values or ENV variables.

We'll use in the following example - the ENV variables. The simplest command sequence to perform full reindex is:

export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9

echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog

node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1
node --harmony cli.js reviews

After installing the 3rd party Magneto module (SnowdogApps/magento2-cms-api) there are two additional imports available:

node --harmony cli.js blocks
node --harmony cli.js pages

Please note:

  • --removeNonExistent option means - all records that were found in the index but currently don't exist in the API feed - will be removed. Please use this option ONLY for the full reindex!
  • INDEX_NAME by default is set to the vue_storefront_catalog but You may set it to any other elastic search index name.
  • The categories importer option --generateUniqueUrlKeys is by default set to true. This is due the fact that in Magento2, the category.url_key field is not mandatory unique and from v. 1.7 Vue Storefront uses the category.url_key to display the category details without any client's side modification.
  • PRODUCTS_EXCLUDE_DISABLED by default is set to false. To only import enabled products set this to true.

Cache invalidation: Recent version of Vue Storefront do support output caching. Output cache is being tagged with the product and categories id (products and categories used on specific page). Mage2vuestorefront can invalidate cache of product and category pages if You set the following ENV variables:

export VS_INVALIDATE_CACHE_URL=http://localhost:3000/invalidate?key=aeSu7aip&tag=
export VS_INVALIDATE_CACHE=1
  • VS_INVALIDATE_CACHE_URL is a cache to the Vue Storefront instance - used as a webhook to clear the output cache.

Please note: After data import - especially when You're not sure about the product attributes data types - please reindex ElasticSearch to establish the correct / current database schema. You may do this using Database tool in the vue-storefront-api folder:

cd vue-storefront-api
npm run db rebuild -- --indexName=vue_storefront_catalog

If You like to create a new, empty index please run:

cd vue-storefront-api
npm run db new -- --indexName=vue_storefront_catalog

Checking indexed data

If you want to see how many products were stored into Elastic data store, you can use Kibana to do so. Kibana is part of vue-storefront-api. Once you start docker containers of vue-storefront-api you can access it on http://localhost:5601/.

To see count of indexed products go to DEV tools and run following query:

GET vue_storefront_catalog/product/_count

See https://www.elastic.co/guide/en/kibana/current/console-kibana.html to find out more.

Delta indexer

After initial setup and full-reindex You may want to add indexer to the crontab to index only modified product records. This is fairly easy - You just need to add the following command to crontab:

node --harmony cli.js productsdelta --partitions=1

This command will execute full reindex at first call - and then will be storing the last index date in the .lastIndex.json and downloading only these products which have updated_at > last index date.

If you have a multistore setup and would like to use the delta indexer for each storeview you can not use the delta timestamp from .lastIndex.json for all stores; instead you will need to set the INDEX_META_PATH to a unique value for each store you are indexing. For instance:

export INDEX_META_PATH=.lastIndex-UK.json && node --harmony cli.js productsdelta --partitions=1

Please note: Magento2 has a bug with altering updated_at field. Please install a fix for that before using this method:

composer require codepeak/magento2-productfix
php bin/magento cache:flush

Parent products updates

Please note if there is a simple product update request coming from Delta Indexer or On Demand indexer mage2vuestorefront will - by default - check and update the configurable/parent product as well. The parent product update is scheduled in the productsworker mode - using KUE queue. Please make sure You're runinng at least one worker instance to process these on-demand request:

cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest

node --harmony cli.js productsworker

This second process will take care of indexing the parent products updates whetever any of it's configurable_children simple products has been changed.

On-demand indexer (experimental!)

Mage2nosql supports an on-demand indexer - where Magento calls a special webhook to update modified products. In the current version the webhook notifies mage2vuestorefront about changed product SKUs and mage2vuestorefront pulls the modified products data via Magento2 APIs.

First. You should install Magento2 module called VsBridge Second. Deploy mage2vuestorefront on the server.

Then You may want to start a webapi process:

cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest

node --harmony webapi.js

The API will be listening on port 6060. Typically non-standard ports like this one are not exposed on the firewall. Please consider setting up simple nginx proxy for this service.

Anyway - this API must be publicly available via Internet OR You must have the mage2vuestorefront installed on the same machine like Magento2.

Go to Your Magento2 admin panel, then to Stores -> Configuration -> VsBridge and set-up "Edit product" url to: http://localhost:6060/magento/products/update. Please note: Product delete endpoint hasn't been implemented yet and it's good chance for Your PR.

After having the webapi up and runing and this endpoint set, any Product save action will call POST http://localhost:6060/magento/products/update with the body set to {"sku": ["modified-sku-list"]}.

Webapi will add the products to the queue.

Please run the queue worker to process all the queued updates (You may run multiple queue workers even distributed across many machines):

cd mage2vuestorefront/src
export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9
export PORT=6060
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest

node --harmony cli.js productsworker

Please note: We're using kue based on Redis queue which may be configured via src/config.js - kue + redis section.

Please note: Redis now supports auth. In order to use Redis with auth simply pass the password to the REDIS_AUTH env variable.

Multistore setup

Multiwebsite support starts with the ElasticSearch indexing. Basically - each store has it's own ElasticSearch index and should be populated separately using mage2vuestorefront tool.

The simplest script to index multi site:

export TIME_TO_EXIT=2000
export MAGENTO_CONSUMER_KEY=byv3730rhoulpopcq64don8ukb8lf2gq
export MAGENTO_CONSUMER_SECRET=u9q4fcobv7vfx9td80oupa6uhexc27rb
export MAGENTO_ACCESS_TOKEN=040xx3qy7s0j28o3q0exrfop579cy20m
export MAGENTO_ACCESS_TOKEN_SECRET=7qunl3p505rubmr7u1ijt7odyialnih9

echo 'German store - de'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/de
export INDEX_NAME=vue_storefront_catalog_de

node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1

echo 'Italian store - it'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest/it  
export INDEX_NAME=vue_storefront_catalog_it

node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1

echo 'Default store - in our case United States / en'
export MAGENTO_URL=http://demo-magento2.vuestorefront.io/rest
export INDEX_NAME=vue_storefront_catalog

node --harmony cli.js categories --removeNonExistent=true
node --harmony cli.js productcategories --partitions=1
node --harmony cli.js attributes --removeNonExistent=true
node --harmony cli.js taxrule --removeNonExistent=true
node --harmony cli.js products --removeNonExistent=true --partitions=1

As You may see it's just a it or de store code which is added to the base Magento2 REST API urls that makes the difference and then the INDEX_NAME set to the dedicated index name.

In the result You should get:

  • vue_storefront_catalog_it - populated with the "it" store data
  • vue_storefront_catalog_de - populated with the "it" store data
  • vue_storefront_catalog - populated with the "default" store data

Then, to use these indexes in the Vue Storefront You should index the database schema using the vue-storefront-api db tool:

npm run db rebuild -- --indexName=vue_storefront_catalog_it
npm run db rebuild -- --indexName=vue_storefront_catalog_de
npm run db rebuild -- --indexName=vue_storefront_catalog

More on how to setup Vue Storefront in the Multistore mode.

Indexing configurable products attributes for filters

If You like to have Category filter working with configurable products - You need to expand the product.configurable_children.attrName to product.attrName_options array. This is automatically done by mage2vuestorefront for all attributes set as product.configurable_options (by default: color, size). If You like to add additional fields like manufacturer to the filters You need to expand product.manufacturer_options field. The easiest way to do so is to set config.product.expandConfigurableFilters to ['manufacturer'] and re-run the mage2vuestorefront indexer.

FAQ

Here You can find some frequently asked questions answered:

I've been playing with VSF for quite a while now and now I see that my catalog rule (-20% on all products) is not applied in one shop.

Please make sure that You've got the config.synchronizeCatalogSpecialPrices (env: PRODUCTS_SPECIAL_PRICES) and config.renderCatalogRegularPrices (env: PRODUCTS_RENDER_PRICES) set to true (default is false). Otherwise only the catalog prices will be synced (without dynamic pricing rules applied). You can also use the Vue Storefront dynamic-pricing option for the same purpose.

Advanced usage

Start Elasticsearch and Redis:

  • docker-compose up

Install:

  • npm install
  • cd src

Config - see: config.js or use following ENV variables:

  • MAGENTO_URL
  • MAGENTO_CONSUMER_KEY
  • MAGENTO_CONSUMER_SECRET
  • MAGENTO_ACCESS_TOKEN
  • MAGENTO_ACCESS_TOKEN_SECRET
  • DATABASE_URL (default: 'elasticsearch://localhost:9200/vue_storefront_catalog')

Run:

  • cd src/ and then:
  • node --harmony cli.js fullreindex - synchronizes all the categories, products and links between products and categories

Other commands supported:

  • node --harmony cli.js products --partitions=10
  • node --harmony cli.js products --partitions=10 --initQueue=false - run the products sync worker (product sync jobs should be populated eslewhere - it's used to run multi-tenant environment of workers)
  • node --harmony cli.js products --partitions=10 --delta=true - check products changed since last run; compared by updated_at field
  • node --harmony cli.js productcategories - to synchronize the links between products and categories it should be run before products synchronization because it populates Redis cache assigments for product-to-category link
  • node --harmony cli.js categories
  • node --harmony cli.js products --adapter=magento --partitions=1 --skus=24-WG082-blue,24-WG082-pink - to pull out only selected SKUs
  • node --harmony cli.js productsworker --adapter=magento --partitions=10 - run queue worker for pulling out individual products (jobs can be assigned by webapi.js microservice triggers; it can be called by webhook for example from within Magento2 plugin)
  • node --harmony webapi.js - run localhost:3000 service endpoint for adding queue tasks

WebAPI:

  • node --harmony webapi.js
  • curl localhost:8080/api/magento/products/pull/WT09-XS-Purple - to schedule data refresh for SKU=WT09-XS-Purple
  • node --harmony cli.js productsworker - to run pull request processor

Available options:

  • partitions=10 - number of concurent processes, by default number of CPUs core given
  • adapter=magento - for now only Magento is supported
  • delta - sync products changed from last run
  • command names: products / attributes / taxrule / categories / productsworker / productcategories / productsdelta
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].