All Projects → media-centre → makenews

media-centre / makenews

Licence: AGPL-3.0 license
MakeNews is for journalists and newsrooms. It helps you track news from web and social media in real-time.

Programming Languages

javascript
184084 projects - #8 most used programming language
SCSS
7915 projects
Jinja
831 projects
shell
77523 projects

Projects that are alternatives of or similar to makenews

CoWin-Vaccine-Notifier
Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.
Stars: ✭ 102 (+121.74%)
Mutual labels:  webscraper
hipposcraper
A Linux terminal tool for parsing and scraping Holberton project pages to automate repetitive tasks.
Stars: ✭ 32 (-30.43%)
Mutual labels:  webscraper
JARR
JARR is a web news aggregator.
Stars: ✭ 99 (+115.22%)
Mutual labels:  news-aggregator
stream-net
NET Client - Build Activity Feeds & Streams with GetStream.io
Stars: ✭ 28 (-39.13%)
Mutual labels:  news-feed
Wuxiaworld-2-eBook
This Python script will download chapters from novels availaible on wuxiaworld.com saves then into the .epub format
Stars: ✭ 90 (+95.65%)
Mutual labels:  webscraper
imgur downloader
Python script/class to download an entire Imgur album in one go into a folder of your choice.
Stars: ✭ 35 (-23.91%)
Mutual labels:  webscraper
vue-rss-feed
Embed RSS Feeds in your Vue web app
Stars: ✭ 37 (-19.57%)
Mutual labels:  news-feed
Ruby Capstone
A simple web scraper built with Ruby and the Nokogiri gem. It crawls a certain website and gets the prices and other data of cryptocurrencies. Rspec was used for testing.
Stars: ✭ 14 (-69.57%)
Mutual labels:  webscraper
fBrowser
Helpful Selenium functions to make web-scraping easier and faster
Stars: ✭ 16 (-65.22%)
Mutual labels:  webscraper
subsocial-node
NOTE: Development continues in https://github.com/dappforce/subsocial-parachain repo. Subsocial full node with Substrate/Polkadot pallets for decentralized communities: blogs, posts, comments, likes, reputation.
Stars: ✭ 73 (+58.7%)
Mutual labels:  news-feed
trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
Stars: ✭ 711 (+1445.65%)
Mutual labels:  news-aggregator
DailyFeed
iOS client for newsapi.org
Stars: ✭ 128 (+178.26%)
Mutual labels:  news-aggregator
HostPanic
Find host header injections and perform Host Header attacks with other kind of bugs like web cache poissoning
Stars: ✭ 23 (-50%)
Mutual labels:  webscraper
stream-python
Python Client - Build Activity Feeds & Streams with GetStream.io
Stars: ✭ 134 (+191.3%)
Mutual labels:  news-feed
archnews
Python utility for displaying Arch news feed in console.
Stars: ✭ 19 (-58.7%)
Mutual labels:  news-feed
super-anime-downloader
A program which takes an Anime name or URL and downloads the specified range of episodes.
Stars: ✭ 26 (-43.48%)
Mutual labels:  webscraper
ShopifyScripts
Various Shopify scripts to generate XML product feeds, remarketing tracking for Google Ads, and others.
Stars: ✭ 98 (+113.04%)
Mutual labels:  facebook-feed
PressCenters.com
News aggregator for the press releases of the Bulgarian government sites written in ASP.NET Core
Stars: ✭ 91 (+97.83%)
Mutual labels:  news-aggregator
scraperx
Library for scraping websites or apis at any scale
Stars: ✭ 49 (+6.52%)
Mutual labels:  webscraper
google-news-scraper
Google News Scraper for languages like Japanese, Chinese... [VPN Support]
Stars: ✭ 88 (+91.3%)
Mutual labels:  news-aggregator

Makenews

Makenews

MakeNews is for journalists and newsrooms. It helps you track news from web and social media in real-time.

Build Status Coverage Status

Pre Requisites:

  1. Install node(version >= 8.0) npm
  2. Install couchdb 2.0
  3. Create an admin user in couchdb with whom you can create, delete and change databases manually on the localhost:5984/_utils/#
    If your admin user in couchdb is not with the name 'admin' and password 'admin', change this data in the file " instal.config " from this very project.
  4. Install couchdb lucene 2.0
  5. Install gulp npm install gulp -g

Test Requisites:

  1. write npm --version in the console and it shall return the number of the certain installed npm version
  2. write 'localhost:5984' on your browser and there you should see the couchdb version and some additional information about couchdb.
  3. check whether you have configured correctly the couchdb-lucene. For instance, you can try querying a database(if you created database in couchdb and you have installed curl) like this:
	curl http://"your_database_ip":5984/database_name/_fti/design_doc/view_name?q=Query
  1. write gulp --version in the console and it shall return the number of the certain installed gulp version(local and cli)

1. Installation:

1.Clone the makenews application repository from git
2. All the configuration related data must be mentioned in the install.config file

  • Facebook appId and app secret key must be given in the file for being able to configure data from FB sources
  • Twitter consumer key and consumer secret must be mentioned under twitter configuration for being able to access twitter data from makenews
  • Refer install.config file for instructions

Comments: Facebook and Twitter appIds are not necessary to start the application, they are necessary for the actual connection between the app and facebook or twitter.
3. Start couchdb
4. Once the configurations are set in the install.config file, run the oneClickInstaller.sh file
5. Running this file installs the makenews application with the configurations set in the install.config
6. Towards the end of the execution, application user creation dialogue would be prompted on the console, enter the username and password for the intended user

2. Start the server:

Once the installation is successful, use the following command from the root folder, to start the server

node dist/server

The application would start running on the configured ip address and port, for the default configuration, http://127.0.0.1:5000

  • Additional Information: after running the server check both consoles(the console of the browser and the cmd prompt) for some errors.
  • In some cases, it is necessary to go to 127.0.0.1:5000 instead of localhost:5000, because of CORS, if you want to log in - it can give some errors in the console of the browser
  • In some other cases, after trying to login the process in the cmd prompt crashes because of the permissions. That's why you should sometimes start the server with sudo(just 'sudo node dist/server')

3. Creating Application users:

run createuser.sh file from dist folder

4. Application configurations:

After changing any configuration,  please run gulp build and restart the server

4.a Server configuration:

The server configuration file path is server/config/application.json

{

 "default": { //default configuration
   "serverIpAddress": "localhost", //ip address of the application server
   "serverPort": 5000, //Port on which the server runs
   "couchDbUrl": "http://localhost:5984", // couch db url
   "searchEngineUrl": "http://localhost:5986/_fti/local", // lucene url
   "userDbPrefix": "db_", //application user dbname prifix
   "adminDetails": { //application admin details
     "username": "admin",
     "password": "admin",
     "db": "common" //Common database for all users for storing web urls
   },
   "facebook": {
     "url": "https://graph.facebook.com/v2.8",
     "appSecretKey": "fb_appsecret", //facebook app appSecretKey
     "appId": "fb_appId", //App id of facebook app
     "timeOut": 2000, //timeout for request
     "limit": 500 //limit for data from fb for a source
   },
   "twitter": {
     "url": "https://api.twitter.com/1.1",
     "authenticateUrl": "https://api.twitter.com/oauth/authenticate",
     "consumerKey": "twitter_consumerKey", // twitter app consumer key
     "consumerSecret": "twitter_consumerSecret" // twitter app consumer secret
   }
 },
 "qa": { //environment specific, if we need diff config for qa environment you can add here

 }
}

4.b Client configuration:

The client config file is makenews/client/config/config.js

if(!window.mediaCenter) {
   window.mediaCenter = {};
}
window.mediaCenter.serverUrl = "";    //application server url
window.mediaCenter.facebookAppId = ""; //facebook appid
window.mediaCenter.autoRefreshSurfFeedsInterval = 300000; //time interval for refreshing feeds
window.mediaCenter.dbSessionInterval = 600000; //time interval for a login session
window.mediaCenter.storyAutoSaveTimeInterval = 300000; // time after which a story is auto saved

5. Troubleshoot:

  • If twitter authentication page shows empty - While creating the twitter app, mention the makenews app url under the callback url, to redirect to the app after authentication
  • If login fails and console shows CORS error - Ensure that the couchdb CORS is enabled
  • If login fails - Make sure Couchdb is running before trying to log into the application
  • In case of error with facebook authentication - Facebook app( whose id is mentioned in the configuration file) must be in active state, for the makenews app to be able to fetch data from Facebook
  • If the default Web URL suggestion are not seen - check lucene if setup is properly done
  • For Windows users, changing the server url to ‘localhost’ will not work, hence keep the server url to the default 127.0.0.1
  • Makenews is configured to work with facebook version of v2.10. There could be issues cropping up if a new version is released from Facebook, which might need fixes in the code

6. Developer options:

6.a Build and Tests:

The procedure to build the application, and get your changes working, are

  • gulp build
  • npm test
  • npm run lint //run static analyser
  • gulp clean //cleaning dist folder

check the gulpfile.js for more gulp tasks

6.b Migration:

If your change need data migration follow below steps:

  1. New migration files can be written in the makenews/server/src/migration folder
  2. Any migration, that has to change the common db, must be specified in the admin folder of migration
  3. A migration for the user db, must be specified inside the db folder of migration
  4. The migration data file should be named following the template of timestamp_filename
  5. The migration file must contain an up method, for the db upgradation
  6. A new condition must be added in the getObject method of the Migration.js file, for every new migration file created
  7. The following command should be used for running the new migration
    node dist/server/src/migration.js [--admin_user_name='username' --admin_password='password']
    

7. Tips:

  • If you have some problems to install couchdb or to connect it to couchdb-lucene, just to know that if you have or use mac os - the installation is very faster and easier. The mac users install only the application couchdb and there is almost no manual configurations between couchdb and couchdb-lucene(just 2-3 line in the console)
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].