URL Tracker v 0.3.0

Change monitoring app that checks the content of web pages in different periods (hourly, daily, weekly, monthly) and detects if they were modified since the last check. It can be used to monitor S3, Azure, JS files, ...etc.

Motivation 🔋

• Blog Post: https://ahussam.me/careem-aws-s3-bucket-takeover/

How to use ▶️

• If you want to track changes on a web page click add new link button on the right side (You must be logged in):

  • Fill description textbox with any text that describes the target e.g: Uber internal system login page.
  • Write the target link in Link of page textbox e.g: http://s3.amazonaws.com/careemcrm/.
  • Write keywords that will be tracked on the page if they appear you will get noticed e.g: 404, new feature, not found Note: They MUST NOT be there in the first request.
  • If you want to track an authenticated page add your cookies in the cookie text box.
  • Select the peroid that you want to check the page at.
  • In tolerance set the accepted differences between the previous fetch and the next one by characters e.g: 100. Note: if you want to ingore minors changes like CSRF tokens, cache tokens, cookie tracking set this parameter. In case you leave it blank the server will ship 2 requests and comapres the differences between them and sets the differences (acceptedChange) in DB.
  • Then click add link button. You should see the new target in the URLs list.

• If you want to get notifications via email:

  • Create a gmail account.
  • Go to User Control > Settings then set the gmail account that will send the notificaiton (DON'T USE YOUR PRIVATE EMAIL).
  • Write your private email in the receiver textbox.
  • Click save changes button.

Features ☑️

• Login system.
• 4 periods checks.
• Email reporting.
• Search in items.
• Dynamic pages fetching.
• Authenticated fetching.
• Diff checker
• Bot reporting (Telegram, Slack, ...etc) SOON!
• Suggest a feature!

Code Organisation 📂

+---api // Controllers, models, helpers 
+---assets // UI assets 
+---config // Config files: routes, security, datastores, ...etc
+---crontab // Fetch function file
+---scripts // Cloud SDK
+---tasks // Grunt tasks
+---view 
       \---layout // App layout
        \---pages // HTML in ejs templates 

Technologies 🛠

Backend ⚙️ :

Frontend 🖥

Object Diagram 🗃

REST API 🔗

Deployment 🚀

Note: make sure to install mongodb on your OS.

[email protected]:~# git clone https://github.com/ahussam/url-tracker.git
[email protected]:~# cd url-tracker
[email protected]:~/url-tracker# npm install 
[email protected]:~/url-tracker# nodejs app.js 

Then go to http://127.0.0.1:1337

For deployment on a live server install MongoDB then replace the lines 50 & 51 in config/env/production.js with:

      adapter: 'sails-mongo',
      url: 'mongodb://localhost/urlTracker', 

For more information check this out: https://sailsjs.com/documentation/concepts/deployment/hosting

Docker Option 🐋

[email protected]:~# git clone https://github.com/ahussam/url-tracker.git
[email protected]:~# cd url-tracker
[email protected]:~/url-tracker# docker-compose build
[email protected]:~/url-tracker# docker-compose up

Then go to http://127.0.0.1:8080

Default Credential 🔑

PLEASE MODIFIY THEM AFTER SIGN IN FOR THE FIRST TIME.

Screenshots 📷

License 📄

The MIT License (MIT)

Copyright (c) 2020 Abdullah Hussam

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.