All Projects → logdna → Logdna Agent

logdna / Logdna Agent

Licence: mit
LogDNA Agent streams from log files to your LogDNA account. Works with Linux, Windows, and macOS Servers

Programming Languages

javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to Logdna Agent

Easylogger
An ultra-lightweight(ROM<1.6K, RAM<0.3k), high-performance C/C++ log library. | 一款超轻量级(ROM<1.6K, RAM<0.3k)、高性能的 C/C++ 日志库
Stars: ✭ 1,968 (+1368.66%)
Mutual labels:  logging, logger
Android Remote Debugger
A library for remote logging, database debugging, shared preferences and network requests
Stars: ✭ 132 (-1.49%)
Mutual labels:  logging, logger
Keylogger
🔐 Open Source Python Keylogger Collection
Stars: ✭ 97 (-27.61%)
Mutual labels:  mac, logging
React Native Logs
Performance-aware simple logger for React-Native with namespaces, custom levels and custom transports (colored console, file writing, etc.)
Stars: ✭ 84 (-37.31%)
Mutual labels:  logging, logger
Logback Access Spring Boot Starter
Spring Boot Starter for Logback-access
Stars: ✭ 118 (-11.94%)
Mutual labels:  logging, logger
Logsip
A simple, concise, colorful logger for Go
Stars: ✭ 94 (-29.85%)
Mutual labels:  logging, logger
Illuminati
This is a Platform that collects all the data accuring in your Application and shows the data in real time by using Kibana or other tools.
Stars: ✭ 106 (-20.9%)
Mutual labels:  logging, agent
Log Viewer
Web UI to viewing logs
Stars: ✭ 59 (-55.97%)
Mutual labels:  logging, logger
Serverlog
A simple, practical and innovative Node.js log library that enables you to view logs in Chrome dev tools and browser Console.
Stars: ✭ 117 (-12.69%)
Mutual labels:  logging, logger
Silencer
Easily suppress the Rails logger
Stars: ✭ 116 (-13.43%)
Mutual labels:  logging, logger
Loguru
Python logging made (stupidly) simple
Stars: ✭ 10,510 (+7743.28%)
Mutual labels:  logging, logger
Docker Logger
Logs collector for docker
Stars: ✭ 126 (-5.97%)
Mutual labels:  logging, logger
Android Filelogger
A general-purpose logging library with built-in support to save logs to file efficiently.
Stars: ✭ 70 (-47.76%)
Mutual labels:  logging, logger
Structured Log
A JavaScript implementation of Serilog's hybrid text/structured logging
Stars: ✭ 133 (-0.75%)
Mutual labels:  logging, logger
Menubarosxapp
Generic Mac OS X menu bar app (agent) template
Stars: ✭ 63 (-52.99%)
Mutual labels:  mac, agent
Iglance
Free system monitor for OSX and macOS. See all system information at a glance in the menu bar.
Stars: ✭ 1,358 (+913.43%)
Mutual labels:  mac, monitor
Yii2 Psr Log Target
Yii 2.0 log target that is able to write messages to PSR-3 compatible logger
Stars: ✭ 58 (-56.72%)
Mutual labels:  logging, logger
Bulk
👨‍💻 Bulk is a library for buffering the objects. Pipeline(Sink) receives the object and emits the object bulked.
Stars: ✭ 59 (-55.97%)
Mutual labels:  logging, logger
Node Lambda Log
Basic logging mechanism for Node 6.10+ Lambda Functions
Stars: ✭ 115 (-14.18%)
Mutual labels:  logging, logger
Plug logger json
Elixir Plug that formats http request logs as json
Stars: ✭ 125 (-6.72%)
Mutual labels:  logging, logger

LogDNA Agent

LogDNA Agent streams from log files to your LogDNA account. Works with Linux, Windows, and macOS Servers.

Getting Started

From an Official Release

Check out the official LogDNA site to learn how to sign up for an account and get started.

From Source

Follow these instructions to run the LogDNA agent from source:

git clone https://github.com/logdna/logdna-agent.git
cd logdna-agent
npm install

# help
sudo node index.js --help

# configure
sudo node index.js -k <YOUR LOGDNA INGESTION KEY>
# On Linux, this will generate a config file at: /etc/logdna.conf
# On Windows, this will generate a config file at: C:\ProgramData\logdna\logdna.conf

# on Linux, /var/log is monitored/added by default (recursively). You can optionally specify more folders
# on Windows, C:\ProgramData\logs is monitored/added by default (recursively). You can optionally specify more folders
sudo node index.js -d /path/to/log/folders -d /path/to/2nd/folder
sudo node index.js -d /var/log                            # folder only assumes *.log + extensionless files
sudo node index.js -d "/var/log/*.txt"                    # supports glob patterns
sudo node index.js -d "/var/log/**/*.txt"                 # *.txt in any subfolder
sudo node index.js -d "/var/log/**/myapp.log"             # myapp.log in any subfolder
sudo node index.js -d "/var/log/+(name1|name2).log"       # supports extended glob patterns
sudo node index.js -e /var/log/nginx/error.log            # exclude specific files from -d
sudo node index.js -f /usr/local/nginx/logs/access.log    # add specific files
sudo node index.js -t production                          # tags
sudo node index.js -t production,app1=/opt/app1           # tags for specific paths

# other commands
sudo node index.js -l                                     # show all saved options from config
sudo node index.js -l tags,key,logdir                     # show specific entries from config
sudo node index.js -u tags                                # unset tags
sudo node index.js -u tags,logdir                         # unset tags and logdir
sudo node index.js -u all                                 # unset everything except ingestion key

# start the agent
sudo node index.js

Note that when using glob patterns with index.js, you must enclose the pattern in double quotes.

Configuration File

Normally a config file is automatically generated (e.g. when you set a key using index.js -k) and updated (e.g. when you add a directory using index.js -d) but you can create your own config file /etc/logdna.conf on Linux and C:\ProgramData\logdna\logdna.conf on Windows and save your settings there:

logdir = /var/log/myapp,/path/to/2nd/dir
key = <YOUR LOGDNA INGESTION KEY>

On Windows, use \\ as a separator:

logdir = C:\\Users\\username\\AppData\\myapp
key = <YOUR LOGDNA INGESTION KEY>

Configuration File Options

  • logdir: sets the paths that the agent will monitor for new files. Multiple paths can be specified, separated by ,. Supports glob patterns + specific files. By default this option is set to monitor .log and extensionless files under /var/log/.
  • exclude: sets files to exclude that would otherwise match what's set in logdir. Multiple paths can be specified, separated by ,. Supports glob patterns + specific files
  • exclude_regex: filters out any log lines matching this pattern in any file. Should not include leading or trailing /.
  • key: your LogDNA Ingestion Key. You can obtain one by creating an account at LogDNA. Once logged in, click on the Gear icon, then Account Profile to find your key.
  • tags: tags can be used e.g. to separate data from production, staging, or autoscaling use cases
  • hostname: set this to override the os hostname

Features

  • The Agent maintains persistent connections to LogDNA ingestion servers with HTTPS encryption
  • Reconnects if disconnected and will queue up new log lines while disconnected
  • Compresses on upload (gzip)
  • Rescans for new files in all logdir paths every minute
  • Handles log rotated files on most operating systems (supports: renamed, truncated & "new file per day" log rotation methods)
  • Init script is available here (rpm/deb packages already include this)
# YUM Repo
echo "[logdna]
name=LogDNA packages
baseurl=https://repo.logdna.com/el6/
enabled=1
gpgcheck=1
gpgkey=https://repo.logdna.com/logdna.gpg" | sudo tee /etc/yum.repos.d/logdna.repo

# APT Repo
echo "deb https://repo.logdna.com stable main" | sudo tee /etc/apt/sources.list.d/logdna.list
wget -O- https://repo.logdna.com/logdna.gpg | sudo apt-key add -
sudo apt-get update

How it Works

The LogDNA agent authenticates using your LogDNA Ingestion Key and opens a secure web socket to LogDNA's ingestion servers. It then 'tails' for new log data, as well as watches for new files added to your specific logging directories.

If you don't have a LogDNA account, you can create one on https://logdna.com. Or if you're using macOS w/Homebrew installed:

brew cask install logdna-cli
logdna register <email>

Kubernetes Logging

Please see our documentation for Kubernetes instructions.

OpenShift Logging

OpenShift logging requires a few additional steps over Kubernetes, but still pretty easy! Like Kubernetes, we extract pertinent metadata: pod name, container name, container id, namespace, project, and labels etc:

oc adm new-project --node-selector='' logdna-agent
oc project logdna-agent
oc create serviceaccount logdna-agent
oc adm policy add-scc-to-user privileged system:serviceaccount:logdna-agent:logdna-agent
oc create secret generic logdna-agent-key --from-literal=logdna-agent-key=<YOUR LOGDNA INGESTION KEY>
oc create -f https://raw.githubusercontent.com/logdna/logdna-agent/master/logdna-agent-ds-os.yaml

This automatically installs a logdna-agent pod into each node in your cluster and ships stdout/stderr from all containers, both application logs and node logs. Note: By default, the agent pod will collect logs from all namespaces on each node, including kube-system. View your logs at https://app.logdna.com. See YAML file for additional options such as LOGDNA_TAGS.

Notes:

  • The oc adm new-project method prevents having to adjust the project namespace's node-selector after creation.
  • This uses JOURNALD=files, you may need to change this if you have changed OpenShift logging configuration.
  • This has been tested on OpenShift 3.5-11

Fedora CoreOS Logging

If you're using a docker-less containerized (e.g. podman) operating system like Fedora CoreOS where logs are forwarded to journald, you will be unable to use logspout (if your OS does use Docker, refer to LogDNA LogSpout for instructions on how to set that up).

You can run logdna-agent inside a container to read from journald with a few modifications. First, you'll need to set up systemd inside the container so that it can read from journalctl. Note that due to the systemd dependency, you may have some difficulties starting from a base image like Alpine. Instead, try starting from a distribution such as Debian or CentOS. The following assumes that you're starting from the Ubuntu based image provided by LogDNA.

In your Containerfile, install systemd:

FROM logdna/logdna-agent

# Install systemd so we can read logs via journalctl
RUN apt-get update \
    && apt-get install -y --no-install-recommends systemd \
    && rm -rf /var/lib/apt/lists/*

Next, you need to ensure the USEJOURNALD environment variable is set. If set to files, the agent will read from journald and forward the logs. The agent can be configured either in your image or entrypoint.

By default the agent will read logs from the /var/log. These logs won't be very useful since they'll be referencing the container and not the host. You cannot omit logdir since the agent will still read from /var/log if logdir is missing from the configuration. Instead, to disable reading from files, set logdir to an empty directory (e.g. /usr/src).

Finally, mount /var/log/journal inside the container so the agent can use journalctl to read logs. An example systemd service configuration could look like the following:

[Unit]
Description=LogDNA Forwarder
After=network-online.target
Wants=network-online.target

[Service]
Restart=on-failure
ExecStartPre=-/bin/podman kill logdna
ExecStartPre=-/bin/podman rm logdna
ExecStartPre=/bin/podman pull my-custom/logdna-agent
ExecStart=/bin/podman run -v /var/log/journal:/var/log/journal:z --name logdna my-custom/logdna-agent

[Install]
WantedBy=multi-user.target

Windows Logging

The LogDNA agent can be installed through Chocolatey. You will need:

  • Windows PowerShell v3+ (not PowerShell Core aka PowerShell 6 yet)
  • Windows 7+/Windows 2003+ (Server Core also, but not Windows Nano Server)
  • .NET Framework 4.x+

For more details, view our Windows Agent docs

LogDNA Pay-per-gig Pricing

Our paid plans start at $1.50/GB per month. Pay only for what you use, with no fixed data buckets. All paid plans include all features.

Contributors

Contributing

Contributions are always welcome. See the contributing guide to learn how you can help. Build instructions for the agent are also in the guide.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].