HA Multiscrape

Important note: troubleshooting

If you don't manage to scrape the value you are looking for, please enable debug logging and log_response. This will provide you with a lot of information for continued investigation. log_response will write all responses to files. If the value you want to scrape is not in the files with the output from BeautifulSoup (*-soup.txt), Multiscrape will not be able to scrape it. Most likely it is retrieved in the background by javascript. Your best chance in this case, is to investigate the network traffic in de developer tools of your browser, and try to find a json response containing the value you are looking for.

If all of this doesn't help, use the home assistant forum. I cannot give everyone personal assistance and please don't create github issues unless you are sure there is a bug. Check the wiki for a scraping guide and other details on the functionality of this component.

HA MultiScrape custom component

This Home Assistant custom component can scrape multiple fields (using CSS selectors) from a single HTTP request (the existing scrape sensor can scrape a single field only). The scraped data becomes available in separate sensors.

It is based on both the existing Rest sensor and the Scrape sensor. Most properties of the Rest and Scrape sensor apply.

Installation

Install via HACS (default store) or install manually by copying the files in a new 'custom_components/multiscrape' directory.

Example configuration (YAML)

multiscrape:
  - resource: https://www.home-assistant.io
    scan_interval: 3600
    sensor:
      - unique_id: ha_latest_version
        name: Latest version
        select: ".current-version > h1:nth-child(1)"
        value_template: '{{ (value.split(":")[1]) }}'
      - unique_id: ha_release_date
        icon: >-
          {% if is_state('binary_sensor.ha_version_check', 'on') %}
            mdi:alarm-light
          {% else %}
            mdi:bat
          {% endif %}
        name: Release date
        select: ".release-date"
    binary_sensor:
      - unique_id: ha_version_check
        name: Latest version == 2021.7.0
        select: ".current-version > h1:nth-child(1)"
        value_template: '{{ (value.split(":")[1]) | trim == "2021.7.0" }}'
        attributes:
          - name: Release notes link
            select: "div.links:nth-child(3) > a:nth-child(1)"
            attribute: href

Options

Based on latest (pre) release.

name	description	required	default	type
name	The name for the integration.	False		string
resource	The url for retrieving the site or a template that will output an url. Not required when `resource_template` is provided.	True		string
resource_template	A template that will output an url after being rendered. Only required when `resource` is not provided.	True		template
authentication	Configure HTTP authentication. `basic` or `digest`. Use this with username and password fields.	False		string
username	The username for accessing the url.	False		string
password	The password for accessing the url.	False		string
headers	The headers for the requests.	False		string - list
params	The query params for the requests.	False		string - list
method	The method for the request. Either `POST` or `GET`.	False	GET	string
payload	Optional payload to send with a POST request.	False		string
verify_ssl	Verify the SSL certificate of the endpoint.	False	True	boolean
log_response	Log the HTTP responses and HTML parsed by BeautifulSoup in files. (Will be written to/config/multiscrape/name_of_config)	False	False	boolean
timeout	Defines max time to wait data from the endpoint.	False	10	int
scan_interval	Determines how often the url will be requested.	False	60	int
parser	Determines the parser to be used with beautifulsoup. Either `lxml` or `html.parser`.	False	lxml	string
form_submit	See Form-submit	False
sensor	See Sensor	False		list
binary_sensor	See Binary sensor	False		list
button	See Refresh button	False		list

Sensor/Binary Sensor

Configure the sensors that will scrape the data.

name	description	required	default	type
unique_id	Will be used as entity_id and enables editing the entity in the UI	False		string
name	Friendly name for the sensor	False		string
select	CSS selector used for retrieving the value of the sensor. Only required when `select_list` is not provided.	True		string/template
select_list	CSS selector for multiple values of multiple elements which will be returned as csv. Only required when `select` is not provided.	True		string/template
attribute	Attribute from the selected element to read as value	False		string
value_template	Defines a template applied on the result of the selector to extract the value. For binary sensors, the sensor is on if the template evaluates as True	False		string/template
attributes	See Sensor attributes	False		list
unit_of_measurement	Defines the units of measurement of the sensor	False		string
device_class	Sets the device_class for sensors or binary sensors	False		string
state_class	Defines the state class of the sensor, if any. (measurement, total or total_increasing) (not for binary_sensor)	False	None	string
icon	Defines the icon or a template for the icon of the sensor. The value of the selector is provided as input for the template. For binary sensors, the value is parsed in a boolean.	False		string/template
picture	Contains a path to a local image and will set it as entity picture	False		string
force_update	Sends update events even if the value hasn’t changed. Useful if you want to have meaningful value graphs in history.	False	False	boolean
on_error	See On-error	False

Refresh button

Configure a refresh button to manually trigger scraping.

name	description	required	default	type
unique_id	Will be used as entity_id and enables editing the entity in the UI	False		string
name	Friendly name for the button	False		string

Sensor attributes

Configure the attributes on the sensor that can be set with additional scraping values.

name	description	required	type
name	Name of the attribute (will be slugified)	True	string
select	CSS selector used for retrieving the value of the attribute. Only required when `select_list` is not provided.	True	string/template
select_list	CSS selector for multiple values of multiple elements which will be returned as csv. Only required when `select` is not provided.	True	string/template
attribute	Attribute from the selected element to read as value	False	string
value_template	Defines a template applied on the result of the selector to extract the value	False	string/template
on_error	See On-error	False

Form-submit

Configure the form-submit functionality which enables you to submit a (login) form before scraping a site. More details on how this works can be found on the wiki.

name	description	required	default	type
resource	The url for the site with the form	False		string
select	CSS selector used for selecting the form in the html	True		string
input	A dictionary with name/values which will be merged with the input fields on the form	False		string - dictionary
input_filter	A list of input fields that should not be submitted with the form	False		string - list
submit_once	Submit the form only once on startup instead of each scan interval	False	False	boolean
resubmit_on_error	Resubmit the form after a scraping error is encountered	False	True	boolean

On-error

Configure what should happen in case of a scraping error (the css selector does not return a value).

name	description	required	default	type
log	Determines if and how something should be logged in case of a scraping error. Value can be either 'false', 'info', 'warning' or 'error'.	False	error	string
value	Determines what value the sensor/attribute should get in case of a scraping error. The value can be 'last' meaning that the value does not change, 'none' which results in HA showing 'Unkown' on the sensor, or 'default' which will show the specified default value.	False	none	string
default	The default value to be used when the on-error value is set to 'default'.	False		string

Services

For each multiscrape instance, a service will be created to trigger a scrape run through an automation. (For manual triggering, the button entity can now be configured.) The services are named multiscrape.trigger_{name of integration}.

Debug logging

Debug logging can be enabled as follows:

logger:
  default: info
  logs:
    custom_components.multiscrape: debug

Depending on your issue, also consider enabling log_response.

Contributions are welcome!

If you want to contribute to this please read the Contribution guidelines

Credits

This project was generated from @oncleben31's Home Assistant Custom Component Cookiecutter template.

Code template was mainly taken from @Ludeeus's integration_blueprint template

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

danieldotnl / ha-multiscrape

Programming Languages

Labels

Projects that are alternatives of or similar to ha-multiscrape