Hasal Performance Test
Master
Dev
"Hasal" /ha'sɑlu/ this word is came from Indigenous Taiwanese "Bunun", and it means hail. Why we choose this word is because we expect our testing could like the hail, fast and weighty. And through our testing could bring more improvement on our performance or quality.
A Framework for testing web performance between different browser
Automatical Installation
If you do not wish to install Hasal automatically, please proceed to next section "Manual Installation". For those who wants to install Hasal, please be notified that a clean environment or
- Ubuntu
.\bootstrap-linux.sh
- Mac OS
.\bootstrap-mac.sh
- Windows 7 Launch a command line console with administrator privilege.
bootstrap.bat
- Windows 8 or above Launch a powershell console with administrator privilege.
Set-ExecutionPolicy Unrestricted
.\bootstrap.ps1
Please noted that Hasal in Windows systems can be run from command line only.
Manual Installation
- Install Firefox
- Install SikluliX (https://launchpad.net/sikuli/sikulix/1.1.0), place the runsikulix and other installation files in hasal/thirdparty. When sikulix setup diaglog come out, please select the Pack 1 and make sure scripting language: Python is checked.
- Install video recording codes and libs (windows/mac:ffmpeg or ubuntu:avconv)
- Install video recording main program (windows/mac:ffmpeg or ubuntu:avconv)
- Install opencv
- Download the client certificate from here (https://goo.gl/yfki48 -- note: needs a mozilla.com account ATM), place all certificates in your hasal working dir.
- Run setup.py
For Ubuntu:
apt-get install virtualenv python-dev
virtualenv ~/.hasalenv # or "make clean dev-env"
source ~/.hasalenv/bin/activate
pip install selenium
sudo apt-get install wget libav-tools ffmpeg libavc1394-0 libavformat-extra-53 libavfilter2 libavutil-extra-51 mencoder libavahi-common-data xsel xclip
wget https://github.com/bgirard/Gecko-Profiler-Addon/blob/master/geckoprofiler-signed.xpi?raw=true
wget https://github.com/Itseez/opencv/archive/3.0.0.zip
follow this link[http://www.pyimagesearch.com/2015/06/22/install-opencv-3-0-and-python-2-7-on-ubuntu/] to install opencv or:
sudo apt-get install build-essential cmake git pkg-config
sudo apt-get install libjpeg8-dev libtiff4-dev libjasper-dev libpng12-dev
sudo apt-get install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libgtk2.0-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libatlas-base-dev gfortran
sudo pip install virtualenv virtualenvwrapper
sudo apt-get install pip
sudo apt-get install python-pip
sudo apt-get install python2.7-dev
pip install numpy
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.1.0
cd ..
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib/
git checkout 3.1.0
cd ../opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_C_EXAMPLES=OFF -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules -D BUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig
ln -s /usr/local/lib/python2.7/site-packages/cv2.so virtualenv_path_you_create/lib/python2.7/site-packages/cv2.so
cd PATH_TO_HASAL
python setup.py install
For Mac OS:
virtualenv PATH_TO_YOUR_VENV # or "make clean dev-env"
source PATH_TO_YOUR_VENV/bin/activate
brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-frei0r --with-libass --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools
brew install libav
brew tap homebrew/science
brew install opencv
ln -s /usr/local//Cellar/opencv/2.4.13/lib/python2.7/site-packages/cv2.so PATH_TO_YOUR_VENV/lib/python2.7/site-packages/cv2.so
cd PATH_TO_HASAL
python setup.py install
OR manual build opencv2
virtualenv PATH_TO_YOUR_VENV # or "make clean dev-env"
source PATH_TO_YOUR_VENV/bin/activate
download the opencv2 package, compile and install
download the opencv2 package here : https://github.com/Itseez/opencv/archive/2.4.13.zip
unzip the package
cmake the folder unzipped
make
make install
ln -s /usr/local/lib/python2.7/site-packages/cv2.so PATH_TO_YOUR_VENV/lib/python2.7/site-packages/cv2.so
cd PATH_TO_HASAL
python setup.py install
Usage
Sample
- Trigger the framework:
python runtest.py
Usage
runtest.py [--exec-config=<str>] [--firefox-config=<str>] [--index-config=<str>] [--online-config=<str>] [--global-config=<str>]
runtest.py (-h | --help)
Options:
-h --help | Show this screen.
--exec-config=<str> | Specify the test execution config file; max-run, max-retry, advance, keep-browser etc. settings can be controlled in here. [default: configs/exec/default.json]
--firefox-config=<str> | Specify the test Firefox config file; [default: configs/firefox/default.json]
--index-config=<str> | Specify the index config file; you can specify which index you want to generate here. [default: configs/index/runtimeDctGenerator.json]
--online-config=<str> | Specify the online config file; you can specify if you want to enable online data submission and other related settings here. [default: configs/online/default.json]
--global-config=<str> | Specify the global config file; you can modify the output fn and status fn here. [default: configs/global/default.json]
Suite file template
-
regression test case format: test_script_path
Example: tests.regression.gdoc.test_firefox_gdoc_read_basic_txt_1
-
pilot test case format: test_sikuli_script_path
Example: tests/pilot/facebook/test_firefox_facebook_load_homepage.sikuli/
About generator
They are all inside configs/index/ directory. Generator is all about what data do we want to generate. We use runtimeDctGenerator.json as the default generator.
| Generator | What can we get? |
|---|---|
| runtimeDctGenerator | The total running time and related statistics |
| speedIndexDctGenerator | Get the SI/PSI for your test cases |
| inputLatencyDctGenerator | The input latency time for specific action (2 images) |
| inputLatencyAnimationDctGenerator | The input latency time for specific action (1 images) |
| frameThroughputDctGenerator | Get the frame throughput time for specific action |
Please be remind that most cases with ail in its naming are using inputLatencyAnimationDctGenerator and some of them are using inputLatencyDctGenerator. You may set it with --index-config= mentioned previously for runtest.py.
Basic Debugging
What if the Hasal framework doesn't work?
- Try to make the resolution of your pc to
1920*1080 - Try to set
about:blankas your browser start-up page - Remember to activate your virtualenv for all the things we set up for you
- Try to update your code and rerun bootstrap. For windows, you might want to recover the environment variable PATH from the backup file generated after you ran the bootstrap.bat
VM Template (out-of-date)
You can download the VM tempalte for Hasal framework environment from vagrant.
- vagrant init shako/hasal
- vagrant up --provider virtualbox
- Default user name and password : hasal/hasal