GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → phillipi → AMT_Real_vs_Fake

phillipi / AMT_Real_vs_Fake

Licence: BSD-2-Clause License
Code for running real vs fake experiments on Amazon Mechanical Turk

Programming Languages

HTML
75241 projects
python
139335 projects - #7 most used programming language

Labels

amtamazon-mechanical-turk

Projects that are alternatives of or similar to AMT Real vs Fake

turktool
Modern React app for bounding box annotation on mturk
Stars: ✭ 46 (+35.29%)
Mutual labels:  amazon-mechanical-turk
amtgo
Intel AMT / vPro mass management tool
Stars: ✭ 25 (-26.47%)
Mutual labels:  amt
open-amt-cloud-toolkit
Open Active Management Technology (Open AMT) Cloud Toolkit provides modular microservices and libraries for integration of hardware provisioning, out-of-band (OOB) manageability, and other Intel® Active Management Technology (Intel® AMT). The toolkit makes it easier for IT departments and independent software vendors (ISVs) to adopt, integrate, …
Stars: ✭ 40 (+17.65%)
Mutual labels:  amt
2017-summer-workshop
Exercises, data, and more for our 2017 summer workshop (funded by the Estes Fund and in partnership with Project Jupyter and Berkeley's D-Lab)
Stars: ✭ 33 (-2.94%)
Mutual labels:  amazon-mechanical-turk
amtc
Threaded Intel AMT/vPro mass management tool
Stars: ✭ 40 (+17.65%)
Mutual labels:  amt

AMT_Real_vs_Fake

Run "real vs fake" experiments on Amazon Mechanical Turk (AMT).

Synopsis

Runs a series "real vs fake" trials. Each trial pits a real image against a "fake" image generated by an algorithm.

Requirements

Python

Usage

  • Put all images to test in a web accessible folder. This folder should have subfolders for the results of each algorithm you would like to test (names of subfolders are specified in opt.which_algs_paths). Must also contain a subfolder for the real images (path: opt['gt_path']). Images should be named "0.jpg", "1.jpg", etc, in consecutive order up to some total number of images N (or they can be named differently, but you will have to specify a lambda function in opt['filename']).
  • Set experiment parameters by modifying opt in getOpts function.
  • Run python mk_expt.py -n EXPT_NAME to generate data csv and index.html for Turk.
  • Create experiment using AMT website or command line tools. For the former option, paste contents of index.html into HIT html code. Upload HIT data from the generated csv.
  • After collecting results, run python process_csv.py -f CSV_FILENAME --N_imgs NUMBER_IMAGES --N_practice NUMBER_PRACTICE. This will compute and run bootstrap statistics.

Features

  • Can enforce that each Turker can only do HIT once (uses http://uniqueturker.myleott.com/; see opt['ut_id'])
  • If multiple algorithms are specified in opt['which_algs_paths'], then each HIT tests all algorithms randomly i.i.d. from this list.
  • If opt['paired'] is true, then "fake/n.jpg" will be pitted against "real/n.jpg"; if false, "fake/n.jpg" will be pitted against "real/m.jpg", for random n and m
  • See getDefaultOpts() for documentation on more features

Citation

This tool was initially developed for Colorful Image Colorization in Matlab (see this branch). This master branch has been translated into Python. Feel free to use this bibtex to cite.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].