#Oink
The aim of this project is to provide a REST based interface for PIG execution. Oink is Pig on Servlet which provides the following functionalities:
- Register/unregister/view a Pig script
- Register/unregister/view a jar file (for customer defined UDF functions)
- Execute a Pig job
- View the status/stats of a Pig job
- Cancel a Pig job
For more information on the targetted use-case and REST documentation please refer to our wiki.
Usage
Requisites
- Tomcat with JDK 7
- Maven
Build
The following steps need to be followed in order to build the war file of the REST service:
- Clone the project on GitHub
- Do a maven build at the top level of the project using
mvn clean install - The war file will be generated in service/target/pig.war
Running test cases
- In order to run test cases use
mvn clean test
Deployment
- Shut down the tomcat if it is running using
bin/shutdown.sh - Kill the process manually if the tomcat hasn't shutdown gracefully
- Place the pig.war in the webapps directory of the Tomcat instance and remove the exploded directory of pig if it exists
- Startup the tomcat using
bin/startup.sh
Default Configuration
Default configuration is to use localhost as the Hadoop cluster. Following are configuration parameters available (under service/src/main/resources/default.properties file) which can be changed :
jobtracker=localhost:8021 (Job tracker address)
jobtracker.ui=http://localhost:50030/jobdetails.jsp?jobid= (Job tracker UI URL used for providing stats information)
fs.default.name=hdfs://localhost:8020 (Hadoop Namenode address)
scripts.basepath=hdfs://localhost:8020/tmp/pig/scripts/ (Location in DFS where PIG scripts will be stored)
jars.basepath=hdfs://localhost:8020/tmp/pig/jars/ (Location in DFS where jar files will be stored)
requests.basepath=hdfs://localhost:8020/tmp/pig/requests/ (Location where request specific information will be stored)
max.threads=20 (size of thread pool for running PIG requests in parallel)
The core-site.xml and mapred-site.xml files are bundled along with the property files to provide cluster specific parameters like namenode address, jobtracker address in order to enable connecting to remote clusters. If the cluster is a secure cluster and uses authentication mechanisms like Kerberos then parameters like kerberos principal need to be provided in the mapred-site.xml and core-site.xml. Also, the queue to which jobs must be submitted also needs to provided in the mapred-site.xml
Different kinds of property files can be created based on the environment in which the service is going to be hosted.
For example a prod.properties file can be created to house the settings for a production instance. To pick up the property
file the following steps should be followed during startup of the Tomcat instance. The environment variable env should
be set as the desired environment as follows:
export env=prod
This would make sure that prod.properties is picked up by the ConfigurationLoader at startup.
Project Structure
The directory structure of our project is explained in the Project Structure wiki
Known Issues and Feature Requests
Please use our Issues page to view the existing issues, to raise bugs and request for new features.
Contribution Guidelines
Eager to contribute? Steps to contribute to our project is available in our Contribution Guidelines wiki