Organizational Research By

Surprising Reserch Topic

running hadoop job remotely

running hadoop job remotely  using -'hadoop'

I am trying to run a  MapReduce job from outside the cluster.

e.g. Hadoop cluster is running on Linux machines.
     We have one web application running on a Windows machine.
     We want to run the hadoop job from this remote web application.
     We want to retrieve the hadoop output directory and present it as a Graph.

We have written the following piece of code:

Configuration conf = new Configuration();

Job job = new Job(conf);

conf.set("mapred.job.tracker", "");

conf.set("", "hdfs://");

job.setJarByClass(Analysis.class) ;




FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));



And this is the error we get. Even if we shut down the hadoop 1.1.2 cluster, the error is still the same.

14/03/07 00:23:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/03/07 00:23:37 ERROR security.UserGroupInformation: PriviledgedActionException as:user Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
Exception in thread "main" Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(
at org.apache.hadoop.fs.FileUtil.setPermission(
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
at org.apache.hadoop.mapred.JobClient$
at org.apache.hadoop.mapred.JobClient$
at Method)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(
at org.apache.hadoop.mapreduce.Job.submit(
at org.apache.hadoop.mapreduce.Job.waitForCompletion(
at LineCounter.main(


asked Oct 11, 2015 by vibhorsingh
0 votes

Related Hot Questions

1 Answer

0 votes

While running from a remote system, you should run as remote user. You can do it in your main class as follows:

public static void main(String a[]) {
     UserGroupInformation ugi
     = UserGroupInformation.createRemoteUser("root");

     try {

        ugi.doAs(new PrivilegedExceptionAction() {

            public Void run() throws Exception {
                 Configuration conf = new Configuration();

                 Job job = new Job(conf);

                 conf.set("hadoop.job.ugi", "root");

                 // write your remaining piece of code here. 

                return null;

    } catch (Exception e) {


Also while submitting a mapreduce job, it should copy your java classes with their dependent jars to hadoop cluster, where it execute mapreduce job.You can read more here.

So you need to create a runnable jar of your code (with main class Analysis in your case) with all dependent jar files inits manifest classpath. Then run your jar file from your commandline using

java -jar job-jar-with-dependencies.jar arguments


answered Oct 11, 2015 by mtabakade