Steps to write a map reduce program

Steps to write a map reduce program:-

  1. make sure that files are available in Hdfs
  2. Design MR flow diagram
  3. Eclipse

HT

pom.xml

src/main/java

package

class

com.hadoop.training.wordcount

  • Wc driver
  • Wc mapper
  • Wc reducer

HTC

  • Pom.xml
  • Pkg
  • Class
  • Toolrunner.rm (wcDriver)

Driver:

  • Job details
  • i/p format
  • mapreduce –K, V
  • Reduce — K, V
  • Mapper
  • Reducer
  1. Mapper<m i/p k, m i/p v, m o/p k, m o/p v>

Mapper (longwritable, text, text, intwritable)

  1. Map <mi/pk key, mi/pv value, context output>

Map(longwritable key, text value, context output)

  1. Reducer<RI/pk, RI/pv, R o/p k, R o/p V>

Reducer(text, longwritable, text, intwritable)

  1. Reduce(text key, iterable (longwritable) value, context text)

Problems in wordcount:

Problem1:

Adhar card problem:

Addcardno ssid1 gender state Have/Not

1467 ssid1 M UP No

1468 ssid2 F Ap Yes

1469 ssid3 M MP NO

1490 ssid4 F TN Yes

1471 ssid5 M KA No

 

Problem2:

Find most popular website:

Reg to N url clicks Size

En google 10 500

En yahoo 20 200

Ind google 50 50

Ind google 60 150

USA yahoo 70 600

USA google 100 800

china google 5 900

china Yahoo 10 400

AMI Yahoo 11 500

AMI google 15 100

 

O/P:

EN

google?

Yahoo ?

problem-3:

wiki data mining:

offset En google 10 500

offset En yahoo 20 200

offset Ind google 50 50

offset Ind google 60 150

offset USA yahoo 70 600

offset USA google 100 800

offset china google 5 900

offset china Yahoo 10 400

offset AMI Yahoo 11 500

offset AMI google 15 100

a[0] a[1] a[2] a[3]

 

mapper code:

google.com 10

yahoo.com 20

google.com 30

yahoo.com 40

 

M o/p K, V

-P

-s

-G

K g(v)

Google (10, 50)

Yahoo (20, 20)

R i/p- K, V

Reduce code

K g(V)

Google (10, 10)

Yahoo (20, 20)

Code:

A[] = line.split(“?”)

If(a[0] contains’en’)

{

Out.writer(a[1],a2[2])

}

Int sum;

For(——–)

{

Sum = sum+val ;

}

Out.writer(key, longwrite(sum));

}

En

Google 60

Yahoo 40

K V

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s