hadoop write file

December 5th, 2008

                        FSDataOutputStream out = fs.create(outFile);

In hadoop, I used the above line to prepare file for writing, Quite easy, and I don't have to worry about mkdir the parent directory if not there.

Previously, when I tried to do multi-thread output files, I need to do:

                        File parent = output.getParentFile();

                        if (!parent.exists()){

                            try {

                                parent.mkdirs();

                            } catch (Exception e) {

                                //multi thread might compete for this mkdirs()

Programming notes