hadoop升级调用FileSystem.get错误
把Hadoop升级以后，调用FileSystem.get会报如下错误：
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:210)
at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:109)
at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.close(GzipCodec.java:126)
at org.apache.hadoop.util.LineReader.close(LineReader.java:87)
at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:214)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:218)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:404)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.securit
请问一下是什么原因导致的？
愤怒大光头
11 years, 7 months ago
Answers
这个问题应该是多线程搜索时出现的，由于每次打开的连接次数太多，导致连接没有关闭。出现上面的错误。
解决办法：
1、在servlet初始化中，加入：
public void init(ServletConfig config) throws ServletException {
try {
this.conf = NutchConfiguration.get(config.getServletContext());
bean = NutchBean.get(config.getServletContext(), this.conf);
} catch (IOException e) {
throw new ServletException(e);
}
MAX_HITS_PER_PAGE = conf.getInt("searcher.max.hits.per.page", -1);
}
2、在web.xml，添加如下代码：
<listener>
<listener-class>org.apache.nutch.searcher.NutchBean$NutchBeanConstructor</listener-class>
</listener>
<servlet>
<servlet-name>Cached</servlet-name>
<servlet-class>org.apache.nutch.servlet.Cached</servlet-class>
</servlet>
3、在自己的servlet中把NutchBean的实例和NutchConfiguration的实例传递过去。保证初始化时只打开一次index。
Loxias
answered 11 years, 7 months ago