Spark經典案例7-非結構數據處理

2019-11-10 18:38:22

字體：大中小

來源：轉載

供稿：網友

需求：根據tomcat日志計算url訪問了情況，具體的url如下，要求：區別統計GET和POST URL訪問量結果為：訪問方式、URL、訪問量測試數據集：在CODE上查看代碼片派生到我的代碼片 196.168.2.1 - - [03/Jul/2014:23:36:38 +0800] “GET /course/detail/3.htm HTTP/1.0” 200 38435 0.038 182.131.89.195 - - [03/Jul/2014:23:37:43 +0800] “GET /html/notes/20140617/888.html HTTP/1.0” 301 - 0.000 196.168.2.1 - - [03/Jul/2014:23:38:27 +0800] “POST /service/notes/addViewTimes_23.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:39:03 +0800] “GET /html/notes/20140617/779.html HTTP/1.0” 200 69539 0.046 196.168.2.1 - - [03/Jul/2014:23:43:00 +0800] “GET /html/notes/20140318/24.html HTTP/1.0” 200 67171 0.049 196.168.2.1 - - [03/Jul/2014:23:43:59 +0800] “POST /service/notes/addViewTimes_779.htm HTTP/1.0” 200 1 0.003 196.168.2.1 - - [03/Jul/2014:23:45:51 +0800] “GET /html/notes/20140617/888.html HTTP/1.0” 200 70044 0.060 196.168.2.1 - - [03/Jul/2014:23:46:17 +0800] “GET /course/list/73.htm HTTP/1.0” 200 12125 0.010 196.168.2.1 - - [03/Jul/2014:23:46:58 +0800] “GET /html/notes/20140609/542.html HTTP/1.0” 200 94971 0.077 196.168.2.1 - - [03/Jul/2014:23:48:31 +0800] “POST /service/notes/addViewTimes_24.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:48:34 +0800] “POST /service/notes/addViewTimes_542.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:49:31 +0800] “GET /notes/index-top-3.htm HTTP/1.0” 200 53494 0.041 196.168.2.1 - - [03/Jul/2014:23:50:55 +0800] “GET /html/notes/20140609/544.html HTTP/1.0” 200 183694 0.076 196.168.2.1 - - [03/Jul/2014:23:53:32 +0800] “POST /service/notes/addViewTimes_544.htm HTTP/1.0” 200 2 0.004 196.168.2.1 - - [03/Jul/2014:23:54:53 +0800] “GET /service/notes/addViewTimes_900.htm HTTP/1.0” 200 151770 0.054 196.168.2.1 - - [03/Jul/2014:23:57:42 +0800] “GET /html/notes/20140620/872.html HTTP/1.0” 200 52373 0.034 196.168.2.1 - - [03/Jul/2014:23:58:17 +0800] “POST /service/notes/addViewTimes_900.htm HTTP/1.0” 200 2 0.003 196.168.2.1 - - [03/Jul/2014:23:58:51 +0800] “GET /html/notes/20140617/888.html HTTP/1.0” 200 70044 0.057 186.76.76.76 - - [03/Jul/2014:23:48:34 +0800] “POST /service/notes/addViewTimes_542.htm HTTP/1.0” 200 2 0.003 186.76.76.76 - - [03/Jul/2014:23:46:17 +0800] “GET /course/list/73.htm HTTP/1.0” 200 12125 0.010 8.8.8.8 - - [03/Jul/2014:23:46:58 +0800] “GET /html/notes/20140609/542.html HTTP/1.0” 200 94971 0.077

由于Tomcat日志是不規則的，需要先過濾清洗數據。

package ClassicCaseimport org.apache.spark.{SparkConf, SparkContext}/** * 業務場景：分析非結構化數據 * Created by YJ on 2017/2/8. */object case7 { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local").setAppName("reduce") val sc = new SparkContext(conf) sc.setLogLevel("ERROR") val data = sc.textFile("hdfs://192.168.109.130:8020//user/flume/ClassicCase/case7/*") //filter 過濾長度小于0，過濾不包含GET與POST的URL val filtered = data.filter(_.length() > 0).filter(line => (line.indexOf("GET") > 0 || line.indexOf("POST") > 0)) //轉換成鍵值對操作 val res = filtered.map(line => { if (line.indexOf("GET") > 0) { //截取 GET 到URL的字符串 (line.substring(line.indexOf("GET"), line.indexOf("HTTP/1.0")).trim, 1) } else { //截取 POST 到URL的字符串 (line.substring(line.indexOf("POST"), line.indexOf("HTTP/1.0")).trim, 1) } //最后通過reduceByKey求sum }).reduceByKey(_ + _) //觸發action事件執行 res.collect() }}

輸出結果 (POST /service/notes/addViewTimes_779.htm,1), (GET /service/notes/addViewTimes_900.htm,1), (POST /service/notes/addViewTimes_900.htm,1), (GET /notes/index-top-3.htm,1), (GET /html/notes/20140318/24.html,1), (GET /html/notes/20140609/544.html,1), (POST /service/notes/addViewTimes_542.htm,2), (POST /service/notes/addViewTimes_544.htm,1), (GET /html/notes/20140609/542.html,2), (POST /service/notes/addViewTimes_23.htm,1), (GET /html/notes/20140617/888.html,3), (POST /service/notes/addViewTimes_24.htm,1), (GET /course/detail/3.htm,1), (GET /course/list/73.htm,2), (GET /html/notes/20140617/779.html,1), (GET /html/notes/20140620/872.html,1)

上一篇：c語言哈夫曼編碼

下一篇：藍橋搜索題