TRANSFORMATIONS AND ACTIONS .pdfpython及scala代码实现的s

文件名称: TRANSFORMATIONS AND ACTIONS .pdf

所属分类: spark

开发工具:

文件大小: 3mb

下载次数: 0

上传时间: 2019-08-30

提供者: zizhuan*******

下载 (3mb)

不能下载？报告错误

详细说明：python及scala代码实现的spark算子及图解，能帮助你形象化的理解算子的意义sdatabricks Song Recommendations yor making big data simple oA-.ctextfile!sn: / /M_1_DCKET fiumI Py SLaI k syt -Ipur L Ruw det tul Lypase Founded in late 2013 1l, key, Loud ess,Btisic tron songs TABLESANPLE(BUCKET 1 auT DFmoa] where year 1934 aad yea-49017 by the creators of Apache Spark Original team from UC Berkeley AMPlab at· nn you take a lck ntia? Raised $47 Million in 2 rounds 55 employees .we'rehiring(http://databricks.workable.com) End oNface in Level 2/3 support partnerships with · Hortonworks MapR Databricks Cloud A unified platform for building Big data pipelines- from EtL to Data stax Exploration and Dashboards, to Advanced Analytics and data Products RDD RDD Elements egene ey original item B transformed partition(s) A type object on driver user inpuf user functions mitted value Legend Randomized operation A(AnB) B)Set Theory/Relational operation Numeric calculation TRANSFORMATIONS Spark Operations ACTIONS - eas medium Essential Core Intermediate Spark Operations genera Math/ Statistical Set Theory/Relational Data Structure/l/o map sample union ° key By ·千i1ter randomSplit intersection zipWithIndex flatMap subtract zipWithUniqueID mapPartitions distinct zipPartitions mappartitionsWithIndex cartesian coalesce groupBy zIp 广 epartit1on · sorte repartitionAndSortwithinPartitions pipe reduce count takeordered veastextfile collect takesample aveAssequencefile aggregate max saveAsObjectFile fold saveAsHadoopdataset first sum saveAshadoopfile ·take histogram saveAsNewAPIHadoopDataset · Foreach mean saveAsNewAPIHadoopFile top variance treeAggregate · treeReduce samplevariance · foreachPartition countApprox °Co11 ectAsMap countApproxDistinct medium Essential Core Intermediate PairRDD Operations genera Math/ Statistical Set Theory/ Relatione Data structure flatMapvalues sampleBykey cogroup (=groupwith) · partition groupByKey oln reduceByKey subtractByKey educebykey locall fullouterjo foldBykey leftouterjoin aggregateBykey ightouter Join g sorbY combine Bykey keys countY values countByValue ountByValueApprox countApproxDistinctBykey countApproxDistinctBykey countByKeyApprox samplebykeyexact VS narrow de each partition of the parent RDd is used by multiple child rdd partitions may depend at most one partition of the child rdd on a single parent RDD partition LINEAGE "One of the challenges in providing rdds as an abstraction is choosing a representation for them that can track lineage across a wide range of 三 transformations The most interesting question in designing this interface is how to represent dependencies between RDDs We found it both sufficient and useful to classify dependencies into two types: narrow dependencies, where each partition of the parent rdd is used by at most one partition of the child RDd wide dependencies, where multiple child partitions may depend on it

(系统自动生成,下载前可以参看下载内容)