SparkEngine - Details for Query 22

Submitted Time: 2025/03/04 00:00:04
Duration: 1 s
Succeeded Jobs: 34 35 40

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time: 0 ms number of written files: 1 job commit time: 39 ms number of output rows: 1 number of dynamic part: 0 written output: 8.5 KiB"]; 2 [labelType="html" label=" WriteFiles "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 156 ms"; 4 [labelType="html" label=" Project "]; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label=" Project "]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 368.0 B number of coalesced partitions: 1"]; 8 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 35 ms (0 ms, 35 ms, 35 ms (stage 49.0: task 34)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 354.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 416.0 B (0.0 B, 416.0 B, 416.0 B (stage 49.0: task 34)) local merged bytes read: 0.0 B number of partitions: 200 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 354.0 B (0.0 B, 354.0 B, 354.0 B (stage 49.0: task 34))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 177 ms"; 10 [labelType="html" label="Scan JDBCRelation(\"SDATA_YEWU\".\"t_materials_list\") [numPartitions=1] number of output rows: 1"]; } 1->0; 2->1; 4->2; 5->4; 6->5; 7->6; 8->7; 10->8; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

WriteFiles

Project [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, cast(quantity#494 as decimal(38,0)) AS quantity#3141, warning_content#495, warehouse_date#496, cast(storage_age#497 as decimal(38,0)) AS storage_age#3144, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, ... 5 more fields]

Project [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, ... 4 more fields]

Project [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, ... 7 more fields]

WholeStageCodegen (3)

AQEShuffleRead coalesced

Exchange hashpartitioning(data_id#482, 200), ENSURE_REQUIREMENTS, [plan_id=551]

Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1] [data_id#482,material_code#483,material_name#484,model#485,unit#486,whether_audited#487,specification#488,brand#489,bom_number#490,material_number_erp#491,material_description#492,material_note#493,quantity#494,warning_content#495,warehouse_date#496,storage_age#497,whether_age_warning#498,del_flag#499,create_member#500,office_id#501,create_time#502,create_member_ip_address#503,last_modifier#504,last_modify_time#505,last_modify_ip_address#506] PushedFilters: [], ReadSchema: struct<data_id:string,material_code:string,material_name:string,model:string,unit:string,whether_...

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (20)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (9)
   +- WriteFiles (8)
      +- * Project (7)
         +- * Project (6)
            +- * Project (5)
               +- AQEShuffleRead (4)
                  +- ShuffleQueryStage (3), Statistics(sizeInBytes=416.0 B, rowCount=1)
                     +- Exchange (2)
                        +- * Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (19)
   +- WriteFiles (18)
      +- Project (17)
         +- Project (16)
            +- SortMergeJoin LeftOuter (15)
               :- Sort (11)
               :  +- Exchange (10)
               :     +- Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  (1)
               +- Sort (14)
                  +- Exchange (13)
                     +- Scan JDBCRelation("SDATA_YEWU"."t_bpbjgl_warning_configure") [numPartitions=1]  (12)


(1) Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  [codegen id : 1]
Output [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
ReadSchema: struct<data_id:string,material_code:string,material_name:string,model:string,unit:string,whether_audited:string,specification:string,brand:string,bom_number:string,material_number_erp:string,material_description:string,material_note:string,quantity:int,warning_content:string,warehouse_date:timestamp,storage_age:int,whether_age_warning:string,del_flag:string,create_member:string,office_id:string,create_time:timestamp,create_member_ip_address:string,last_modifier:string,last_modify_time:timestamp,last_modify_ip_address:string>

(2) Exchange
Input [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
Arguments: hashpartitioning(data_id#482, 200), ENSURE_REQUIREMENTS, [plan_id=551]

(3) ShuffleQueryStage
Output [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
Arguments: 0

(4) AQEShuffleRead
Input [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
Arguments: coalesced

(5) Project [codegen id : 3]
Output [31]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, cast(null as string) AS material_number#648, cast(null as int) AS lower#650, cast(null as int) AS low#651, cast(null as int) AS HIGH#652, cast(null as int) AS higher#653, cast(null as int) AS warning_days#654]
Input [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]

(6) Project [codegen id : 3]
Output [28]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#654, CASE WHEN (quantity#494 <= lower#650) THEN 库存数量低低限 WHEN ((quantity#494 <= low#651) AND (quantity#494 > lower#650)) THEN 库存数量低限 WHEN ((quantity#494 > low#651) AND (quantity#494 < HIGH#652)) THEN null WHEN ((quantity#494 >= HIGH#652) AND (quantity#494 < higher#653)) THEN 库存数量高限 WHEN (quantity#494 >= higher#653) THEN 库存数量高高限 END AS inventory_warning#953, datediff(2025-03-04, cast(warehouse_date#496 as date)) AS kl#955]
Input [31]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, material_number#648, lower#650, low#651, HIGH#652, higher#653, warning_days#654]

(7) Project [codegen id : 3]
Output [29]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, cast(quantity#494 as decimal(38,0)) AS quantity#3141, warning_content#495, warehouse_date#496, cast(storage_age#497 as decimal(38,0)) AS storage_age#3144, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, cast(warning_days#654 as decimal(38,0)) AS warning_days#3154, inventory_warning#953, cast(kl#955 as decimal(38,0)) AS kl#3156, CASE WHEN (kl#955 >= warning_days#654) THEN 1 WHEN (kl#955 < warning_days#654) THEN 0 END AS warning_or_not#3157]
Input [28]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#654, inventory_warning#953, kl#955]

(8) WriteFiles
Input [29]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#3141, warning_content#495, warehouse_date#496, storage_age#3144, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#3154, inventory_warning#953, kl#3156, warning_or_not#3157]

(9) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

(10) Exchange
Input [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
Arguments: hashpartitioning(data_id#482, 200), ENSURE_REQUIREMENTS, [plan_id=538]

(11) Sort
Input [25]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506]
Arguments: [data_id#482 ASC NULLS FIRST], false, 0

(12) Scan JDBCRelation("SDATA_YEWU"."t_bpbjgl_warning_configure") [numPartitions=1] 
Output [6]: [material_number#648, lower#650, low#651, HIGH#652, higher#653, warning_days#654]
PushedFilters: [*IsNotNull(material_number)]
ReadSchema: struct<material_number:string,lower:int,low:int,HIGH:int,higher:int,warning_days:int>

(13) Exchange
Input [6]: [material_number#648, lower#650, low#651, HIGH#652, higher#653, warning_days#654]
Arguments: hashpartitioning(material_number#648, 200), ENSURE_REQUIREMENTS, [plan_id=539]

(14) Sort
Input [6]: [material_number#648, lower#650, low#651, HIGH#652, higher#653, warning_days#654]
Arguments: [material_number#648 ASC NULLS FIRST], false, 0

(15) SortMergeJoin
Left keys [1]: [data_id#482]
Right keys [1]: [material_number#648]
Join type: LeftOuter
Join condition: None

(16) Project
Output [28]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#654, CASE WHEN (quantity#494 <= lower#650) THEN 库存数量低低限 WHEN ((quantity#494 <= low#651) AND (quantity#494 > lower#650)) THEN 库存数量低限 WHEN ((quantity#494 > low#651) AND (quantity#494 < HIGH#652)) THEN null WHEN ((quantity#494 >= HIGH#652) AND (quantity#494 < higher#653)) THEN 库存数量高限 WHEN (quantity#494 >= higher#653) THEN 库存数量高高限 END AS inventory_warning#953, datediff(2025-03-04, cast(warehouse_date#496 as date)) AS kl#955]
Input [31]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, material_number#648, lower#650, low#651, HIGH#652, higher#653, warning_days#654]

(17) Project
Output [29]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, cast(quantity#494 as decimal(38,0)) AS quantity#3141, warning_content#495, warehouse_date#496, cast(storage_age#497 as decimal(38,0)) AS storage_age#3144, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, cast(warning_days#654 as decimal(38,0)) AS warning_days#3154, inventory_warning#953, cast(kl#955 as decimal(38,0)) AS kl#3156, CASE WHEN (kl#955 >= warning_days#654) THEN 1 WHEN (kl#955 < warning_days#654) THEN 0 END AS warning_or_not#3157]
Input [28]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#494, warning_content#495, warehouse_date#496, storage_age#497, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#654, inventory_warning#953, kl#955]

(18) WriteFiles
Input [29]: [data_id#482, material_code#483, material_name#484, model#485, unit#486, whether_audited#487, specification#488, brand#489, bom_number#490, material_number_erp#491, material_description#492, material_note#493, quantity#3141, warning_content#495, warehouse_date#496, storage_age#3144, whether_age_warning#498, del_flag#499, create_member#500, office_id#501, create_time#502, create_member_ip_address#503, last_modifier#504, last_modify_time#505, last_modify_ip_address#506, warning_days#3154, inventory_warning#953, kl#3156, warning_or_not#3157]

(19) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4846938189466624], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

(20) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true