SparkEngine - Details for Query 56

Submitted Time: 2025/03/05 00:00:02
Duration: 0.7 s
Succeeded Jobs: 97 98 100

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand task commit time: 0 ms number of written files: 1 job commit time: 24 ms number of output rows: 1 number of dynamic part: 0 written output: 8.5 KiB"]; 2 [labelType="html" label=" WriteFiles "]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 103 ms"; 4 [labelType="html" label=" Project "]; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label=" Project "]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 368.0 B number of coalesced partitions: 1"]; 8 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 2 ms (0 ms, 2 ms, 2 ms (stage 142.0: task 97)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 352.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 416.0 B (0.0 B, 416.0 B, 416.0 B (stage 142.0: task 97)) local merged bytes read: 0.0 B number of partitions: 200 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 352.0 B (0.0 B, 352.0 B, 352.0 B (stage 142.0: task 97))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 3 ms"; 10 [labelType="html" label="Scan JDBCRelation(\"SDATA_YEWU\".\"t_materials_list\") [numPartitions=1] number of output rows: 1"]; } 1->0; 2->1; 4->2; 5->4; 6->5; 7->6; 8->7; 10->8; }

AdaptiveSparkPlan isFinalPlan=true

Execute InsertIntoHadoopFsRelationCommand file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

WriteFiles

Project [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, cast(quantity#4418 as decimal(38,0)) AS quantity#6740, warning_content#4419, warehouse_date#4420, cast(storage_age#4421 as decimal(38,0)) AS storage_age#6743, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, ... 5 more fields]

Project [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, ... 4 more fields]

Project [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, ... 7 more fields]

WholeStageCodegen (3)

AQEShuffleRead coalesced

Exchange hashpartitioning(data_id#4406, 200), ENSURE_REQUIREMENTS, [plan_id=1720]

Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1] [data_id#4406,material_code#4407,material_name#4408,model#4409,unit#4410,whether_audited#4411,specification#4412,brand#4413,bom_number#4414,material_number_erp#4415,material_description#4416,material_note#4417,quantity#4418,warning_content#4419,warehouse_date#4420,storage_age#4421,whether_age_warning#4422,del_flag#4423,create_member#4424,office_id#4425,create_time#4426,create_member_ip_address#4427,last_modifier#4428,last_modify_time#4429,last_modify_ip_address#4430] PushedFilters: [], ReadSchema: struct<data_id:string,material_code:string,material_name:string,model:string,unit:string,whether_...

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (20)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (9)
   +- WriteFiles (8)
      +- * Project (7)
         +- * Project (6)
            +- * Project (5)
               +- AQEShuffleRead (4)
                  +- ShuffleQueryStage (3), Statistics(sizeInBytes=416.0 B, rowCount=1)
                     +- Exchange (2)
                        +- * Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (19)
   +- WriteFiles (18)
      +- Project (17)
         +- Project (16)
            +- SortMergeJoin LeftOuter (15)
               :- Sort (11)
               :  +- Exchange (10)
               :     +- Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  (1)
               +- Sort (14)
                  +- Exchange (13)
                     +- Scan JDBCRelation("SDATA_YEWU"."t_bpbjgl_warning_configure") [numPartitions=1]  (12)


(1) Scan JDBCRelation("SDATA_YEWU"."t_materials_list") [numPartitions=1]  [codegen id : 1]
Output [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
ReadSchema: struct<data_id:string,material_code:string,material_name:string,model:string,unit:string,whether_audited:string,specification:string,brand:string,bom_number:string,material_number_erp:string,material_description:string,material_note:string,quantity:int,warning_content:string,warehouse_date:timestamp,storage_age:int,whether_age_warning:string,del_flag:string,create_member:string,office_id:string,create_time:timestamp,create_member_ip_address:string,last_modifier:string,last_modify_time:timestamp,last_modify_ip_address:string>

(2) Exchange
Input [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
Arguments: hashpartitioning(data_id#4406, 200), ENSURE_REQUIREMENTS, [plan_id=1720]

(3) ShuffleQueryStage
Output [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
Arguments: 0

(4) AQEShuffleRead
Input [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
Arguments: coalesced

(5) Project [codegen id : 3]
Output [31]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, cast(null as string) AS material_number#4572, cast(null as int) AS lower#4574, cast(null as int) AS low#4575, cast(null as int) AS HIGH#4576, cast(null as int) AS higher#4577, cast(null as int) AS warning_days#4578]
Input [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]

(6) Project [codegen id : 3]
Output [28]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#4578, CASE WHEN (quantity#4418 <= lower#4574) THEN 库存数量低低限 WHEN ((quantity#4418 <= low#4575) AND (quantity#4418 > lower#4574)) THEN 库存数量低限 WHEN ((quantity#4418 > low#4575) AND (quantity#4418 < HIGH#4576)) THEN null WHEN ((quantity#4418 >= HIGH#4576) AND (quantity#4418 < higher#4577)) THEN 库存数量高限 WHEN (quantity#4418 >= higher#4577) THEN 库存数量高高限 END AS inventory_warning#4877, datediff(2025-03-05, cast(warehouse_date#4420 as date)) AS kl#4878]
Input [31]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, material_number#4572, lower#4574, low#4575, HIGH#4576, higher#4577, warning_days#4578]

(7) Project [codegen id : 3]
Output [29]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, cast(quantity#4418 as decimal(38,0)) AS quantity#6740, warning_content#4419, warehouse_date#4420, cast(storage_age#4421 as decimal(38,0)) AS storage_age#6743, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, cast(warning_days#4578 as decimal(38,0)) AS warning_days#6753, inventory_warning#4877, cast(kl#4878 as decimal(38,0)) AS kl#6755, CASE WHEN (kl#4878 >= warning_days#4578) THEN 1 WHEN (kl#4878 < warning_days#4578) THEN 0 END AS warning_or_not#6756]
Input [28]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#4578, inventory_warning#4877, kl#4878]

(8) WriteFiles
Input [29]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#6740, warning_content#4419, warehouse_date#4420, storage_age#6743, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#6753, inventory_warning#4877, kl#6755, warning_or_not#6756]

(9) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

(10) Exchange
Input [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
Arguments: hashpartitioning(data_id#4406, 200), ENSURE_REQUIREMENTS, [plan_id=1707]

(11) Sort
Input [25]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430]
Arguments: [data_id#4406 ASC NULLS FIRST], false, 0

(12) Scan JDBCRelation("SDATA_YEWU"."t_bpbjgl_warning_configure") [numPartitions=1] 
Output [6]: [material_number#4572, lower#4574, low#4575, HIGH#4576, higher#4577, warning_days#4578]
PushedFilters: [*IsNotNull(material_number)]
ReadSchema: struct<material_number:string,lower:int,low:int,HIGH:int,higher:int,warning_days:int>

(13) Exchange
Input [6]: [material_number#4572, lower#4574, low#4575, HIGH#4576, higher#4577, warning_days#4578]
Arguments: hashpartitioning(material_number#4572, 200), ENSURE_REQUIREMENTS, [plan_id=1708]

(14) Sort
Input [6]: [material_number#4572, lower#4574, low#4575, HIGH#4576, higher#4577, warning_days#4578]
Arguments: [material_number#4572 ASC NULLS FIRST], false, 0

(15) SortMergeJoin
Left keys [1]: [data_id#4406]
Right keys [1]: [material_number#4572]
Join type: LeftOuter
Join condition: None

(16) Project
Output [28]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#4578, CASE WHEN (quantity#4418 <= lower#4574) THEN 库存数量低低限 WHEN ((quantity#4418 <= low#4575) AND (quantity#4418 > lower#4574)) THEN 库存数量低限 WHEN ((quantity#4418 > low#4575) AND (quantity#4418 < HIGH#4576)) THEN null WHEN ((quantity#4418 >= HIGH#4576) AND (quantity#4418 < higher#4577)) THEN 库存数量高限 WHEN (quantity#4418 >= higher#4577) THEN 库存数量高高限 END AS inventory_warning#4877, datediff(2025-03-05, cast(warehouse_date#4420 as date)) AS kl#4878]
Input [31]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, material_number#4572, lower#4574, low#4575, HIGH#4576, higher#4577, warning_days#4578]

(17) Project
Output [29]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, cast(quantity#4418 as decimal(38,0)) AS quantity#6740, warning_content#4419, warehouse_date#4420, cast(storage_age#4421 as decimal(38,0)) AS storage_age#6743, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, cast(warning_days#4578 as decimal(38,0)) AS warning_days#6753, inventory_warning#4877, cast(kl#4878 as decimal(38,0)) AS kl#6755, CASE WHEN (kl#4878 >= warning_days#4578) THEN 1 WHEN (kl#4878 < warning_days#4578) THEN 0 END AS warning_or_not#6756]
Input [28]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#4418, warning_content#4419, warehouse_date#4420, storage_age#4421, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#4578, inventory_warning#4877, kl#4878]

(18) WriteFiles
Input [29]: [data_id#4406, material_code#4407, material_name#4408, model#4409, unit#4410, whether_audited#4411, specification#4412, brand#4413, bom_number#4414, material_number_erp#4415, material_description#4416, material_note#4417, quantity#6740, warning_content#4419, warehouse_date#4420, storage_age#6743, whether_age_warning#4422, del_flag#4423, create_member#4424, office_id#4425, create_time#4426, create_member_ip_address#4427, last_modifier#4428, last_modify_time#4429, last_modify_ip_address#4430, warning_days#6753, inventory_warning#4877, kl#6755, warning_or_not#6756]

(19) Execute InsertIntoHadoopFsRelationCommand
Input: []
Arguments: file:/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240, false, Parquet, [path=/home/sdata/tomcat/webapps/storage_area/private/config/123456789/钱嘉乐shurui/asset/datafile/tmp/4858262574730240], Overwrite, [data_id, material_code, material_name, model, unit, whether_audited, specification, brand, bom_number, material_number_erp, material_description, material_note, quantity, warning_content, warehouse_date, storage_age, whether_age_warning, del_flag, create_member, office_id, create_time, create_member_ip_address, last_modifier, last_modify_time, ... 5 more fields]

(20) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true