o <&i$@s ddlZddlZddlZddlZddlZddlmZmZmZddl m Z m Z ddl m Z mZmZmZmZmZmZmZmZmZmZddlZddlmmZddlmmZddlmmZddl mm!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4dd l5m6Z6m7Z7m8Z8m9Z9dd l:m;Z;dd lZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQmRZRmSZSmTZTdd lUmVZVmWZWmXZXmYZYdd lZm[Z[ddl\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdddlemfZfmgZgedZhGdddegZiiajiakedddgZl   ddemdeendeeeBdendeodelf ddZpddemdendeenfd d!Zqejrdd"dendeenfd#d$Zs dddd%dend&endeeeBdeod'eemdelf d(d)Ztd*eeIddfd+d,Zud-endenfd.d/Zvd0ee>deeBfd1d2Zwd3eId4e>deeBfd5d6Zxd3eId4ee>deenfd7d8Zyd0ee>deenfd9d:Zzd;ee*e,fdZ{d;ee*e,fd3eId4e>denfd?d@Z|d;ee*e,fd3eIdee>denfdAdBZ}d;ee*e,fd3eIdee>denfdCdDZ~e dEdFGdGdHdHZe dEdFGdIdJdJZe dEdFGdKdLdLZe dEdFGdMdNdNZe dEdFGdOdPdPZe3d3eIdenfdQdRZdSeJdeenfdTdUZd3eIdVe[deofdWdXZe dEdFGdYdZdZZd[emdenfd\d]Zd-endemfd^d_Zd`eRdenfdadbZdceeSdeenfdddeZd3eIdeeeenenfeenenfffdfdgZdhe)dieodjeendkeendleenenfdemf dmdnZdoe=dieodjeendkeendleenenfdemf dpdqZe3d3eIdemfdrdsZd3eIdeofdtduZe4d3eIdeeBe>fdenfdvdwZdxeeendyeende[fdzd{ZdeeIdeeeIeKffd|d}ZdeeIdeeeIeJffd~dZe#jddeeeIeJfdeeBe>fdeeeJeIfe>geenfdeeneenffddZdeeneenfdeenfddZe#jddeeeIeJfdeeBe>fdeeeJeIfe>geenfdeenfddZd3eeIeJfde>denfddZde`deeeIeJfdeBde>dVe[deodeodeodeodeenfddZdeeeIeJfdeBde>dVe[deodeodeenfddZdeeIde[deeenenffddZdeeIdeeeIeJfdeeJdee>dVe[deeBe>fde`de`deeBdeeBdeoddfddZdeeIdeeeIeJfdee>dVe[deeBe>fde`de`de`deeBdeeBdeoddfddZdeeIdeendeeeIeJfdeeJdee>dVe[deeBe>fde`de`de`de`deeBdeeBdeodeoddf ddZdeeIdeeeIeJfdeeJdeeKdVe[dee>deeBe>fde`de`de`de`deeBdeeBdeodeodeodeoddf$ddZde`deeIddfddZdejfddZdddZedkredSdS)N) defaultdict namedtuple OrderedDict) dataclassfield) AnyCallableDictListLiteralOptionalSequenceSetTupleTypeVarUnion)cpp) translate)Binding CppSignatureCppSignatureGroupDispatcherSignature NamedCTypeNativeSignatureSpecialArgName)method_with_native_functionnative_function_managerwith_native_function with_native_function_and_indices) gen_functionalization_definition"gen_functionalization_registration.gen_functionalization_view_inverse_declarationGenCompositeViewCopyKernelgen_all_vmap_plumbing)Argument BackendIndexBackendMetadataBaseOperatorNameDEFAULT_KERNEL_NAMESPACE DispatchKeyFRAGMENT_NAMESPACESFunctionSchemais_cuda_dispatch_keyis_generic_dispatch_keyis_ufunc_dispatch_keyLocationNativeFunctionNativeFunctionsGroupNativeFunctionsViewGroup OperatorName OptionalType SchemaKind SelfArgumentSTRUCTURED_DISPATCH_KEYSTensorOptionsArgumentsTypeVariantViewSchemaKind)add_generated_native_functionsgen_composite_functional_kernelgen_composite_out_kernelpre_group_native_functions)SelectiveBuilder) assert_never concatMapcontext FileManagermake_file_managermapMaybeNamespaceHelperTarget) YamlDumper YamlLoaderTcseZdZdfdd ZZS) LineLoaderFcs$tj||d}|jjd|d<|S)N)deep__line__)superconstruct_mappingZ start_markline)selfnoderNmapping __class__7C:\wamp64\www\opt\env\Lib\site-packages\torchgen/gen.pyrRszLineLoader.construct_mapping)F)__name__ __module__ __qualname__rR __classcell__rYrYrWrZrMsrM ParsedYamlnative_functionsbackend_indicesFes valid_tags ignore_keyspathskip_native_fns_genreturnc st|tsJg}tt}|D]H}t|dtsJ|t||d|dtfddt |||\}} | |t || Wdn1sRwYqt |tdd} |sit|||D]\} } t | ddt| | d| | <qmt|| S) NrPfunccsddSNzin z: rYrYfuncslocrYrZz*parse_native_yaml_struct..cSsttjdddidS)NTF dispatch_keyZuse_out_as_primaryZexternal device_guardindex)r&r* UndefinedrYrYrYrZrnsTFrp) isinstancelistrdictgetintr0rDr1Z from_yamlappendr&Z grow_indexerror_check_native_functionsr=itemsr-r_) rcrdrerfrgrsbserimindiceskvrYrkrZparse_native_yaml_structs8    rc st|tsJt}|D]M}t|dtsJ|t||d|dtfdd"|}|d}|dd}|dksEJ| |Wdn1sTwYq |S)NrPtagcsddSrjrYrYrmtagsrYrZrnroz(parse_tags_yaml_struct..desc) rurvsetrxryr0rDcopypopadd)rcrfr}rZe_inamerrYrrZparse_tags_yaml_structs     r)maxsizecCs\|tvr*t|}tj|td}t||dt|<Wdt|S1s%wYt|S)NLoader)rf)_GLOBAL_PARSE_TAGS_YAML_CACHEopenyamlloadrMr)rffrcrYrYrZparse_tags_yamls  r)rg loaded_yamltags_yaml_pathrcCsr|tvr5t|}|dur(t|}tj|td}Wdn1s"wYn|}t|||||dt|<t|S)Nr)rfrg)_GLOBAL_PARSE_NATIVE_YAML_CACHErrrrrMr)rfrrergrrdrrcrYrYrZparse_native_yamls    rrlcCsi}tt}|D]}|||jj<||jjj|q|D]g}|jdur=||j}|js=J|jjd|jd|jdd|jvrt|jjdkrt|jjdkr|jjj}|jjj }|j shJ|jjdt |j d|j }t||d ksJ|jjd |d qdS) Nz0 is marked as a structured_delegate pointing to z, but zY is not marked as structured. Consider adding 'structured=True' to the delegated operatorZ inplace_viewZresize_Z resize_as_z is marked with tag: inplace_view, but it doesn't follow the naming convention for inplace ops - the codegen expects the base name to have a trailing underscore. Frzw is marked with tag: inplace_view. The codegen expects there to be a corresponding out-of-place view op with the name 'z/' and matching schema, but it didn't find one. )rrvrirrzstructured_delegate structuredrstr overload_nameinplacer(baseZ dunder_methodlen)rlZfunc_mapZ base_func_maprZ delegate_func base_namerZout_of_place_base_namerYrYrZr{sB          r{scCsl|dd}|dd}|dd}|dd}|d d }|d d }|d d}|dd}d|dS)z1Convert a python string into a c++ string literal\z\\"z\"z\az\b z\f z\n z\v z\t)replacerrYrYrZ cpp_string%s         rbackendscCs2t|dkrgSdd|DtjtjtjtjgS)NrcSg|]}|jqSrY)rq).0backendrYrYrZ Cz(static_dispatch_keys..)rr*CompositeImplicitAutograd%CompositeImplicitAutogradNestedTensorCompositeExplicitAutograd&CompositeExplicitAutogradNonFunctionalrrYrYrZstatic_dispatch_keys?s  rr backend_indexcCsN|jdus ||r |jS|jrtjS|jrtjS|jrtj S|j r%tj SdSN) r has_kernelrq&has_composite_explicit_autograd_kernelr*r5has_composite_explicit_autograd_non_functional_kernelr&has_composite_implicit_autograd_kernelr4has_composite_implicit_autograd_nested_tensor_kernelr)rrrYrYrZget_static_dispatch_backendKsrcCsZ|dus|jr dSg}|D]}t||}|dur'|d|jd|dq d|S)N#include r)manual_kernel_registrationrrz root_namelowerjoin)rroutputrsrqrYrYrZstatic_dispatch_ops_header_s  rcCsddt|DS)NcSg|]}d|dqS)#include rY)rrqrYrYrZrps z1static_dispatch_extra_headers..)rrrYrYrZstatic_dispatch_extra_headersosrsigcpp_sigcCstdttdttfdd}t|}t|}|D]}|jjtjkr*||}nqt||}d dd|DS)Ninput_bindingsrhcSsTg}|D]#}|jdkr"tttj|jj|j|j|jd}| |q| |q|S)NZ memory_format)nctyperdefaultargument) rrrr possibly_redundant_memory_formatrtyperrrz)rZoutput_bindingsZbindingZspl_mem_format_bindingrYrYrZadd_spl_memory_format_binding~s  z5translate_args..add_spl_memory_format_binding, cs|]}|jVqdSrexprrarYrYrZ z!translate_args..) r rrv argumentsrrrrrr)rrrZ src_bindingsZ goal_bindingsargexprsrYrYrZtranslate_argsys   rc Cstj|ddd}|jr|jr|j}n|j}|dusJ|}t||}| |}|r3|j r3|j nt }| dd} d| d|j d|d|d S) NFmethodZfallback_binding::nativerreturn ::();)rfrom_native_functionsymintri has_symintsymint_signature signaturerr get_kernel cpp_namespacer)rrqr) rrrcpp_sigsrrrbackend_metadataZ kernel_nsnsrYrYrZ%generate_static_dispatch_backend_calls$    $rc Cs"tj|ddd}|jr|jr|j}n|j}|dusJ|}t||}t dd}|j rAd|dt j d|d|d S|jrVd|dt jd|d|d S|jrkd|dt jd|d|d S|jrd|dt jd|d|d Sd |d d d d |DdS)NFrrrrrrrz5TORCH_CHECK(false, "Static dispatch does not support z forrcSsg|]}t|jqSrY)rrq)rrsrYrYrZrsz:generate_static_dispatch_fallback_call..z ");)rrrrirrrrrr)rrr*rrrrrrrrr)rrrarrrrrrYrYrZ&generate_static_dispatch_fallback_calls*   $$$$rc sHt|dks jr dSfdd|D}t|dkr"t||dSt|dkr.t||Sdd|D}d|}jjj}g}g}|durN|d |dkr[|d |d |d d |d|dg} |D]} | d| j d| dt|| dqpt||} d} d| |d| | d| dS)a  For a given `NativeFunction`, find out the corresponding backend and dispatch to it. If more than one backends exsit, fallback to static dispatch by determining dispatch key from inputs. Arguments: sig: A CppSignature or DispatcherSignature for this native function we want to use. f: NativeFunction to generate static dispatch. backend_indices: All available backends. Return: C++ code to call backend-specific functions, e.g., "return at::cpu::add(self, other, scale);" rrcs.g|]}|sjdur|jtvr|qSr)rrrqr8)rbrrYrZrs  z#static_dispatch..rOcSs6g|]}t|jtst|jtr|jjr|jqSrY)rurr7r%ris_tensor_likerrrYrYrZrs   rNz>DispatchKeySet(c10::computeDispatchKey(dtype, layout, device))z$c10::detail::multi_dispatch_key_set()zDispatchKeySet _dk_set = z | ;z=DispatchKey _dk = c10::highestPriorityBackendTypeId(_dk_set);zcase DispatchKey:::rz  z switch (_dk) { z default: z } ) rrrrrrritensor_optionsrzrq) rrrakeysnative_tensor_args tensor_argsZ tensor_optsZstmtsZsubexprsZ dispatch_codersfallbackZ connectorrYrrZstatic_dispatchsN        rT)frozenc@sJeZdZUeed<eedZee e fed<e de de e fddZdS) RegisterSchemaselector)default_factory known_tagsrrhcCs|j|sdSddddt|jDd}|dkr(dtt|jdSd }||jvrBt |j}||j|<d |d |d }|dtt|jd |j|dS)N{rcss|]}d|VqdS)z at::Tag::NrY)rrrYrYrZr'z*RegisterSchema.__call__..}z{}zm.def(z, {}); rz const std::vector tags_z = ; z, tags_z); ) r is_native_function_selectedrsortedrrrrir r)rTrrZ maybe_tagsidxrYrYrZ__call__#s "   &zRegisterSchema.__call__N)r[r\r]rA__annotations__rrwr r rryrr1r rrYrYrYrZrs rc@sDeZdZUeejejfed<ee ed<e de de fddZ dS)ComputeOperatorstargetstatic_dispatch_backend_indicesrrhc Cst|j}|jj}|jtjurAd|d|d|jjjd|jjj dt t |jd|j ddd d |j d d d d S|jtj urd|d|jjjd|d|jjj d|dt t |jd|jd|d|d|d|d|d}dD]^}|rddgdd|D}d }nddd|D}d}|}|d |} d!|d"|d#|d$} |st|jd%krt|||jd&} |d'|jd(|j | |d d)| d*7}q||St|jdS)+Nz struct TORCH_API z { using schema = z; using ptr_schema = schema*; // See Note [static constexpr char* members for windows NVCC] STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(name, "aten::z>") STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(overload_name, "z:") STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(schema_str, z ) static callF)ris_redispatching_fnz ; static Z redispatchTz; };z+ STATIC_CONST_STR_OUT_OF_LINE_FOR_WIN_CUDA(z, name, "aten::z-") STATIC_CONST_STR_OUT_OF_LINE_FOR_WIN_CUDA(z, overload_name, "z, schema_str, z ) // aten::z. static C10_NOINLINE c10::TypedOperatorHandle create_zQ_typed_handle() { return c10::Dispatcher::singleton() .findSchemaOrThrow(z::name, z::overload_name) .typed(); } )FTrdispatchKeySetcSrrYrrrYrYrZrurz-ComputeOperators.__call__..cSrrYrrrYrYrZryrrz static auto op = create_z_typed_handle(); return op.rrrra // aten::r { z } )r from_schemarirunambiguous_namerrI DECLARATIONrrrrdefn DEFINITIONrrrrrrB) rTrrrZdefnsrZdispatcher_exprs_strZ method_baseZdispatcher_call method_nameZfn_bodyrYrYrZr<s            zComputeOperators.__call__N)r[r\r]r rIr!r#rr r&rr1rrrYrYrYrZr7s  rc@&eZdZededeefddZdS)ComputeFunctionrrhc Cstj|d|jd}|j}d}|D]\}t|j}t| | }d dd|D}|j r5d} nd} t j |jvrU|d |jd |d |jjd |d 7}|rp|d| d|jddd |jjd |d 7}q|S)NFrrrcSrrYrrrrYrYrZrrz,ComputeFunction.__call__..z c10::SymIntZint64_tr inline  { return at::_ops::::call(z); }zX namespace symint { template ::value>> T)Zsuppress_symint_suffixz ); } } )rrmanual_cpp_bindingrir signaturesrrrrrrr;functionvariantsdeclrr ) rTr sig_grouprresultr target_sigr exprs_strZ intlike_trYrYrZrsD          zComputeFunction.__call__Nr[r\r]rr1r rrrYrYrYrZr&r&c@sHeZdZUeejejfed<ee ed<e de de e fddZdS)ComputeTensorMethodrrrrhc Cs tj|jvrdS|jrJ|jjjdusJtj|d|j d}|j t j ur;d}| D] }||d7}q-|S|j t jurFt|j d}| D]7}t|j}t||dd}ddd|D}|d |jd |jd d d |jjd|d 7}qL|S)NTrrz const; )rrcSrrYrr'rYrYrZrrz0ComputeTensorMethod.__call__..rr(zTensor::)prefixz const { return at::_ops::r*); } )r;rr.ri is_out_fnrZself_argrrr+rrIr!r,r/r#rBrrrrr"rr rTrr0r1rr2rr3rYrYrZrs:          zComputeTensorMethod.__call__N)r[r\r]r rIr!r#rr r&rr1r rrrYrYrYrZr6s  r6c@r%)ComputeRedispatchFunctionrrhc Cstj|d|jd}d}|D]8}t|j}t||}d dgdd|D}|d|jd |j d d d |jj d |d 7}q|S)NFrrrrcSrrYrrrYrYrZrrz6ComputeRedispatchFunction.__call__..rr(T)rr)z ::redispatch(r8) rrr+r,rrrirrrr/rr r:rYrYrZrs$     z"ComputeRedispatchFunction.__call__Nr4rYrYrYrZr;r5r;cCsd|jjjd|jjjdS)Nz{"aten::z", "z"},)rirrrrYrYrZcompute_aten_opsr<gcCs|jsdSt|j@t|}t|}ddd|D}|jj}|dur*d}d}|jr3|jjnd}|r+g|j |j }dd|D}dd|D} dd d| D} d | d } d d|D} d dd| D} g}t |D]\}}d| d|dg| |dd}d|d }| |j dd}|d|jd|d}d||jd}d| |d|d}g}||dt |D]!\}}||kr|d|jdq|d|jd|jd q|d!d"|}|d#|d$|d%|d&qwd"|}ddgt| }d'|d(}d)}d*| d+|d#| d,}nd-}d-}d.|d/|d0|d1|d1|d2|d3 WdS1sOwYdS)4Nrcss|]}|VqdSr)r/rrYrYrZr!sz4compute_meta_function_declaration..zat::impl::MetaBaseZvoidcSsg|] }|D]}|qqSrYrY)rZ replace_listelemrYrYrZr-s z5compute_meta_function_declaration..cSsg|]}|jqSrY)rupperrr>rYrYrZr0 css|] }d|dVqdS)zbool z = falseNrY)rparamrYrYrZr3s  z template <>cSsg|] }tj||jdqS))binds)r argument_typerr@rYrYrZr9rcss(|]}|jddd|jVqdS)TZ strip_ref N)cpp_typerr@rYrYrZr>s  truerOzprecompute_outrz return ret;rz z { z z' } z'using meta_return_ty = precompute_out ;Zmeta_return_tyz zG struct TORCH_API precompute_out { z; };rzstruct TORCH_API structured_z : public rrz meta(z); }; )rroutmetarZmeta_argumentsrZstructured_inherits precomputedrvaluesr enumeraterIrzr)r=rargsargs_strZ parent_classZ meta_returnrMZprecomputed_valuesZprecomputed_elementsZprecomputed_template_parametersZprecomputed_template_params_strZprecompute_template_declZ#precomputed_elements_with_cpp_typesZprecomputed_elements_declZsetter_methodsir>Zreturn_ty_templatesZ return_tyZ elem_cpp_tyrZ assert_msgZ assert_stmtZconstruction_stmtsjZconstruction_blockZsetter_methods_declZmeta_return_template_paramsZmeta_return_typedefZprecomputed_declrYrYrZ!compute_meta_function_declarations             &rTr cCsBt|jjj}|ds|drdS|jjjdurdS||S)NZ_likeZnew_F)rrirendswith startswithrrr)rr rrYrYrZneeds_backend_selects  rWc@sDeZdZUeejejfed<eed<e de de e fddZ dS)ComputeBackendSelectrr rrhc Cs0t||jsdSt|j}t|jdd}dd|D}t|j}|}| }d}|j t j ur|rP|jj s.z.c10::computeDispatchKey(dtype, layout, device)rcsrrrrrYrYrZrrz0ComputeBackendSelect.__call__..z-DispatchKeySet _dk_set = c10::DispatchKeySet(z() | c10::detail::multi_dispatch_key_set(z); DispatchKeySet _dk_mask = c10::DispatchKeySet(DispatchKeySet::FULL_AFTER, DispatchKey::BackendSelect); DispatchKeySet _dk = c10::impl::computeDispatchKeySet(_dk_set, _dk_mask);z)DispatchKeySet _dk = c10::DispatchKeySet(rz // aten::z C10_ALWAYS_INLINE z { z return at::_ops::z::redispatch( _dk, csrrrrrYrYrZrrr8zm.impl("aten::z ", TORCH_FN(z));)rWr nativerrirrrrrrrIr#Zhas_tensor_argrr"r  REGISTRATIONrB) rTrrZ native_sigrZdispatcher_sigrZdispatcher_exprsrqrZ compute_dkrYrYrZrsJ        zComputeBackendSelect.__call__N)r[r\r]r rIr#r[rrArr1r rrrYrYrYrZrXs rXdatacCs>ddt_dtdtdtfdd}tt|tj|dtd d S) NcSsdS)NTrY)rTr\rYrYrZrnzformat_yaml..dumperr\rhcSs||Sr)Zrepresent_dictr|)r^r\rYrYrZdict_representersz%format_yaml..dict_representerFgeA)Zdefault_flow_styleZDumperwidth)rJZignore_aliasesrZadd_representerrrdump)r\r_rYrYrZ format_yamls  rbc Cs\|dkrdS|dkr dSzt|WSty-zt|WYSty,|YYSww)NrJTfalseF)ry ValueErrorfloatrrYrYrZpythonify_defaults    rftcCs:t|tr t|jSt|dkrdStj|ddddS)NTensorz at::TensorF__placeholder__)mutablerDr)rur5 dynamic_typer>rrZargumenttype_typerI)rgrYrYrZrks   rkr.cCs2dg}tj|vr |dtj|vr|d|S)Nr:rh namespace)r;rrzr-)r. method_ofrYrYrZcompute_method_of_yaml s     rncCsi}t|}g}tt|jj|D]3\}\}}t|j|tj|dd d}|j r@|j |d<|j r@|j ||jj j |j <||q||fS)NFrY)rkrr field_name)rZ return_namesrOziprireturnsrkr return_typerIrr9rrKrz)rname_to_field_namenamesrqrRrrretrYrYrZcompute_returns_yaml+s*    rwcpp_a schema_orderkwarg_only_set out_arg_setrscCspt|jtrddd|j|jdd}|jdur|j|d<|St|jtr&tt|jtr6t |j||||dSdS)Nzat::TensorOptionsFT) annotationrk is_nullablerr kwarg_onlyrryrzr{rs) rurr9rrrr7AssertionErrorr%compute_argument_yaml)rxryrzr{rsrrYrYrZcompute_cpp_argument_yamlns,     rrc Cs|jrt|jndt|j|j|jtj|dddd}|j dur2t tj |j |jdd|d<|j|vr;d|d<|j|vrTd|d <d|d <|j|vrT||j|d <|j }|durn|j durnt|jd krn|j |d <|S)NriF)rDr)r|rkr}rrrYrTr~rallocateroboolsize)r|rrkrr}rrrErIrrfZ default_exprZ is_list_likerr>)rryrzr{rsrlrYrYrZrs*        rc st|\}dd|jjjDdd|jjjDtj|ddd}|j}fdd|D}t|j }fdd|D}d d|D}t j |jj dd  }|d d |d } tdd|Dootj|jv} tdt |jfdt|jjjfdt|jjjfd|jfd|jdur|jndfdd|jfd|fd| fd|fdt|jfdd|jdurdn|jfd|fd|jjjjfd | fd!|jfd"|jfd#d$d%|jfgS)&NcSh|]}|jqSrYrrrYrYrZ rz+compute_declaration_yaml..cSrrYrrrYrYrZrrFrc g|] }t|ddqS)Fr)r)rrxrzrsr{rYrZrz,compute_declaration_yaml..c r)Tr)rrrrYrZrrc Ss0g|]}tj|dtddddD]}|jqqS)F)rZcpp_no_default_argsZfaithfulrZhas_tensor_options)rrrr)rrrurYrYrZrsrYz (rrcss|] }t|jtVqdSr)rurr9rrYrYrZrz+compute_declaration_yaml..rZ operator_namerrcategory_overriderZ schema_stringaten::rschema_order_cpp_signatureschema_order_argumentsrm)moderZ python_modulerqris_factory_methodZabstractrr)Zwith_gilF) deprecatedFZhas_math_kernel)rwrirZflat_kwarg_onlyrKrrrrvrr returns_typerqrIranyr;rr.rrrrrrrnrrZ is_abstractrrr) rrqr0Zcpp_argsrZschema_order_jit_argumentsrZcpp_schema_order_typesZ cpp_returnsrrrYrrZcompute_declaration_yamls^      rcCs0|js|jduo|jtjkp|jtjkSr)rrrikindr6 functionalrrrYrYrZ"has_autogenerated_composite_kernel src stj}tjj}tj}ddd|D}djtfdd| Dt j hkoDfdd| Dt j t j hktj pLtd}|d |d |d t|d S) Nrcss|] }|VqdSr)Z no_defaultZdecl_registration_declarationsrrYrYrZrrz4compute_registration_declarations..rch|] \}}|r|qSrYrrrrrrYrZr z4compute_registration_declarations..crrYrrrrYrZr"r)schemadispatchrrHrz); // r) dispatcherrrirrqZ"cpp_type_registration_declarationsrrrr|r*rrZhas_composite_kernelrjsondumps)rrarrrPrQZ comment_datarYrrZ!compute_registration_declarationss,    "r"provided_op_registration_allowlistop_selection_yaml_pathcCsh|dur |dur Jdd}|durt|}|dur#t|dd}|S|dur.t|}|St}|S)NzhBoth provided_op_registration_allowlist and op_selection_yaml_path can NOT be provided at the same time.TF)rrAZ&from_legacy_op_registration_allow_listZfrom_yaml_pathget_nop_selector)rrZop_registration_allowlistr rYrYrZget_custom_build_selector5s&   rcCsdttttftfdttttffdd}tt}|D]1}|j }|j }|tj kr@|j }|||vs9J||||<q|||vsHJ||||<qtt||S)NdrhcSsfg}tj|vr%|tj}|tjd}|tjd}|t|||d|D]}||q)|S)N)view view_copy view_inplace) r<ZaliasingrZaliasing_inplacer6rrzr3rN)rrlrrrrirYrYrZmaybe_create_view_groupWs   zEget_grouped_by_view_native_functions..maybe_create_view_group)r rr<r6r1r r3rrwriZview_signatureZview_schema_kindZ non_aliasingrrvrCrN)r`rZgrouped_by_viewsrrZ view_kindrrYrYrZ$get_grouped_by_view_native_functionsTs"    rcCsDdtttfdttttffdd}t|}tt|t| S)NrrhcSs>t|}|durtdd|DrJt|S|gS)Ncss|]}d|jvVqdS) generatedN)rrrrYrYrZrr zJget_grouped_native_functions..flatten_pre_group..)r2 from_dictrrNrv)rrurYrYrZflatten_pre_groups  z7get_grouped_native_functions..flatten_pre_group) r r6r1r rr2r@rvrCrN)r`rZpre_grouped_native_functionsrYrYrZget_grouped_native_functionss   r)native_function_decl_gengrouped_native_functionsrc Cstt}|D]B}t}t}|D]5\}}||} | r+| j} |||| nt} t|dks=Jd|d|||  |||qq|S)NrOz6Codegen only supports one namespace per operator, got z from ) rrvrr|rrrr)rextend) rrarns_grouped_kernelsrZnative_function_namespaces dispatch_keysrq backend_idxrrlrYrYrZget_ns_grouped_kernelss&   rrc Cshg}d}|D])\}}t|ddd}tt|}|d|jd||d|jd |q|S)Nrr namespace_str entity_nameZ max_level ) r|rHrvrfromkeysrprologuerepiloguesplit)r declarationsnewlinerlkernels ns_helperordered_kernelsrYrYrZ8get_native_function_declarations_from_ns_grouped_kernelss, rcCst|||d}t|dS)a Generate kernel declarations, in `NativeFunction(s).h`. :param grouped_native_functions: a sequence of `NativeFunction` or `NativeFunctionGroup`. :param backend_indices: kernel collections grouped by dispatch key. :param native_function_decl_gen: callable to generate kernel declaration for each `NativeFunction`. :return: a list of string, from the string with all declarations, grouped by namespaces, split by newline. rrar)r)rr)rrarrrYrYrZ get_native_function_declarationssrrcCs^||}|r(d|jvs(Jdt|tr|jjn|jjjd|jd|jd|r-|jStS)NrzThe kernel for function z with dispatch key z has a namespace z% and it's not ending with '::native'.) rrrur1rirrrqr))rrrrYrYrZget_kernel_namespaces  rfmrqrocmrskip_dispatcher_op_registrationgen_dispatch_helpersc sg} tttttt} d} tjtj|||dd} tjtj|||dd} tjtj|||dd}|D]A}t |d dd | | | |t |t r_|jn|jj}|| vrott| <| | ||q9D]Mtdkrq}tdd| D]}| |sqd|d d | | |d 7q| |d f d d| q}| S)Nrrrclass_method_namerrrrrr)rz TORCH_LIBRARY_IMPL(rz , m) {  };zRegisterDispatchDefinitions.inic s<jjr tngrdnddS)Nr)Z ns_prologueZ ns_epiloguedispatch_helpersZdispatch_anonymous_definitionsZ"static_init_dispatch_registrationsZdeferred_dispatch_registrationsdispatch_namespaceZdispatch_namespaced_definitions)rrdestZgen_registration_helpersrrY Zanonymous_definitionsrrqrZkernel_namespaceZns_definitionsrZregistration_bodyrrYrZrnJs z1get_native_function_definitions..)rrvrwrRegisterDispatchKeyrIZNAMESPACED_DEFINITIONZANONYMOUS_DEFINITIONr[rrrrur1rlrrrHrZsubstitute_with_templater)rrrqrr rrrr definitionsZ registrationsrZns_genZ anonymous_genZreg_genrrlrYrrZget_native_function_definitionss           rc Csg}tt}d}tj|tj||dd|d} |D]} t| |dd|} ||  | | q| D]0\} } t | dkr>q3t | ddd } tt | }| d| jd||d| jd |q3|S) NrF)rrrrrrZrrrr)rrvrrrINAMESPACED_DECLARATIONrrrrr|rrHrrrrrr)rrqrr rrrrrrirrlrrrrYrYrZget_namespaced_declarationbsN    rschema_selectorc Cstt}|D] }||j|qd}g}d}|D]1\}}ttt||} |dkr/| }q|}d} |tvr9dnd} |d| d|d| | d 7}q||fS) NrZatenrZTORCH_LIBRARY_FRAGMENTZ TORCH_LIBRARYrrz , m) { r) rrvrlrzr|rGrr+r) r`rZns_native_functionsZnative_functionschema_registrationsaten_schema_registrationscustom_namespacerlrlZschema_registrations_bodytabZtorch_library_macrorYrYrZ(get_native_function_schema_registrationss4  rstructured_native_functionsstatic_dispatch_idxcpu_fmcuda_fmfunctions_keysrc s |d fddddDfddD|d fdd|d  fd d|d  fd dtd |dfdd| D]6trU|n|} |vrdd| d dfdd| dd fdd~ qMdS)NzNativeMetaFunctions.hcsgtttdS)N)ZNativeMetaFunctions_includesZ NativeMetaFunctions_declarationsrvrGrTrY)rrYrZrns z(gen_aggregated_headers..cSsg|] }tj|jvr|qSrYr;rr.rfnrYrYrZrsz*gen_aggregated_headers..csg|]}|vr|qSrYrYr)method_native_functionsrYrZrsMethodOperators.hcsgttttjddS)NrZMethodOperators_includesZMethodOperators_declarationsrvrGrrIr!rY)rrrYrZrnsz Operators.hcs dgttttjddS)Nz!#include r)ZOperators_includesZOperators_declarationsrrY)non_method_native_functionsrrYrZrns Functions.hcstdgtttdS)Nz#include )rZFunctions_includesZFunctions_declarations)rrvrGr&rYr`rrYrZrns)rrazNativeFunctions.hcs dgdS)Nz%#include )ZNativeFunctions_includesZNativeFunctions_declarationsrYrY)rrYrZrn srFunctions_inl.h>DispatchKeyFunctions.hctdSN)rqZinline_headersrrYrq inl_headersrYrZrnFunctions_inl.hDispatchKeyFunctions_inl.hc s$gtdddS)NT)rrqrr rr)!DispatchKeyFunctions_inl_includesr dispatch_namespaced_declarations)rrrY)rarqrrr rYrZrn s)writerr-write_with_template) r`rrrr rarrrrrrrY) rarrqrrrr`rrr rrrZgen_aggregated_headerss\         rops_fmc  sNtt|D] } | j| qtt} |D] } | j | | qD]^\ | dd fdd| dd fdd|  g}dd |D t d krn| d d fd dt||t j d| dd fddq(dD]\ | d fddq| D]{|vrq gD]<\ |  g}tt t j|tj|| dddd|td krq | dddfddqtr|n|}dd|ddfdd|d d!fd"d~q| d#fd$ddS)%Nz_ops.hz Operator.hcsdttttjdiS)NrrrrY) functionsrrYrZrnPsz*gen_per_operator_headers..z.hz Function.hcs2ttfddddtttdS)Ncs t|dS)Nr)rrrrYrZrncsz...r_ops.h>)Zstatic_dispatch_ops_headersZoperator_includesZfunction_definitions)rvrGr&rY)rrrrYrZrn`s  cSs g|] }t|tr|jr|qSrY)rur2rrrYrYrZrtsz,gen_per_operator_headers..rz_meta.hzNativeMetaFunction.hcdtttiS)NZmeta_function_declarationsrrY)structured_functionsrYrZrns rz _native.hzNativeFunction.hcs r dddSgdS)Nrz_meta.h>)Zextra_includesZnative_function_declarationsrYrY)r is_structuredrrYrZrns ))Z Functionsr)Z OperatorsZ_ops)ZNativeMetaFunctions_meta)ZNativeFunctions_nativecs.dfddtDdgiS)NZ _includescsg|] }d|dqS)r.h>rYrr)suffixrYrZrrF>gen_per_operator_headers....Z _declarations)rrrY)categoryfunctions_by_root_namerrYrZrns   TFrrz _dispatch.hzDispatchKeyFunction.hcs dS)N)rrrYrY)rrrYrZrnsrrrrcrrrrYrrYrZrnrrrcsfddtDgdS)Ncsg|] }d|ddqS)rrrrYr)rrYrZrsr)rrr)rrY)dispatch_namesrrYrZrns  rcstddDgdS)Ncss2|]\}}tdd|Drd|dVqdS)css|] }tj|jvVqdSrrrrYrYrZrrzGgen_per_operator_headers.....rr N)r)rrrrYrYrZrs  z=gen_per_operator_headers....r)rr|rY)rrYrZrns )rrvrrzr|rrxrrr#compute_native_function_declarationrrrCrrIrr-)r`rrr rarrrrrrrZgrouped_functions_by_root_namegroupZgrouped_functionsrrY) rrrqrrrrrr rrr rrZgen_per_operator_headers1s                 rcore_fmper_operator_headersc s|rt|||| | | | | d nt||||| | | | d |dfdd|dfdd|dfd d|d fd dd tttfffd d }|d|d tttfffdd }|d|dS)N) r`rrr rarrrrrr) r`rrrr rarrrrrz TensorBody.hcs2ttttjdttttjddS)N)rr)Ztensor_method_declarationsZtensor_method_definitions)rvrGr6rIr!r#rYrrYrZrn$s& zgen_headers..zRedispatchFunctions.hcsdtttiS)NZfunction_redispatch_definitions)rvrGr;rYr`rYrZrn<s zRegistrationDeclarations.hcsdfddDiS)NZregistration_declarationscsg|]}t|qSrY)rrrrYrZrFz1gen_headers....rYrY)rar`rYrZrnEs zVmapGeneratedPlumbing.hcstSrr#rYrrYrZrnNsrhcst}t}D]#}|t|jjj||jjjj|jD]}||jq"q|hd8}dddt|Ddddt|DdS)N> orZand_eqZbitorZxor_eqandnotZcomplZbitandZnot_eqZor_eqxorz \ cSr)z_(aten, rrYrrYrYrZrozBgen_headers..gen_aten_interned_strings..cSr)z_(attr, rrYrrYrYrZrrr")Z aten_symbolsZ attr_symbols) rrrrirrrrr)attrsrtrirrrYrZgen_aten_interned_stringsQs  z.gen_headers..gen_aten_interned_stringszaten_interned_strings.hcsddtiS)NZenum_of_valid_tagsz, )rrrY)rdrYrZ gen_tags_enumxsz"gen_headers..gen_tags_enumz enum_tag.h)rrrr r)r`rdrrrr rarrrrrrrrr$r%rY)rar`rrdrZ gen_headerss\      %r& view_groups cpu_vec_fmforce_schema_registrationc sdrd| D]݉tr| n|}r$dttf fdd n dttffdd tt} D]}t|trD|jn|jj}|| |q:t t j k}t | d|d |d d d  fd d D]d jjrtsq jjjj t jur||usJ|d d d fdd | d d d fdd qt jurdrd|d dd fdd qtdd~qdttttff fdd }|d|}|rt}t |d\|d fd!d d"ttttfdtfd#d$}|jd% |fd&d d't id(hd)d*|d+d,d |d-d.d |d/ fd0d d1ttttfdttttfffd2d3 }tt}d4d5t!d6d D}d7d5t!d8d D} D]}|jj|vr|jj|vr| |q|jd9|||d:hd;d<|d=fd>d |d?fd@d dS)ANz#include #include #include #include z#include #include #include #include rhcsg}D][}d}|rd}n"t|tr%tfdd|Dr%d}n |jr2tjtjfvr2d}|s5q| d|j dtjkrN| d|j dvr_| d|j dd qt t |S) NFTc3s|]}|VqdSrrrrrYrZrs  z=gen_source_files..operator_headers..r _native.h>rrr) rrur2rrrr*Metarrzrrr)headersr=Z is_registered)rrqrrrrYrZoperator_headerss6  z*gen_source_files..operator_headerscs8dg}tjkr |dvr|dd|S)Nz!#include z#include rr)r*rrz)r,)rqrrYrZr-s   T) rrrqrr rrrrZRegisterz.cppzRegisterDispatchKey.cppcs*trnddtddS)Nr)extra_cuda_headersZexternal_backend_headersZdispatch_headers ops_headersrdispatch_definitions)r-rZgen_registration_headersrY)rr0rqr.r-rrrYrZrnsz"gen_source_files..Z UfuncCPU_z UfuncCPU.cppcs"tttdS)N)meta_declarationnative_declarationnative_definitions)rTrrZcompute_ufunc_cpurY)rarqr=rYrZrn s ZUfuncCPUKernel_zUfuncCPUKernel.cppcstdS)N)rr3)rZcompute_ufunc_cpu_kernelrY)r=rrYrZrn sz%#include z$#include Z UfuncCUDA_z.cuz UfuncCUDA.cucs&tttdS)N)r cuda_headersr1r2r3)rTrrZcompute_ufunc_cudarY)rar4rqr=rrYrZrn# sz unrecognized z for ufunccsLfddD}dd|Dttttj|ttttj|dS)Ncsg|] }t|r|qSrY)rWrr rYrZr4  z@gen_source_files..gen_backend_select..cSsg|] }d|jdqS)rr rrrYrYrZr8 s)r/Z!backend_select_method_definitionsZ%backend_select_function_registrations)rvrGrXrIr#r[)Z relevant_fns)r`r rYrZgen_backend_select3 s"   z,gen_source_files..gen_backend_selectzRegisterBackendSelect.cpp)r`rzRegisterSchema.cppcsrgnr gdSdS)N)rrrYrY)rrrrYrZrnU srcSs|jSrr7rrYrYrZkey_func_ sz"gen_source_files..key_funcz Operators.cppcs&d|jdgttjd|gdS)Nrrr)r-r)rrrIr#rr rYrZrnh sr>rr-r)key_fn env_callableZbase_env num_shards sharded_keysz Functions.cppcSiSrrYrYrYrYrZrn~ r]zTensorMethods.cppcSr?rrYrYrYrYrZrn r]zATenOpList.cppcr )NZaten_ops)rvrGr<rYrrYrZrn sr=csDdttttfdttfdd}||t|t|tj dS)Nr=rhcSst|tr.d|jjdd|jjdg}|jdur,|d|jjdd|jjdg7}|St|trd|jjdd|jjdd|jjdd|jjdg}|jdurh|d|jjdd|jjdg7}|j dur|d|j jdd|j jdg7}|Sd|jdd|jdgS)Nrr*r ) rur3rrrr2rrKrrj)r=r,rYrYrZgen_op_headers s<       zPgen_source_files..functionalization_env_callable..gen_op_headers)r/func_definitionsfunc_registrations) rr1r2r3r rrr r*r)r=r@)rar rYrZfunctionalization_env_callable s   (z8gen_source_files..functionalization_env_callablecSi|]}|jj|qSrYrirrrYrYrZ rz$gen_source_files..cS t|Srrvrr=rYrYrZrn  cSrDrYrErrYrYrZrF rAcSrGrrHrIrYrYrZrn rJzRegisterFunctionalization.cppr>Z!func_add_back_views_registrationsr/rArBZfunc_add_back_views_definitions)r;r<r=r>zFunctionalInverses.hcsdttfddiS)NZview_inverse_declarationscs t|Sr)r!rIr5rYrZrn sz4gen_source_files....)rvrGrY)r r'rYrZrn s zCompositeViewCopyKernels.cppcsNddDddDttttjttttttdS)NcSs:g|]}ddd|jdur|jgn|j|jgDqS)rcss&|]}d|jd|jdVqdS)rz_ops.h> #include ....N)rrrrr=rYrYrZr s  z6gen_source_files....cSs,g|]}ddd|j|j|jfDqS)rcss0|]}|durd|jvrd|jdVqdS)Nrrr )rrrrYrYrZr s rK)rrrjrrLrYrYrZr s  )r/Z#CompositeViewCopyKernel_DefinitionsZ(GeneratedCompositeFunctional_DefinitionsZ!GeneratedCompositeOut_Definitions)rvrGr"r*rr>r?rY)rarr'rYrZrn s<   )"r-r rrrvrur1rlrrzrr*rrrrKZufunc_inner_loopr/rirCPUCUDArr rrArrrr2r3Z write_shardedrrC)r`rrr'r rrarrr(rrrrr)rrrZns_grouped_native_functionsZgrouped_native_functionrlrr8rr9rCZ all_groupsZstructured_mapZview_maprrY)rrrar4r0rqrr.rr=rrr`r-rrrr rrrr'rZgen_source_files~s *                  9    rOcs|dfdddS)NzDeclarations.yamlcstddDS)NcSsg|]}t|qSrY)rrrYrYrZr9 sz;gen_declarations_yaml....)rbrYrrYrZrn9 rz'gen_declarations_yaml..)r)rr`rYrrZgen_declarations_yaml4 s rPcCsttjS)z If you're depending on torchgen out-of-tree, you can use the root to figure out the path to native_functions.yaml )pathlibPath__file__parentresolverYrYrYrZget_torchgen_root= srVcs,tjdd}|jddddd|jdd d d |jd d dd|jdd dd|jdddddd|jdd dd|jdd dd|jddddd|jd d!d"d |jd#d$dd%d|jd&d'dd(d|jd)d*d d+d|jd,d-d d.d|jd/tdgd0gd0d1d2|tjj}tj j d3}tj j d4}d5d6l m }t}js|tjtj|vr||tj=t|||}t|}|j|j}t|} d7d8| D} t|} d9d8| D} jd:} t| jd;d;d<jd=}t|jd;d;d<t| d>}td?}td?}td?}t|d>}tjtj tj!tj"tj#tj$tj%h}jr:|tjj&rGfd@d8|D}g}j'rmfdAd8j'D}j'D]}t(|}||vrk||qZdBj)vrt*d]idC|dD| dE| dF| dG|dH|dIdJ|dK|dL|dM|dN|dO|dPj+dQj,dRj-dSj.dTj)vrt/||| | ||||||||j+j-dUdVj)vrt0||dWj1rtj12}|j3}|j4}|dXf|dYf|dZf|d[f|d\ffD]\}}||}|j5||}|6|t|qdSdS)^NzGenerate ATen source files) descriptionz-sz --source-pathz!path to source directory for ATenz aten/src/ATen)helprz-oz--output-dependenciesz:output a list of dependencies into the given file and exit)rXz --dry-run store_truez5run without writing any files (still updates outputs))actionrXz--per-operator-headersz2generate separate headers per operator in ATen/opsz-dz --install-dirz --install_dirzoutput directoryzbuild/aten/src/ATenz--rocmz=reinterpret CUDA as ROCm/HIP and adjust filepaths accordinglyz--mpsz'Generate MPS registration code when setz--op-registration-whitelistz--op_registration_whitelist*zfilter op registrations by the whitelist (if set); each item is `namespace`::`operator name` without overload name; e.g.: aten::empty aten::conv2d ...)nargsrXz--op-selection-yaml-pathz--op_selection_yaml_pathaDProvide a path to the operator selection (for custom build) YAML that contains the information about the set of selected operators and their categories (training, ...). Each operator is either a full operator name with overload or just a bare operator name. The operator names also contain the namespace prefix (e.g. aten::)z--backend-whitelistz--backend_whitelistzRfilter dispatch backend by the whitelist (if set), e.g.: CPU CUDA QuantizedCPU ...z--static-dispatch-backendz--static_dispatch_backendz?generate static dispatch code for the specific backend (if set)z!--skip-dispatcher-op-registrationz!--skip_dispatcher_op_registrationz0Avoid registering operators into the dispatcher.z--force-schema-registrationz--force_schema_registrationz}force it to generate schema-only registrations for all ops, includingthose that are not listed on --op-registration-whitelistz --generate)r,sourcesdeclarations_yamlzGenerate only a subset of files)rr\choicesrrXznative/native_functions.yamlznative/tags.yamlr)rcSg|] }t|tr|qSrY)rur2rLrYrYrZr r6zmain..cSr`rY)rur3rLrYrYrZr s z/coreT)parentsexist_okz/ops)options install_dirrccs&g|]}t|st|jvr|qSrY)r.rbackend_whitelist)rrrerYrZr scsg|] }t|qSrY)r*parse)rkeyrrYrZr s r]r`rrr'r rrarrr(rrrrr)rrr,)r`rdrrrr rarrrrrrrrr^)r`rrZcpu_vec_Zcore_Zcuda_Zops_rY)7argparseArgumentParser add_argumentr parse_argsrZop_registration_whitelistrosrfr source_pathtorchgen.modelrrZmpsrr*ZMPSrsrrr`rarrrdrQrRmkdirrFrMrNrrrrr+rfZstatic_dispatch_backendrggeneraterOrr)rrr&rPZoutput_dependenciesrUrstemrTZ write_outputs)parserr Znative_yaml_pathrrreZ parsed_yamlrdr`rrZ!native_functions_with_view_groupsr'Zcore_install_dirZops_install_dirrrr(rrrrrhZdp_keyZ depfile_pathZ depfile_nameZ depfile_stemrr7varnamerfrY)rarcrZmainE s                               ru__main__)NrbF)rbr)rhN)ri functoolsrrmrQ collectionsrrr dataclassesrrtypingrrr r r r r rrrrrZtorchgen.api.dispatcherapirZtorchgen.api.metarLZtorchgen.api.nativerZZtorchgen.api.structuredrZ torchgen.destrZ torchgen.apirZtorchgen.api.translaterZtorchgen.api.typesrrrrrrrZtorchgen.contextrrrrZ#torchgen.gen_functionalization_typerr r!r"Ztorchgen.gen_vmap_plumbingr$ror%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<Z#torchgen.native_function_generationr=r>r?r@Z!torchgen.selective_build.selectorrAZtorchgen.utilsrBrCrDrErFrGrHrIZtorchgen.yaml_utilsrJrKrLrMrrr_objectrrrr lru_cacherrr{rrrrrrrrrrrr&r6r;r<rTrWrXrbrfrkrnrwrrrrrrrrrrrrrrrrrrr&rOrPrRrVrur[rYrYrYrZs<4   $  h ( !  .     '     #      L`/-| D   C     $Y  "   .    "       g 2 $      y      C               9   z