1 //! This module contains the bulk of the interesting code performing the translation between 2 //! WebAssembly and Cranelift IR. 3 //! 4 //! The translation is done in one pass, opcode by opcode. Two main data structures are used during 5 //! code translations: the value stack and the control stack. The value stack mimics the execution 6 //! of the WebAssembly stack machine: each instruction result is pushed onto the stack and 7 //! instruction arguments are popped off the stack. Similarly, when encountering a control flow 8 //! block, it is pushed onto the control stack and popped off when encountering the corresponding 9 //! `End`. 10 //! 11 //! Another data structure, the translation state, records information concerning unreachable code 12 //! status and about if inserting a return at the end of the function is necessary. 13 //! 14 //! Some of the WebAssembly instructions need information about the environment for which they 15 //! are being translated: 16 //! 17 //! - the loads and stores need the memory base address; 18 //! - the `get_global` and `set_global` instructions depend on how the globals are implemented; 19 //! - `memory.size` and `memory.grow` are runtime functions; 20 //! - `call_indirect` has to translate the function index into the address of where this 21 //! is; 22 //! 23 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as 24 //! argument. 25 //! 26 //! There is extra complexity associated with translation of 128-bit SIMD instructions. 27 //! Wasm only considers there to be a single 128-bit vector type. But CLIF's type system 28 //! distinguishes different lane configurations, so considers 8X16, 16X8, 32X4 and 64X2 to be 29 //! different types. The result is that, in wasm, it's perfectly OK to take the output of (eg) 30 //! an `add.16x8` and use that as an operand of a `sub.32x4`, without using any cast. But when 31 //! translated into CLIF, that will cause a verifier error due to the apparent type mismatch. 32 //! 33 //! This file works around that problem by liberally inserting `bitcast` instructions in many 34 //! places -- mostly, before the use of vector values, either as arguments to CLIF instructions 35 //! or as block actual parameters. These are no-op casts which nevertheless have different 36 //! input and output types, and are used (mostly) to "convert" 16X8, 32X4 and 64X2-typed vectors 37 //! to the "canonical" type, 8X16. Hence the functions `optionally_bitcast_vector`, 38 //! `bitcast_arguments`, `pop*_with_bitcast`, `canonicalise_then_jump`, 39 //! `canonicalise_then_br{z,nz}`, `is_non_canonical_v128` and `canonicalise_v128_values`. 40 //! Note that the `bitcast*` functions are occasionally used to convert to some type other than 41 //! 8X16, but the `canonicalise*` functions always convert to type 8X16. 42 //! 43 //! Be careful when adding support for new vector instructions. And when adding new jumps, even 44 //! if they are apparently don't have any connection to vectors. Never generate any kind of 45 //! (inter-block) jump directly. Instead use `canonicalise_then_jump` and 46 //! `canonicalise_then_br{z,nz}`. 47 //! 48 //! The use of bitcasts is ugly and inefficient, but currently unavoidable: 49 //! 50 //! * they make the logic in this file fragile: miss out a bitcast for any reason, and there is 51 //! the risk of the system failing in the verifier. At least for debug builds. 52 //! 53 //! * in the new backends, they potentially interfere with pattern matching on CLIF -- the 54 //! patterns need to take into account the presence of bitcast nodes. 55 //! 56 //! * in the new backends, they get translated into machine-level vector-register-copy 57 //! instructions, none of which are actually necessary. We then depend on the register 58 //! allocator to coalesce them all out. 59 //! 60 //! * they increase the total number of CLIF nodes that have to be processed, hence slowing down 61 //! the compilation pipeline. Also, the extra coalescing work generates a slowdown. 62 //! 63 //! A better solution which would avoid all four problems would be to remove the 8X16, 16X8, 64 //! 32X4 and 64X2 types from CLIF and instead have a single V128 type. 65 //! 66 //! For further background see also: 67 //! <https://github.com/bytecodealliance/wasmtime/issues/1147> 68 //! ("Too many raw_bitcasts in SIMD code") 69 //! <https://github.com/bytecodealliance/cranelift/pull/1251> 70 //! ("Add X128 type to represent WebAssembly's V128 type") 71 //! <https://github.com/bytecodealliance/cranelift/pull/1236> 72 //! ("Relax verification to allow I8X16 to act as a default vector type") 73 74 use crate::Reachability; 75 use crate::bounds_checks::{BoundsCheck, bounds_check_and_compute_addr}; 76 use crate::func_environ::{Extension, FuncEnvironment}; 77 use crate::translate::TargetEnvironment; 78 use crate::translate::environ::StructFieldsVec; 79 use crate::translate::stack::{ControlStackFrame, ElseData}; 80 use crate::translate::translation_utils::{ 81 block_with_params, blocktype_params_results, f32_translation, f64_translation, 82 }; 83 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC}; 84 use cranelift_codegen::ir::immediates::Offset32; 85 use cranelift_codegen::ir::{ 86 self, AtomicRmwOp, ExceptionTag, InstBuilder, JumpTableData, MemFlags, Value, ValueLabel, 87 }; 88 use cranelift_codegen::ir::{BlockArg, types::*}; 89 use cranelift_codegen::packed_option::ReservedValue; 90 use cranelift_frontend::{FunctionBuilder, Variable}; 91 use itertools::Itertools; 92 use smallvec::{SmallVec, ToSmallVec}; 93 use std::collections::{HashMap, hash_map}; 94 use std::vec::Vec; 95 use wasmparser::{FuncValidator, MemArg, Operator, WasmModuleResources}; 96 use wasmtime_environ::{ 97 DataIndex, ElemIndex, FuncIndex, GlobalIndex, MemoryIndex, TableIndex, TagIndex, TypeConvert, 98 TypeIndex, WasmHeapType, WasmRefType, WasmResult, WasmValType, wasm_unsupported, 99 }; 100 101 /// Given a `Reachability<T>`, unwrap the inner `T` or, when unreachable, set 102 /// `state.reachable = false` and return. 103 /// 104 /// Used in combination with calling `prepare_addr` and `prepare_atomic_addr` 105 /// when we can statically determine that a Wasm access will unconditionally 106 /// trap. 107 macro_rules! unwrap_or_return_unreachable_state { 108 ($environ:ident, $value:expr) => { 109 match $value { 110 Reachability::Reachable(x) => x, 111 Reachability::Unreachable => { 112 $environ.stacks.reachable = false; 113 return Ok(()); 114 } 115 } 116 }; 117 } 118 119 /// Translates wasm operators into Cranelift IR instructions. 120 pub fn translate_operator( 121 validator: &mut FuncValidator<impl WasmModuleResources>, 122 op: &Operator, 123 operand_types: Option<&[WasmValType]>, 124 builder: &mut FunctionBuilder, 125 environ: &mut FuncEnvironment<'_>, 126 ) -> WasmResult<()> { 127 log::trace!("Translating Wasm opcode: {op:?}"); 128 129 if !environ.is_reachable() { 130 translate_unreachable_operator(validator, &op, builder, environ)?; 131 return Ok(()); 132 } 133 134 // Given that we believe the current block is reachable, the FunctionBuilder ought to agree. 135 debug_assert!(!builder.is_unreachable()); 136 let srcloc = builder.srcloc(); 137 138 let operand_types = operand_types.unwrap_or_else(|| { 139 panic!("should always have operand types available for valid, reachable ops; op = {op:?}") 140 }); 141 142 // This big match treats all Wasm code operators. 143 match op { 144 /********************************** Locals **************************************** 145 * `get_local` and `set_local` are treated as non-SSA variables and will completely 146 * disappear in the Cranelift Code 147 ***********************************************************************************/ 148 Operator::LocalGet { local_index } => { 149 let val = builder.use_var(Variable::from_u32(*local_index)); 150 environ.stacks.push1(val); 151 let label = ValueLabel::from_u32(*local_index); 152 builder.set_val_label(val, label); 153 } 154 Operator::LocalSet { local_index } => { 155 let mut val = environ.stacks.pop1(); 156 157 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 158 let ty = builder.func.dfg.value_type(val); 159 if ty.is_vector() { 160 val = optionally_bitcast_vector(val, I8X16, builder); 161 } 162 163 builder.def_var(Variable::from_u32(*local_index), val); 164 let label = ValueLabel::from_u32(*local_index); 165 builder.set_val_label(val, label); 166 environ.state_slot_local_set(builder, *local_index, val); 167 } 168 Operator::LocalTee { local_index } => { 169 let mut val = environ.stacks.peek1(); 170 171 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 172 let ty = builder.func.dfg.value_type(val); 173 if ty.is_vector() { 174 val = optionally_bitcast_vector(val, I8X16, builder); 175 } 176 177 builder.def_var(Variable::from_u32(*local_index), val); 178 let label = ValueLabel::from_u32(*local_index); 179 builder.set_val_label(val, label); 180 environ.state_slot_local_set(builder, *local_index, val); 181 } 182 /********************************** Globals **************************************** 183 * `get_global` and `set_global` are handled by the environment. 184 ***********************************************************************************/ 185 Operator::GlobalGet { global_index } => { 186 let global_index = GlobalIndex::from_u32(*global_index); 187 let val = environ.translate_global_get(builder, global_index)?; 188 environ.stacks.push1(val); 189 } 190 Operator::GlobalSet { global_index } => { 191 let global_index = GlobalIndex::from_u32(*global_index); 192 let mut val = environ.stacks.pop1(); 193 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 194 if builder.func.dfg.value_type(val).is_vector() { 195 val = optionally_bitcast_vector(val, I8X16, builder); 196 } 197 environ.translate_global_set(builder, global_index, val)?; 198 } 199 /********************************* Stack misc *************************************** 200 * `drop`, `nop`, `unreachable` and `select`. 201 ***********************************************************************************/ 202 Operator::Drop => { 203 environ.stacks.pop1(); 204 } 205 Operator::Select => { 206 let (mut arg1, mut arg2, cond) = environ.stacks.pop3(); 207 if builder.func.dfg.value_type(arg1).is_vector() { 208 arg1 = optionally_bitcast_vector(arg1, I8X16, builder); 209 } 210 if builder.func.dfg.value_type(arg2).is_vector() { 211 arg2 = optionally_bitcast_vector(arg2, I8X16, builder); 212 } 213 environ.stacks.push1(builder.ins().select(cond, arg1, arg2)); 214 } 215 Operator::TypedSelect { ty: _ } => { 216 // We ignore the explicit type parameter as it is only needed for 217 // validation, which we require to have been performed before 218 // translation. 219 let (mut arg1, mut arg2, cond) = environ.stacks.pop3(); 220 if builder.func.dfg.value_type(arg1).is_vector() { 221 arg1 = optionally_bitcast_vector(arg1, I8X16, builder); 222 } 223 if builder.func.dfg.value_type(arg2).is_vector() { 224 arg2 = optionally_bitcast_vector(arg2, I8X16, builder); 225 } 226 environ.stacks.push1(builder.ins().select(cond, arg1, arg2)); 227 } 228 Operator::Nop => { 229 // We do nothing 230 } 231 Operator::Unreachable => { 232 environ.trap(builder, crate::TRAP_UNREACHABLE); 233 environ.stacks.reachable = false; 234 } 235 /***************************** Control flow blocks ********************************** 236 * When starting a control flow block, we create a new `Block` that will hold the code 237 * after the block, and we push a frame on the control stack. Depending on the type 238 * of block, we create a new `Block` for the body of the block with an associated 239 * jump instruction. 240 * 241 * The `End` instruction pops the last control frame from the control stack, seals 242 * the destination block (since `br` instructions targeting it only appear inside the 243 * block and have already been translated) and modify the value stack to use the 244 * possible `Block`'s arguments values. 245 ***********************************************************************************/ 246 Operator::Block { blockty } => { 247 let (params, results) = blocktype_params_results(validator, *blockty)?; 248 let next = block_with_params(builder, results.clone(), environ)?; 249 environ.stacks.push_block(next, params.len(), results.len()); 250 } 251 Operator::Loop { blockty } => { 252 let (params, results) = blocktype_params_results(validator, *blockty)?; 253 let loop_body = block_with_params(builder, params.clone(), environ)?; 254 let next = block_with_params(builder, results.clone(), environ)?; 255 canonicalise_then_jump(builder, loop_body, environ.stacks.peekn(params.len())); 256 environ 257 .stacks 258 .push_loop(loop_body, next, params.len(), results.len()); 259 260 // Pop the initial `Block` actuals and replace them with the `Block`'s 261 // params since control flow joins at the top of the loop. 262 environ.stacks.popn(params.len()); 263 environ 264 .stacks 265 .stack 266 .extend_from_slice(builder.block_params(loop_body)); 267 268 builder.switch_to_block(loop_body); 269 environ.translate_loop_header(builder)?; 270 } 271 Operator::If { blockty } => { 272 let val = environ.stacks.pop1(); 273 274 let next_block = builder.create_block(); 275 let (params, results) = blocktype_params_results(validator, *blockty)?; 276 let (destination, else_data) = if params.clone().eq(results.clone()) { 277 // It is possible there is no `else` block, so we will only 278 // allocate a block for it if/when we find the `else`. For now, 279 // we if the condition isn't true, then we jump directly to the 280 // destination block following the whole `if...end`. If we do end 281 // up discovering an `else`, then we will allocate a block for it 282 // and go back and patch the jump. 283 let destination = block_with_params(builder, results.clone(), environ)?; 284 let branch_inst = canonicalise_brif( 285 builder, 286 val, 287 next_block, 288 &[], 289 destination, 290 environ.stacks.peekn(params.len()), 291 ); 292 ( 293 destination, 294 ElseData::NoElse { 295 branch_inst, 296 placeholder: destination, 297 }, 298 ) 299 } else { 300 // The `if` type signature is not valid without an `else` block, 301 // so we eagerly allocate the `else` block here. 302 let destination = block_with_params(builder, results.clone(), environ)?; 303 let else_block = block_with_params(builder, params.clone(), environ)?; 304 canonicalise_brif( 305 builder, 306 val, 307 next_block, 308 &[], 309 else_block, 310 environ.stacks.peekn(params.len()), 311 ); 312 builder.seal_block(else_block); 313 (destination, ElseData::WithElse { else_block }) 314 }; 315 316 builder.seal_block(next_block); // Only predecessor is the current block. 317 builder.switch_to_block(next_block); 318 319 // Here we append an argument to a Block targeted by an argumentless jump instruction 320 // But in fact there are two cases: 321 // - either the If does not have a Else clause, in that case ty = EmptyBlock 322 // and we add nothing; 323 // - either the If have an Else clause, in that case the destination of this jump 324 // instruction will be changed later when we translate the Else operator. 325 environ.stacks.push_if( 326 destination, 327 else_data, 328 params.len(), 329 results.len(), 330 *blockty, 331 ); 332 } 333 Operator::Else => { 334 let i = environ.stacks.control_stack.len() - 1; 335 let reachable = environ.is_reachable(); 336 match environ.stacks.control_stack[i] { 337 ControlStackFrame::If { 338 ref else_data, 339 head_is_reachable, 340 ref mut consequent_ends_reachable, 341 num_return_values, 342 blocktype, 343 destination, 344 .. 345 } => { 346 // We finished the consequent, so record its final 347 // reachability state. 348 debug_assert!(consequent_ends_reachable.is_none()); 349 *consequent_ends_reachable = Some(reachable); 350 351 if head_is_reachable { 352 // We have a branch from the head of the `if` to the `else`. 353 environ.stacks.reachable = true; 354 355 // Ensure we have a block for the `else` block (it may have 356 // already been pre-allocated, see `ElseData` for details). 357 let else_block = match *else_data { 358 ElseData::NoElse { 359 branch_inst, 360 placeholder, 361 } => { 362 let (params, _results) = 363 blocktype_params_results(validator, blocktype)?; 364 debug_assert_eq!(params.len(), num_return_values); 365 let else_block = 366 block_with_params(builder, params.clone(), environ)?; 367 canonicalise_then_jump( 368 builder, 369 destination, 370 environ.stacks.peekn(params.len()), 371 ); 372 environ.stacks.popn(params.len()); 373 374 builder.change_jump_destination( 375 branch_inst, 376 placeholder, 377 else_block, 378 ); 379 builder.seal_block(else_block); 380 else_block 381 } 382 ElseData::WithElse { else_block } => { 383 canonicalise_then_jump( 384 builder, 385 destination, 386 environ.stacks.peekn(num_return_values), 387 ); 388 environ.stacks.popn(num_return_values); 389 else_block 390 } 391 }; 392 393 // You might be expecting that we push the parameters for this 394 // `else` block here, something like this: 395 // 396 // state.pushn(&control_stack_frame.params); 397 // 398 // We don't do that because they are already on the top of the stack 399 // for us: we pushed the parameters twice when we saw the initial 400 // `if` so that we wouldn't have to save the parameters in the 401 // `ControlStackFrame` as another `Vec` allocation. 402 403 builder.switch_to_block(else_block); 404 405 // We don't bother updating the control frame's `ElseData` 406 // to `WithElse` because nothing else will read it. 407 } 408 } 409 _ => unreachable!(), 410 } 411 } 412 Operator::End => { 413 let frame = environ.stacks.control_stack.pop().unwrap(); 414 let next_block = frame.following_code(); 415 let return_count = frame.num_return_values(); 416 let return_args = environ.stacks.peekn_mut(return_count); 417 418 canonicalise_then_jump(builder, next_block, return_args); 419 // You might expect that if we just finished an `if` block that 420 // didn't have a corresponding `else` block, then we would clean 421 // up our duplicate set of parameters that we pushed earlier 422 // right here. However, we don't have to explicitly do that, 423 // since we truncate the stack back to the original height 424 // below. 425 426 builder.switch_to_block(next_block); 427 builder.seal_block(next_block); 428 429 // If it is a loop we also have to seal the body loop block 430 if let ControlStackFrame::Loop { header, .. } = frame { 431 builder.seal_block(header) 432 } 433 434 frame.restore_catch_handlers(&mut environ.stacks.handlers, builder); 435 436 frame.truncate_value_stack_to_original_size( 437 &mut environ.stacks.stack, 438 &mut environ.stacks.stack_shape, 439 ); 440 environ 441 .stacks 442 .stack 443 .extend_from_slice(builder.block_params(next_block)); 444 } 445 /**************************** Branch instructions ********************************* 446 * The branch instructions all have as arguments a target nesting level, which 447 * corresponds to how many control stack frames do we have to pop to get the 448 * destination `Block`. 449 * 450 * Once the destination `Block` is found, we sometimes have to declare a certain depth 451 * of the stack unreachable, because some branch instructions are terminator. 452 * 453 * The `br_table` case is much more complicated because Cranelift's `br_table` instruction 454 * does not support jump arguments like all the other branch instructions. That is why, in 455 * the case where we would use jump arguments for every other branch instruction, we 456 * need to split the critical edges leaving the `br_tables` by creating one `Block` per 457 * table destination; the `br_table` will point to these newly created `Blocks` and these 458 * `Block`s contain only a jump instruction pointing to the final destination, this time with 459 * jump arguments. 460 * 461 * This system is also implemented in Cranelift's SSA construction algorithm, because 462 * `use_var` located in a destination `Block` of a `br_table` might trigger the addition 463 * of jump arguments in each predecessor branch instruction, one of which might be a 464 * `br_table`. 465 ***********************************************************************************/ 466 Operator::Br { relative_depth } => { 467 let i = environ.stacks.control_stack.len() - 1 - (*relative_depth as usize); 468 let (return_count, br_destination) = { 469 let frame = &mut environ.stacks.control_stack[i]; 470 // We signal that all the code that follows until the next End is unreachable 471 frame.set_branched_to_exit(); 472 let return_count = if frame.is_loop() { 473 frame.num_param_values() 474 } else { 475 frame.num_return_values() 476 }; 477 (return_count, frame.br_destination()) 478 }; 479 let destination_args = environ.stacks.peekn_mut(return_count); 480 canonicalise_then_jump(builder, br_destination, destination_args); 481 environ.stacks.popn(return_count); 482 environ.stacks.reachable = false; 483 } 484 Operator::BrIf { relative_depth } => translate_br_if(*relative_depth, builder, environ), 485 Operator::BrTable { targets } => { 486 let default = targets.default(); 487 let mut min_depth = default; 488 for depth in targets.targets() { 489 let depth = depth?; 490 if depth < min_depth { 491 min_depth = depth; 492 } 493 } 494 let jump_args_count = { 495 let i = environ.stacks.control_stack.len() - 1 - (min_depth as usize); 496 let min_depth_frame = &environ.stacks.control_stack[i]; 497 if min_depth_frame.is_loop() { 498 min_depth_frame.num_param_values() 499 } else { 500 min_depth_frame.num_return_values() 501 } 502 }; 503 let val = environ.stacks.pop1(); 504 let mut data = Vec::with_capacity(targets.len() as usize); 505 if jump_args_count == 0 { 506 // No jump arguments 507 for depth in targets.targets() { 508 let depth = depth?; 509 let block = { 510 let i = environ.stacks.control_stack.len() - 1 - (depth as usize); 511 let frame = &mut environ.stacks.control_stack[i]; 512 frame.set_branched_to_exit(); 513 frame.br_destination() 514 }; 515 data.push(builder.func.dfg.block_call(block, &[])); 516 } 517 let block = { 518 let i = environ.stacks.control_stack.len() - 1 - (default as usize); 519 let frame = &mut environ.stacks.control_stack[i]; 520 frame.set_branched_to_exit(); 521 frame.br_destination() 522 }; 523 let block = builder.func.dfg.block_call(block, &[]); 524 let jt = builder.create_jump_table(JumpTableData::new(block, &data)); 525 builder.ins().br_table(val, jt); 526 } else { 527 // Here we have jump arguments, but Cranelift's br_table doesn't support them 528 // We then proceed to split the edges going out of the br_table 529 let return_count = jump_args_count; 530 let mut dest_block_sequence = vec![]; 531 let mut dest_block_map = HashMap::new(); 532 for depth in targets.targets() { 533 let depth = depth?; 534 let branch_block = match dest_block_map.entry(depth as usize) { 535 hash_map::Entry::Occupied(entry) => *entry.get(), 536 hash_map::Entry::Vacant(entry) => { 537 let block = builder.create_block(); 538 dest_block_sequence.push((depth as usize, block)); 539 *entry.insert(block) 540 } 541 }; 542 data.push(builder.func.dfg.block_call(branch_block, &[])); 543 } 544 let default_branch_block = match dest_block_map.entry(default as usize) { 545 hash_map::Entry::Occupied(entry) => *entry.get(), 546 hash_map::Entry::Vacant(entry) => { 547 let block = builder.create_block(); 548 dest_block_sequence.push((default as usize, block)); 549 *entry.insert(block) 550 } 551 }; 552 let default_branch_block = builder.func.dfg.block_call(default_branch_block, &[]); 553 let jt = builder.create_jump_table(JumpTableData::new(default_branch_block, &data)); 554 builder.ins().br_table(val, jt); 555 for (depth, dest_block) in dest_block_sequence { 556 builder.switch_to_block(dest_block); 557 builder.seal_block(dest_block); 558 let real_dest_block = { 559 let i = environ.stacks.control_stack.len() - 1 - depth; 560 let frame = &mut environ.stacks.control_stack[i]; 561 frame.set_branched_to_exit(); 562 frame.br_destination() 563 }; 564 let destination_args = environ.stacks.peekn_mut(return_count); 565 canonicalise_then_jump(builder, real_dest_block, destination_args); 566 } 567 environ.stacks.popn(return_count); 568 } 569 environ.stacks.reachable = false; 570 } 571 Operator::Return => { 572 let return_count = { 573 let frame = &mut environ.stacks.control_stack[0]; 574 frame.num_return_values() 575 }; 576 { 577 let mut return_args = environ.stacks.peekn(return_count).to_vec(); 578 environ.handle_before_return(&return_args, builder); 579 bitcast_wasm_returns(&mut return_args, builder); 580 builder.ins().return_(&return_args); 581 } 582 environ.stacks.popn(return_count); 583 environ.stacks.reachable = false; 584 } 585 /********************************** Exception handling **********************************/ 586 Operator::Catch { .. } 587 | Operator::Rethrow { .. } 588 | Operator::Delegate { .. } 589 | Operator::CatchAll => { 590 return Err(wasm_unsupported!( 591 "legacy exception handling proposal is not supported" 592 )); 593 } 594 595 Operator::TryTable { try_table } => { 596 // First, create a block on the control stack. This also 597 // updates the handler state that is attached to all calls 598 // made within this block. 599 let body = builder.create_block(); 600 let (params, results) = blocktype_params_results(validator, try_table.ty)?; 601 let next = block_with_params(builder, results.clone(), environ)?; 602 builder.ins().jump(body, []); 603 builder.seal_block(body); 604 605 // For each catch clause, create a block with the 606 // equivalent of `br` to the target (unboxing the exnref 607 // into stack values or pushing it directly, depending on 608 // the kind of clause). 609 let ckpt = environ.stacks.handlers.take_checkpoint(); 610 let mut catch_blocks = vec![]; 611 // Process in *reverse* order: see the comment on 612 // [`HandlerState`]. In brief, this allows us to unify the 613 // left-to-right matching semantics of a single 614 // `try_table`'s catch clauses with the inside-out 615 // (deepest scope first) semantics of nested `try_table`s. 616 for catch in try_table.catches.iter().rev() { 617 // This will register the block in `state.handlers` 618 // under the appropriate tag. 619 catch_blocks.push(create_catch_block(builder, catch, environ)?); 620 } 621 622 environ.stacks.push_try_table_block( 623 next, 624 catch_blocks, 625 params.len(), 626 results.len(), 627 ckpt, 628 ); 629 630 // Continue codegen into the main body block. 631 builder.switch_to_block(body); 632 } 633 634 Operator::Throw { tag_index } => { 635 let tag_index = TagIndex::from_u32(*tag_index); 636 let arity = environ.tag_param_arity(tag_index); 637 let args = environ.stacks.peekn(arity).to_vec(); 638 environ.translate_exn_throw(builder, tag_index, &args)?; 639 environ.stacks.popn(arity); 640 environ.stacks.reachable = false; 641 } 642 643 Operator::ThrowRef => { 644 let exnref = environ.stacks.pop1(); 645 environ.translate_exn_throw_ref(builder, exnref)?; 646 environ.stacks.reachable = false; 647 } 648 649 /************************************ Calls **************************************** 650 * The call instructions pop off their arguments from the stack and append their 651 * return values to it. `call_indirect` needs environment support because there is an 652 * argument referring to an index in the external functions table of the module. 653 ************************************************************************************/ 654 Operator::Call { function_index } => { 655 let function_index = FuncIndex::from_u32(*function_index); 656 let ty = environ.module.functions[function_index] 657 .signature 658 .unwrap_module_type_index(); 659 let sig_ref = environ.get_or_create_interned_sig_ref(builder.func, ty); 660 let num_args = environ.num_params_for_func(function_index); 661 662 // Bitcast any vector arguments to their default type, I8X16, before calling. 663 let mut args = environ.stacks.peekn(num_args).to_vec(); 664 bitcast_wasm_params(environ, sig_ref, &mut args, builder); 665 666 let inst_results = 667 environ.translate_call(builder, srcloc, function_index, sig_ref, &args)?; 668 669 debug_assert_eq!( 670 inst_results.len(), 671 builder.func.dfg.signatures[sig_ref].returns.len(), 672 "translate_call results should match the call signature" 673 ); 674 environ.stacks.popn(num_args); 675 environ.stacks.pushn(&inst_results); 676 } 677 Operator::CallIndirect { 678 type_index, 679 table_index, 680 } => { 681 // `type_index` is the index of the function's signature and 682 // `table_index` is the index of the table to search the function 683 // in. 684 let type_index = TypeIndex::from_u32(*type_index); 685 let sigref = environ.get_or_create_sig_ref(builder.func, type_index); 686 let num_args = environ.num_params_for_function_type(type_index); 687 let callee = environ.stacks.pop1(); 688 689 // Bitcast any vector arguments to their default type, I8X16, before calling. 690 let mut args = environ.stacks.peekn(num_args).to_vec(); 691 bitcast_wasm_params(environ, sigref, &mut args, builder); 692 693 let inst_results = environ.translate_call_indirect( 694 builder, 695 srcloc, 696 validator.features(), 697 TableIndex::from_u32(*table_index), 698 type_index, 699 sigref, 700 callee, 701 &args, 702 )?; 703 let inst_results = match inst_results { 704 Some(results) => results, 705 None => { 706 environ.stacks.reachable = false; 707 return Ok(()); 708 } 709 }; 710 711 debug_assert_eq!( 712 inst_results.len(), 713 builder.func.dfg.signatures[sigref].returns.len(), 714 "translate_call_indirect results should match the call signature" 715 ); 716 environ.stacks.popn(num_args); 717 environ.stacks.pushn(&inst_results); 718 } 719 /******************************* Tail Calls ****************************************** 720 * The tail call instructions pop their arguments from the stack and 721 * then permanently transfer control to their callee. The indirect 722 * version requires environment support (while the direct version can 723 * optionally be hooked but doesn't require it) it interacts with the 724 * VM's runtime state via tables. 725 ************************************************************************************/ 726 Operator::ReturnCall { function_index } => { 727 let function_index = FuncIndex::from_u32(*function_index); 728 let ty = environ.module.functions[function_index] 729 .signature 730 .unwrap_module_type_index(); 731 let sig_ref = environ.get_or_create_interned_sig_ref(builder.func, ty); 732 let num_args = environ.num_params_for_func(function_index); 733 734 // Bitcast any vector arguments to their default type, I8X16, before calling. 735 let mut args = environ.stacks.peekn(num_args).to_vec(); 736 bitcast_wasm_params(environ, sig_ref, &mut args, builder); 737 738 environ.translate_return_call(builder, srcloc, function_index, sig_ref, &args)?; 739 740 environ.stacks.popn(num_args); 741 environ.stacks.reachable = false; 742 } 743 Operator::ReturnCallIndirect { 744 type_index, 745 table_index, 746 } => { 747 // `type_index` is the index of the function's signature and 748 // `table_index` is the index of the table to search the function 749 // in. 750 let type_index = TypeIndex::from_u32(*type_index); 751 let sigref = environ.get_or_create_sig_ref(builder.func, type_index); 752 let num_args = environ.num_params_for_function_type(type_index); 753 let callee = environ.stacks.pop1(); 754 755 // Bitcast any vector arguments to their default type, I8X16, before calling. 756 let mut args = environ.stacks.peekn(num_args).to_vec(); 757 bitcast_wasm_params(environ, sigref, &mut args, builder); 758 759 environ.translate_return_call_indirect( 760 builder, 761 srcloc, 762 validator.features(), 763 TableIndex::from_u32(*table_index), 764 type_index, 765 sigref, 766 callee, 767 &args, 768 )?; 769 770 environ.stacks.popn(num_args); 771 environ.stacks.reachable = false; 772 } 773 Operator::ReturnCallRef { type_index } => { 774 // Get function signature 775 // `index` is the index of the function's signature and `table_index` is the index of 776 // the table to search the function in. 777 let type_index = TypeIndex::from_u32(*type_index); 778 let sigref = environ.get_or_create_sig_ref(builder.func, type_index); 779 let num_args = environ.num_params_for_function_type(type_index); 780 let callee = environ.stacks.pop1(); 781 782 // Bitcast any vector arguments to their default type, I8X16, before calling. 783 let mut args = environ.stacks.peekn(num_args).to_vec(); 784 bitcast_wasm_params(environ, sigref, &mut args, builder); 785 786 environ.translate_return_call_ref(builder, srcloc, sigref, callee, &args)?; 787 788 environ.stacks.popn(num_args); 789 environ.stacks.reachable = false; 790 } 791 /******************************* Memory management *********************************** 792 * Memory management is handled by environment. It is usually translated into calls to 793 * special functions. 794 ************************************************************************************/ 795 Operator::MemoryGrow { mem } => { 796 // The WebAssembly MVP only supports one linear memory, but we expect the reserved 797 // argument to be a memory index. 798 let mem = MemoryIndex::from_u32(*mem); 799 let _heap = environ.get_or_create_heap(builder.func, mem); 800 let val = environ.stacks.pop1(); 801 environ.before_memory_grow(builder, val, mem); 802 let result = environ.translate_memory_grow(builder, mem, val)?; 803 environ.stacks.push1(result); 804 } 805 Operator::MemorySize { mem } => { 806 let mem = MemoryIndex::from_u32(*mem); 807 let _heap = environ.get_or_create_heap(builder.func, mem); 808 let result = environ.translate_memory_size(builder.cursor(), mem)?; 809 environ.stacks.push1(result); 810 } 811 /******************************* Load instructions *********************************** 812 * Wasm specifies an integer alignment flag but we drop it in Cranelift. 813 * The memory base address is provided by the environment. 814 ************************************************************************************/ 815 Operator::I32Load8U { memarg } => { 816 unwrap_or_return_unreachable_state!( 817 environ, 818 translate_load(memarg, ir::Opcode::Uload8, I32, builder, environ)? 819 ); 820 } 821 Operator::I32Load16U { memarg } => { 822 unwrap_or_return_unreachable_state!( 823 environ, 824 translate_load(memarg, ir::Opcode::Uload16, I32, builder, environ)? 825 ); 826 } 827 Operator::I32Load8S { memarg } => { 828 unwrap_or_return_unreachable_state!( 829 environ, 830 translate_load(memarg, ir::Opcode::Sload8, I32, builder, environ)? 831 ); 832 } 833 Operator::I32Load16S { memarg } => { 834 unwrap_or_return_unreachable_state!( 835 environ, 836 translate_load(memarg, ir::Opcode::Sload16, I32, builder, environ)? 837 ); 838 } 839 Operator::I64Load8U { memarg } => { 840 unwrap_or_return_unreachable_state!( 841 environ, 842 translate_load(memarg, ir::Opcode::Uload8, I64, builder, environ)? 843 ); 844 } 845 Operator::I64Load16U { memarg } => { 846 unwrap_or_return_unreachable_state!( 847 environ, 848 translate_load(memarg, ir::Opcode::Uload16, I64, builder, environ)? 849 ); 850 } 851 Operator::I64Load8S { memarg } => { 852 unwrap_or_return_unreachable_state!( 853 environ, 854 translate_load(memarg, ir::Opcode::Sload8, I64, builder, environ)? 855 ); 856 } 857 Operator::I64Load16S { memarg } => { 858 unwrap_or_return_unreachable_state!( 859 environ, 860 translate_load(memarg, ir::Opcode::Sload16, I64, builder, environ)? 861 ); 862 } 863 Operator::I64Load32S { memarg } => { 864 unwrap_or_return_unreachable_state!( 865 environ, 866 translate_load(memarg, ir::Opcode::Sload32, I64, builder, environ)? 867 ); 868 } 869 Operator::I64Load32U { memarg } => { 870 unwrap_or_return_unreachable_state!( 871 environ, 872 translate_load(memarg, ir::Opcode::Uload32, I64, builder, environ)? 873 ); 874 } 875 Operator::I32Load { memarg } => { 876 unwrap_or_return_unreachable_state!( 877 environ, 878 translate_load(memarg, ir::Opcode::Load, I32, builder, environ)? 879 ); 880 } 881 Operator::F32Load { memarg } => { 882 unwrap_or_return_unreachable_state!( 883 environ, 884 translate_load(memarg, ir::Opcode::Load, F32, builder, environ)? 885 ); 886 } 887 Operator::I64Load { memarg } => { 888 unwrap_or_return_unreachable_state!( 889 environ, 890 translate_load(memarg, ir::Opcode::Load, I64, builder, environ)? 891 ); 892 } 893 Operator::F64Load { memarg } => { 894 unwrap_or_return_unreachable_state!( 895 environ, 896 translate_load(memarg, ir::Opcode::Load, F64, builder, environ)? 897 ); 898 } 899 Operator::V128Load { memarg } => { 900 unwrap_or_return_unreachable_state!( 901 environ, 902 translate_load(memarg, ir::Opcode::Load, I8X16, builder, environ)? 903 ); 904 } 905 Operator::V128Load8x8S { memarg } => { 906 //TODO(#6829): add before_load() and before_store() hooks for SIMD loads and stores. 907 let (flags, _, base) = unwrap_or_return_unreachable_state!( 908 environ, 909 prepare_addr(memarg, 8, builder, environ)? 910 ); 911 let loaded = builder.ins().sload8x8(flags, base, 0); 912 environ.stacks.push1(loaded); 913 } 914 Operator::V128Load8x8U { memarg } => { 915 let (flags, _, base) = unwrap_or_return_unreachable_state!( 916 environ, 917 prepare_addr(memarg, 8, builder, environ)? 918 ); 919 let loaded = builder.ins().uload8x8(flags, base, 0); 920 environ.stacks.push1(loaded); 921 } 922 Operator::V128Load16x4S { memarg } => { 923 let (flags, _, base) = unwrap_or_return_unreachable_state!( 924 environ, 925 prepare_addr(memarg, 8, builder, environ)? 926 ); 927 let loaded = builder.ins().sload16x4(flags, base, 0); 928 environ.stacks.push1(loaded); 929 } 930 Operator::V128Load16x4U { memarg } => { 931 let (flags, _, base) = unwrap_or_return_unreachable_state!( 932 environ, 933 prepare_addr(memarg, 8, builder, environ)? 934 ); 935 let loaded = builder.ins().uload16x4(flags, base, 0); 936 environ.stacks.push1(loaded); 937 } 938 Operator::V128Load32x2S { memarg } => { 939 let (flags, _, base) = unwrap_or_return_unreachable_state!( 940 environ, 941 prepare_addr(memarg, 8, builder, environ)? 942 ); 943 let loaded = builder.ins().sload32x2(flags, base, 0); 944 environ.stacks.push1(loaded); 945 } 946 Operator::V128Load32x2U { memarg } => { 947 let (flags, _, base) = unwrap_or_return_unreachable_state!( 948 environ, 949 prepare_addr(memarg, 8, builder, environ)? 950 ); 951 let loaded = builder.ins().uload32x2(flags, base, 0); 952 environ.stacks.push1(loaded); 953 } 954 /****************************** Store instructions *********************************** 955 * Wasm specifies an integer alignment flag but we drop it in Cranelift. 956 * The memory base address is provided by the environment. 957 ************************************************************************************/ 958 Operator::I32Store { memarg } 959 | Operator::I64Store { memarg } 960 | Operator::F32Store { memarg } 961 | Operator::F64Store { memarg } => { 962 translate_store(memarg, ir::Opcode::Store, builder, environ)?; 963 } 964 Operator::I32Store8 { memarg } | Operator::I64Store8 { memarg } => { 965 translate_store(memarg, ir::Opcode::Istore8, builder, environ)?; 966 } 967 Operator::I32Store16 { memarg } | Operator::I64Store16 { memarg } => { 968 translate_store(memarg, ir::Opcode::Istore16, builder, environ)?; 969 } 970 Operator::I64Store32 { memarg } => { 971 translate_store(memarg, ir::Opcode::Istore32, builder, environ)?; 972 } 973 Operator::V128Store { memarg } => { 974 translate_store(memarg, ir::Opcode::Store, builder, environ)?; 975 } 976 /****************************** Nullary Operators ************************************/ 977 Operator::I32Const { value } => { 978 environ 979 .stacks 980 .push1(builder.ins().iconst(I32, i64::from(value.cast_unsigned()))); 981 } 982 Operator::I64Const { value } => environ.stacks.push1(builder.ins().iconst(I64, *value)), 983 Operator::F32Const { value } => { 984 environ 985 .stacks 986 .push1(builder.ins().f32const(f32_translation(*value))); 987 } 988 Operator::F64Const { value } => { 989 environ 990 .stacks 991 .push1(builder.ins().f64const(f64_translation(*value))); 992 } 993 /******************************* Unary Operators *************************************/ 994 Operator::I32Clz | Operator::I64Clz => { 995 let arg = environ.stacks.pop1(); 996 environ.stacks.push1(builder.ins().clz(arg)); 997 } 998 Operator::I32Ctz | Operator::I64Ctz => { 999 let arg = environ.stacks.pop1(); 1000 environ.stacks.push1(builder.ins().ctz(arg)); 1001 } 1002 Operator::I32Popcnt | Operator::I64Popcnt => { 1003 let arg = environ.stacks.pop1(); 1004 environ.stacks.push1(builder.ins().popcnt(arg)); 1005 } 1006 Operator::I64ExtendI32S => { 1007 let val = environ.stacks.pop1(); 1008 environ.stacks.push1(builder.ins().sextend(I64, val)); 1009 } 1010 Operator::I64ExtendI32U => { 1011 let val = environ.stacks.pop1(); 1012 environ.stacks.push1(builder.ins().uextend(I64, val)); 1013 } 1014 Operator::I32WrapI64 => { 1015 let val = environ.stacks.pop1(); 1016 environ.stacks.push1(builder.ins().ireduce(I32, val)); 1017 } 1018 Operator::F32Sqrt | Operator::F64Sqrt => { 1019 let arg = environ.stacks.pop1(); 1020 environ.stacks.push1(builder.ins().sqrt(arg)); 1021 } 1022 Operator::F32Ceil => { 1023 let arg = environ.stacks.pop1(); 1024 let result = environ.ceil_f32(builder, arg); 1025 environ.stacks.push1(result); 1026 } 1027 Operator::F64Ceil => { 1028 let arg = environ.stacks.pop1(); 1029 let result = environ.ceil_f64(builder, arg); 1030 environ.stacks.push1(result); 1031 } 1032 Operator::F32Floor => { 1033 let arg = environ.stacks.pop1(); 1034 let result = environ.floor_f32(builder, arg); 1035 environ.stacks.push1(result); 1036 } 1037 Operator::F64Floor => { 1038 let arg = environ.stacks.pop1(); 1039 let result = environ.floor_f64(builder, arg); 1040 environ.stacks.push1(result); 1041 } 1042 Operator::F32Trunc => { 1043 let arg = environ.stacks.pop1(); 1044 let result = environ.trunc_f32(builder, arg); 1045 environ.stacks.push1(result); 1046 } 1047 Operator::F64Trunc => { 1048 let arg = environ.stacks.pop1(); 1049 let result = environ.trunc_f64(builder, arg); 1050 environ.stacks.push1(result); 1051 } 1052 Operator::F32Nearest => { 1053 let arg = environ.stacks.pop1(); 1054 let result = environ.nearest_f32(builder, arg); 1055 environ.stacks.push1(result); 1056 } 1057 Operator::F64Nearest => { 1058 let arg = environ.stacks.pop1(); 1059 let result = environ.nearest_f64(builder, arg); 1060 environ.stacks.push1(result); 1061 } 1062 Operator::F32Abs | Operator::F64Abs => { 1063 let val = environ.stacks.pop1(); 1064 environ.stacks.push1(builder.ins().fabs(val)); 1065 } 1066 Operator::F32Neg | Operator::F64Neg => { 1067 let arg = environ.stacks.pop1(); 1068 environ.stacks.push1(builder.ins().fneg(arg)); 1069 } 1070 Operator::F64ConvertI64U | Operator::F64ConvertI32U => { 1071 let val = environ.stacks.pop1(); 1072 environ.stacks.push1(builder.ins().fcvt_from_uint(F64, val)); 1073 } 1074 Operator::F64ConvertI64S | Operator::F64ConvertI32S => { 1075 let val = environ.stacks.pop1(); 1076 environ.stacks.push1(builder.ins().fcvt_from_sint(F64, val)); 1077 } 1078 Operator::F32ConvertI64S | Operator::F32ConvertI32S => { 1079 let val = environ.stacks.pop1(); 1080 environ.stacks.push1(builder.ins().fcvt_from_sint(F32, val)); 1081 } 1082 Operator::F32ConvertI64U | Operator::F32ConvertI32U => { 1083 let val = environ.stacks.pop1(); 1084 environ.stacks.push1(builder.ins().fcvt_from_uint(F32, val)); 1085 } 1086 Operator::F64PromoteF32 => { 1087 let val = environ.stacks.pop1(); 1088 environ.stacks.push1(builder.ins().fpromote(F64, val)); 1089 } 1090 Operator::F32DemoteF64 => { 1091 let val = environ.stacks.pop1(); 1092 environ.stacks.push1(builder.ins().fdemote(F32, val)); 1093 } 1094 Operator::I64TruncF64S | Operator::I64TruncF32S => { 1095 let val = environ.stacks.pop1(); 1096 let result = environ.translate_fcvt_to_sint(builder, I64, val); 1097 environ.stacks.push1(result); 1098 } 1099 Operator::I32TruncF64S | Operator::I32TruncF32S => { 1100 let val = environ.stacks.pop1(); 1101 let result = environ.translate_fcvt_to_sint(builder, I32, val); 1102 environ.stacks.push1(result); 1103 } 1104 Operator::I64TruncF64U | Operator::I64TruncF32U => { 1105 let val = environ.stacks.pop1(); 1106 let result = environ.translate_fcvt_to_uint(builder, I64, val); 1107 environ.stacks.push1(result); 1108 } 1109 Operator::I32TruncF64U | Operator::I32TruncF32U => { 1110 let val = environ.stacks.pop1(); 1111 let result = environ.translate_fcvt_to_uint(builder, I32, val); 1112 environ.stacks.push1(result); 1113 } 1114 Operator::I64TruncSatF64S | Operator::I64TruncSatF32S => { 1115 let val = environ.stacks.pop1(); 1116 environ 1117 .stacks 1118 .push1(builder.ins().fcvt_to_sint_sat(I64, val)); 1119 } 1120 Operator::I32TruncSatF64S | Operator::I32TruncSatF32S => { 1121 let val = environ.stacks.pop1(); 1122 environ 1123 .stacks 1124 .push1(builder.ins().fcvt_to_sint_sat(I32, val)); 1125 } 1126 Operator::I64TruncSatF64U | Operator::I64TruncSatF32U => { 1127 let val = environ.stacks.pop1(); 1128 environ 1129 .stacks 1130 .push1(builder.ins().fcvt_to_uint_sat(I64, val)); 1131 } 1132 Operator::I32TruncSatF64U | Operator::I32TruncSatF32U => { 1133 let val = environ.stacks.pop1(); 1134 environ 1135 .stacks 1136 .push1(builder.ins().fcvt_to_uint_sat(I32, val)); 1137 } 1138 Operator::F32ReinterpretI32 => { 1139 let val = environ.stacks.pop1(); 1140 environ 1141 .stacks 1142 .push1(builder.ins().bitcast(F32, MemFlags::new(), val)); 1143 } 1144 Operator::F64ReinterpretI64 => { 1145 let val = environ.stacks.pop1(); 1146 environ 1147 .stacks 1148 .push1(builder.ins().bitcast(F64, MemFlags::new(), val)); 1149 } 1150 Operator::I32ReinterpretF32 => { 1151 let val = environ.stacks.pop1(); 1152 environ 1153 .stacks 1154 .push1(builder.ins().bitcast(I32, MemFlags::new(), val)); 1155 } 1156 Operator::I64ReinterpretF64 => { 1157 let val = environ.stacks.pop1(); 1158 environ 1159 .stacks 1160 .push1(builder.ins().bitcast(I64, MemFlags::new(), val)); 1161 } 1162 Operator::I32Extend8S => { 1163 let val = environ.stacks.pop1(); 1164 environ.stacks.push1(builder.ins().ireduce(I8, val)); 1165 let val = environ.stacks.pop1(); 1166 environ.stacks.push1(builder.ins().sextend(I32, val)); 1167 } 1168 Operator::I32Extend16S => { 1169 let val = environ.stacks.pop1(); 1170 environ.stacks.push1(builder.ins().ireduce(I16, val)); 1171 let val = environ.stacks.pop1(); 1172 environ.stacks.push1(builder.ins().sextend(I32, val)); 1173 } 1174 Operator::I64Extend8S => { 1175 let val = environ.stacks.pop1(); 1176 environ.stacks.push1(builder.ins().ireduce(I8, val)); 1177 let val = environ.stacks.pop1(); 1178 environ.stacks.push1(builder.ins().sextend(I64, val)); 1179 } 1180 Operator::I64Extend16S => { 1181 let val = environ.stacks.pop1(); 1182 environ.stacks.push1(builder.ins().ireduce(I16, val)); 1183 let val = environ.stacks.pop1(); 1184 environ.stacks.push1(builder.ins().sextend(I64, val)); 1185 } 1186 Operator::I64Extend32S => { 1187 let val = environ.stacks.pop1(); 1188 environ.stacks.push1(builder.ins().ireduce(I32, val)); 1189 let val = environ.stacks.pop1(); 1190 environ.stacks.push1(builder.ins().sextend(I64, val)); 1191 } 1192 /****************************** Binary Operators ************************************/ 1193 Operator::I32Add | Operator::I64Add => { 1194 let (arg1, arg2) = environ.stacks.pop2(); 1195 environ.stacks.push1(builder.ins().iadd(arg1, arg2)); 1196 } 1197 Operator::I32And | Operator::I64And => { 1198 let (arg1, arg2) = environ.stacks.pop2(); 1199 environ.stacks.push1(builder.ins().band(arg1, arg2)); 1200 } 1201 Operator::I32Or | Operator::I64Or => { 1202 let (arg1, arg2) = environ.stacks.pop2(); 1203 environ.stacks.push1(builder.ins().bor(arg1, arg2)); 1204 } 1205 Operator::I32Xor | Operator::I64Xor => { 1206 let (arg1, arg2) = environ.stacks.pop2(); 1207 environ.stacks.push1(builder.ins().bxor(arg1, arg2)); 1208 } 1209 Operator::I32Shl | Operator::I64Shl => { 1210 let (arg1, arg2) = environ.stacks.pop2(); 1211 environ.stacks.push1(builder.ins().ishl(arg1, arg2)); 1212 } 1213 Operator::I32ShrS | Operator::I64ShrS => { 1214 let (arg1, arg2) = environ.stacks.pop2(); 1215 environ.stacks.push1(builder.ins().sshr(arg1, arg2)); 1216 } 1217 Operator::I32ShrU | Operator::I64ShrU => { 1218 let (arg1, arg2) = environ.stacks.pop2(); 1219 environ.stacks.push1(builder.ins().ushr(arg1, arg2)); 1220 } 1221 Operator::I32Rotl | Operator::I64Rotl => { 1222 let (arg1, arg2) = environ.stacks.pop2(); 1223 environ.stacks.push1(builder.ins().rotl(arg1, arg2)); 1224 } 1225 Operator::I32Rotr | Operator::I64Rotr => { 1226 let (arg1, arg2) = environ.stacks.pop2(); 1227 environ.stacks.push1(builder.ins().rotr(arg1, arg2)); 1228 } 1229 Operator::F32Add | Operator::F64Add => { 1230 let (arg1, arg2) = environ.stacks.pop2(); 1231 environ.stacks.push1(builder.ins().fadd(arg1, arg2)); 1232 } 1233 Operator::I32Sub | Operator::I64Sub => { 1234 let (arg1, arg2) = environ.stacks.pop2(); 1235 environ.stacks.push1(builder.ins().isub(arg1, arg2)); 1236 } 1237 Operator::F32Sub | Operator::F64Sub => { 1238 let (arg1, arg2) = environ.stacks.pop2(); 1239 environ.stacks.push1(builder.ins().fsub(arg1, arg2)); 1240 } 1241 Operator::I32Mul | Operator::I64Mul => { 1242 let (arg1, arg2) = environ.stacks.pop2(); 1243 environ.stacks.push1(builder.ins().imul(arg1, arg2)); 1244 } 1245 Operator::F32Mul | Operator::F64Mul => { 1246 let (arg1, arg2) = environ.stacks.pop2(); 1247 environ.stacks.push1(builder.ins().fmul(arg1, arg2)); 1248 } 1249 Operator::F32Div | Operator::F64Div => { 1250 let (arg1, arg2) = environ.stacks.pop2(); 1251 environ.stacks.push1(builder.ins().fdiv(arg1, arg2)); 1252 } 1253 Operator::I32DivS | Operator::I64DivS => { 1254 let (arg1, arg2) = environ.stacks.pop2(); 1255 let result = environ.translate_sdiv(builder, arg1, arg2); 1256 environ.stacks.push1(result); 1257 } 1258 Operator::I32DivU | Operator::I64DivU => { 1259 let (arg1, arg2) = environ.stacks.pop2(); 1260 let result = environ.translate_udiv(builder, arg1, arg2); 1261 environ.stacks.push1(result); 1262 } 1263 Operator::I32RemS | Operator::I64RemS => { 1264 let (arg1, arg2) = environ.stacks.pop2(); 1265 let result = environ.translate_srem(builder, arg1, arg2); 1266 environ.stacks.push1(result); 1267 } 1268 Operator::I32RemU | Operator::I64RemU => { 1269 let (arg1, arg2) = environ.stacks.pop2(); 1270 let result = environ.translate_urem(builder, arg1, arg2); 1271 environ.stacks.push1(result); 1272 } 1273 Operator::F32Min | Operator::F64Min => { 1274 let (arg1, arg2) = environ.stacks.pop2(); 1275 environ.stacks.push1(builder.ins().fmin(arg1, arg2)); 1276 } 1277 Operator::F32Max | Operator::F64Max => { 1278 let (arg1, arg2) = environ.stacks.pop2(); 1279 environ.stacks.push1(builder.ins().fmax(arg1, arg2)); 1280 } 1281 Operator::F32Copysign | Operator::F64Copysign => { 1282 let (arg1, arg2) = environ.stacks.pop2(); 1283 environ.stacks.push1(builder.ins().fcopysign(arg1, arg2)); 1284 } 1285 /**************************** Comparison Operators **********************************/ 1286 Operator::I32LtS | Operator::I64LtS => { 1287 translate_icmp(IntCC::SignedLessThan, builder, environ) 1288 } 1289 Operator::I32LtU | Operator::I64LtU => { 1290 translate_icmp(IntCC::UnsignedLessThan, builder, environ) 1291 } 1292 Operator::I32LeS | Operator::I64LeS => { 1293 translate_icmp(IntCC::SignedLessThanOrEqual, builder, environ) 1294 } 1295 Operator::I32LeU | Operator::I64LeU => { 1296 translate_icmp(IntCC::UnsignedLessThanOrEqual, builder, environ) 1297 } 1298 Operator::I32GtS | Operator::I64GtS => { 1299 translate_icmp(IntCC::SignedGreaterThan, builder, environ) 1300 } 1301 Operator::I32GtU | Operator::I64GtU => { 1302 translate_icmp(IntCC::UnsignedGreaterThan, builder, environ) 1303 } 1304 Operator::I32GeS | Operator::I64GeS => { 1305 translate_icmp(IntCC::SignedGreaterThanOrEqual, builder, environ) 1306 } 1307 Operator::I32GeU | Operator::I64GeU => { 1308 translate_icmp(IntCC::UnsignedGreaterThanOrEqual, builder, environ) 1309 } 1310 Operator::I32Eqz | Operator::I64Eqz => { 1311 let arg = environ.stacks.pop1(); 1312 let val = builder.ins().icmp_imm(IntCC::Equal, arg, 0); 1313 environ.stacks.push1(builder.ins().uextend(I32, val)); 1314 } 1315 Operator::I32Eq | Operator::I64Eq => translate_icmp(IntCC::Equal, builder, environ), 1316 Operator::F32Eq | Operator::F64Eq => translate_fcmp(FloatCC::Equal, builder, environ), 1317 Operator::I32Ne | Operator::I64Ne => translate_icmp(IntCC::NotEqual, builder, environ), 1318 Operator::F32Ne | Operator::F64Ne => translate_fcmp(FloatCC::NotEqual, builder, environ), 1319 Operator::F32Gt | Operator::F64Gt => translate_fcmp(FloatCC::GreaterThan, builder, environ), 1320 Operator::F32Ge | Operator::F64Ge => { 1321 translate_fcmp(FloatCC::GreaterThanOrEqual, builder, environ) 1322 } 1323 Operator::F32Lt | Operator::F64Lt => translate_fcmp(FloatCC::LessThan, builder, environ), 1324 Operator::F32Le | Operator::F64Le => { 1325 translate_fcmp(FloatCC::LessThanOrEqual, builder, environ) 1326 } 1327 Operator::RefNull { hty } => { 1328 let hty = environ.convert_heap_type(*hty)?; 1329 let result = environ.translate_ref_null(builder.cursor(), hty)?; 1330 environ.stacks.push1(result); 1331 } 1332 Operator::RefIsNull => { 1333 let value = environ.stacks.pop1(); 1334 let [WasmValType::Ref(ty)] = operand_types else { 1335 unreachable!("validation") 1336 }; 1337 let result = environ.translate_ref_is_null(builder.cursor(), value, *ty)?; 1338 environ.stacks.push1(result); 1339 } 1340 Operator::RefFunc { function_index } => { 1341 let index = FuncIndex::from_u32(*function_index); 1342 let result = environ.translate_ref_func(builder.cursor(), index)?; 1343 environ.stacks.push1(result); 1344 } 1345 Operator::MemoryAtomicWait32 { memarg } | Operator::MemoryAtomicWait64 { memarg } => { 1346 // The WebAssembly MVP only supports one linear memory and 1347 // wasmparser will ensure that the memory indices specified are 1348 // zero. 1349 let implied_ty = match op { 1350 Operator::MemoryAtomicWait64 { .. } => I64, 1351 Operator::MemoryAtomicWait32 { .. } => I32, 1352 _ => unreachable!(), 1353 }; 1354 let memory_index = MemoryIndex::from_u32(memarg.memory); 1355 let heap = environ.get_or_create_heap(builder.func, memory_index); 1356 let timeout = environ.stacks.pop1(); // 64 (fixed) 1357 let expected = environ.stacks.pop1(); // 32 or 64 (per the `Ixx` in `IxxAtomicWait`) 1358 assert!(builder.func.dfg.value_type(expected) == implied_ty); 1359 let addr = environ.stacks.pop1(); 1360 let effective_addr = if memarg.offset == 0 { 1361 addr 1362 } else { 1363 let index_type = environ.heaps()[heap].index_type(); 1364 let offset = builder.ins().iconst(index_type, memarg.offset as i64); 1365 environ.uadd_overflow_trap(builder, addr, offset, ir::TrapCode::HEAP_OUT_OF_BOUNDS) 1366 }; 1367 // `fn translate_atomic_wait` can inspect the type of `expected` to figure out what 1368 // code it needs to generate, if it wants. 1369 let res = environ.translate_atomic_wait( 1370 builder, 1371 memory_index, 1372 heap, 1373 effective_addr, 1374 expected, 1375 timeout, 1376 )?; 1377 environ.stacks.push1(res); 1378 } 1379 Operator::MemoryAtomicNotify { memarg } => { 1380 let memory_index = MemoryIndex::from_u32(memarg.memory); 1381 let heap = environ.get_or_create_heap(builder.func, memory_index); 1382 let count = environ.stacks.pop1(); // 32 (fixed) 1383 let addr = environ.stacks.pop1(); 1384 let effective_addr = if memarg.offset == 0 { 1385 addr 1386 } else { 1387 let index_type = environ.heaps()[heap].index_type(); 1388 let offset = builder.ins().iconst(index_type, memarg.offset as i64); 1389 environ.uadd_overflow_trap(builder, addr, offset, ir::TrapCode::HEAP_OUT_OF_BOUNDS) 1390 }; 1391 let res = environ.translate_atomic_notify( 1392 builder, 1393 memory_index, 1394 heap, 1395 effective_addr, 1396 count, 1397 )?; 1398 environ.stacks.push1(res); 1399 } 1400 Operator::I32AtomicLoad { memarg } => { 1401 translate_atomic_load(I32, I32, memarg, builder, environ)? 1402 } 1403 Operator::I64AtomicLoad { memarg } => { 1404 translate_atomic_load(I64, I64, memarg, builder, environ)? 1405 } 1406 Operator::I32AtomicLoad8U { memarg } => { 1407 translate_atomic_load(I32, I8, memarg, builder, environ)? 1408 } 1409 Operator::I32AtomicLoad16U { memarg } => { 1410 translate_atomic_load(I32, I16, memarg, builder, environ)? 1411 } 1412 Operator::I64AtomicLoad8U { memarg } => { 1413 translate_atomic_load(I64, I8, memarg, builder, environ)? 1414 } 1415 Operator::I64AtomicLoad16U { memarg } => { 1416 translate_atomic_load(I64, I16, memarg, builder, environ)? 1417 } 1418 Operator::I64AtomicLoad32U { memarg } => { 1419 translate_atomic_load(I64, I32, memarg, builder, environ)? 1420 } 1421 1422 Operator::I32AtomicStore { memarg } => { 1423 translate_atomic_store(I32, memarg, builder, environ)? 1424 } 1425 Operator::I64AtomicStore { memarg } => { 1426 translate_atomic_store(I64, memarg, builder, environ)? 1427 } 1428 Operator::I32AtomicStore8 { memarg } => { 1429 translate_atomic_store(I8, memarg, builder, environ)? 1430 } 1431 Operator::I32AtomicStore16 { memarg } => { 1432 translate_atomic_store(I16, memarg, builder, environ)? 1433 } 1434 Operator::I64AtomicStore8 { memarg } => { 1435 translate_atomic_store(I8, memarg, builder, environ)? 1436 } 1437 Operator::I64AtomicStore16 { memarg } => { 1438 translate_atomic_store(I16, memarg, builder, environ)? 1439 } 1440 Operator::I64AtomicStore32 { memarg } => { 1441 translate_atomic_store(I32, memarg, builder, environ)? 1442 } 1443 1444 Operator::I32AtomicRmwAdd { memarg } => { 1445 translate_atomic_rmw(I32, I32, AtomicRmwOp::Add, memarg, builder, environ)? 1446 } 1447 Operator::I64AtomicRmwAdd { memarg } => { 1448 translate_atomic_rmw(I64, I64, AtomicRmwOp::Add, memarg, builder, environ)? 1449 } 1450 Operator::I32AtomicRmw8AddU { memarg } => { 1451 translate_atomic_rmw(I32, I8, AtomicRmwOp::Add, memarg, builder, environ)? 1452 } 1453 Operator::I32AtomicRmw16AddU { memarg } => { 1454 translate_atomic_rmw(I32, I16, AtomicRmwOp::Add, memarg, builder, environ)? 1455 } 1456 Operator::I64AtomicRmw8AddU { memarg } => { 1457 translate_atomic_rmw(I64, I8, AtomicRmwOp::Add, memarg, builder, environ)? 1458 } 1459 Operator::I64AtomicRmw16AddU { memarg } => { 1460 translate_atomic_rmw(I64, I16, AtomicRmwOp::Add, memarg, builder, environ)? 1461 } 1462 Operator::I64AtomicRmw32AddU { memarg } => { 1463 translate_atomic_rmw(I64, I32, AtomicRmwOp::Add, memarg, builder, environ)? 1464 } 1465 1466 Operator::I32AtomicRmwSub { memarg } => { 1467 translate_atomic_rmw(I32, I32, AtomicRmwOp::Sub, memarg, builder, environ)? 1468 } 1469 Operator::I64AtomicRmwSub { memarg } => { 1470 translate_atomic_rmw(I64, I64, AtomicRmwOp::Sub, memarg, builder, environ)? 1471 } 1472 Operator::I32AtomicRmw8SubU { memarg } => { 1473 translate_atomic_rmw(I32, I8, AtomicRmwOp::Sub, memarg, builder, environ)? 1474 } 1475 Operator::I32AtomicRmw16SubU { memarg } => { 1476 translate_atomic_rmw(I32, I16, AtomicRmwOp::Sub, memarg, builder, environ)? 1477 } 1478 Operator::I64AtomicRmw8SubU { memarg } => { 1479 translate_atomic_rmw(I64, I8, AtomicRmwOp::Sub, memarg, builder, environ)? 1480 } 1481 Operator::I64AtomicRmw16SubU { memarg } => { 1482 translate_atomic_rmw(I64, I16, AtomicRmwOp::Sub, memarg, builder, environ)? 1483 } 1484 Operator::I64AtomicRmw32SubU { memarg } => { 1485 translate_atomic_rmw(I64, I32, AtomicRmwOp::Sub, memarg, builder, environ)? 1486 } 1487 1488 Operator::I32AtomicRmwAnd { memarg } => { 1489 translate_atomic_rmw(I32, I32, AtomicRmwOp::And, memarg, builder, environ)? 1490 } 1491 Operator::I64AtomicRmwAnd { memarg } => { 1492 translate_atomic_rmw(I64, I64, AtomicRmwOp::And, memarg, builder, environ)? 1493 } 1494 Operator::I32AtomicRmw8AndU { memarg } => { 1495 translate_atomic_rmw(I32, I8, AtomicRmwOp::And, memarg, builder, environ)? 1496 } 1497 Operator::I32AtomicRmw16AndU { memarg } => { 1498 translate_atomic_rmw(I32, I16, AtomicRmwOp::And, memarg, builder, environ)? 1499 } 1500 Operator::I64AtomicRmw8AndU { memarg } => { 1501 translate_atomic_rmw(I64, I8, AtomicRmwOp::And, memarg, builder, environ)? 1502 } 1503 Operator::I64AtomicRmw16AndU { memarg } => { 1504 translate_atomic_rmw(I64, I16, AtomicRmwOp::And, memarg, builder, environ)? 1505 } 1506 Operator::I64AtomicRmw32AndU { memarg } => { 1507 translate_atomic_rmw(I64, I32, AtomicRmwOp::And, memarg, builder, environ)? 1508 } 1509 1510 Operator::I32AtomicRmwOr { memarg } => { 1511 translate_atomic_rmw(I32, I32, AtomicRmwOp::Or, memarg, builder, environ)? 1512 } 1513 Operator::I64AtomicRmwOr { memarg } => { 1514 translate_atomic_rmw(I64, I64, AtomicRmwOp::Or, memarg, builder, environ)? 1515 } 1516 Operator::I32AtomicRmw8OrU { memarg } => { 1517 translate_atomic_rmw(I32, I8, AtomicRmwOp::Or, memarg, builder, environ)? 1518 } 1519 Operator::I32AtomicRmw16OrU { memarg } => { 1520 translate_atomic_rmw(I32, I16, AtomicRmwOp::Or, memarg, builder, environ)? 1521 } 1522 Operator::I64AtomicRmw8OrU { memarg } => { 1523 translate_atomic_rmw(I64, I8, AtomicRmwOp::Or, memarg, builder, environ)? 1524 } 1525 Operator::I64AtomicRmw16OrU { memarg } => { 1526 translate_atomic_rmw(I64, I16, AtomicRmwOp::Or, memarg, builder, environ)? 1527 } 1528 Operator::I64AtomicRmw32OrU { memarg } => { 1529 translate_atomic_rmw(I64, I32, AtomicRmwOp::Or, memarg, builder, environ)? 1530 } 1531 1532 Operator::I32AtomicRmwXor { memarg } => { 1533 translate_atomic_rmw(I32, I32, AtomicRmwOp::Xor, memarg, builder, environ)? 1534 } 1535 Operator::I64AtomicRmwXor { memarg } => { 1536 translate_atomic_rmw(I64, I64, AtomicRmwOp::Xor, memarg, builder, environ)? 1537 } 1538 Operator::I32AtomicRmw8XorU { memarg } => { 1539 translate_atomic_rmw(I32, I8, AtomicRmwOp::Xor, memarg, builder, environ)? 1540 } 1541 Operator::I32AtomicRmw16XorU { memarg } => { 1542 translate_atomic_rmw(I32, I16, AtomicRmwOp::Xor, memarg, builder, environ)? 1543 } 1544 Operator::I64AtomicRmw8XorU { memarg } => { 1545 translate_atomic_rmw(I64, I8, AtomicRmwOp::Xor, memarg, builder, environ)? 1546 } 1547 Operator::I64AtomicRmw16XorU { memarg } => { 1548 translate_atomic_rmw(I64, I16, AtomicRmwOp::Xor, memarg, builder, environ)? 1549 } 1550 Operator::I64AtomicRmw32XorU { memarg } => { 1551 translate_atomic_rmw(I64, I32, AtomicRmwOp::Xor, memarg, builder, environ)? 1552 } 1553 1554 Operator::I32AtomicRmwXchg { memarg } => { 1555 translate_atomic_rmw(I32, I32, AtomicRmwOp::Xchg, memarg, builder, environ)? 1556 } 1557 Operator::I64AtomicRmwXchg { memarg } => { 1558 translate_atomic_rmw(I64, I64, AtomicRmwOp::Xchg, memarg, builder, environ)? 1559 } 1560 Operator::I32AtomicRmw8XchgU { memarg } => { 1561 translate_atomic_rmw(I32, I8, AtomicRmwOp::Xchg, memarg, builder, environ)? 1562 } 1563 Operator::I32AtomicRmw16XchgU { memarg } => { 1564 translate_atomic_rmw(I32, I16, AtomicRmwOp::Xchg, memarg, builder, environ)? 1565 } 1566 Operator::I64AtomicRmw8XchgU { memarg } => { 1567 translate_atomic_rmw(I64, I8, AtomicRmwOp::Xchg, memarg, builder, environ)? 1568 } 1569 Operator::I64AtomicRmw16XchgU { memarg } => { 1570 translate_atomic_rmw(I64, I16, AtomicRmwOp::Xchg, memarg, builder, environ)? 1571 } 1572 Operator::I64AtomicRmw32XchgU { memarg } => { 1573 translate_atomic_rmw(I64, I32, AtomicRmwOp::Xchg, memarg, builder, environ)? 1574 } 1575 1576 Operator::I32AtomicRmwCmpxchg { memarg } => { 1577 translate_atomic_cas(I32, I32, memarg, builder, environ)? 1578 } 1579 Operator::I64AtomicRmwCmpxchg { memarg } => { 1580 translate_atomic_cas(I64, I64, memarg, builder, environ)? 1581 } 1582 Operator::I32AtomicRmw8CmpxchgU { memarg } => { 1583 translate_atomic_cas(I32, I8, memarg, builder, environ)? 1584 } 1585 Operator::I32AtomicRmw16CmpxchgU { memarg } => { 1586 translate_atomic_cas(I32, I16, memarg, builder, environ)? 1587 } 1588 Operator::I64AtomicRmw8CmpxchgU { memarg } => { 1589 translate_atomic_cas(I64, I8, memarg, builder, environ)? 1590 } 1591 Operator::I64AtomicRmw16CmpxchgU { memarg } => { 1592 translate_atomic_cas(I64, I16, memarg, builder, environ)? 1593 } 1594 Operator::I64AtomicRmw32CmpxchgU { memarg } => { 1595 translate_atomic_cas(I64, I32, memarg, builder, environ)? 1596 } 1597 1598 Operator::AtomicFence { .. } => { 1599 builder.ins().fence(); 1600 } 1601 Operator::MemoryCopy { src_mem, dst_mem } => { 1602 let src_index = MemoryIndex::from_u32(*src_mem); 1603 let _src_heap = environ.get_or_create_heap(builder.func, src_index); 1604 1605 let dst_index = MemoryIndex::from_u32(*dst_mem); 1606 let _dst_heap = environ.get_or_create_heap(builder.func, dst_index); 1607 1608 let len = environ.stacks.pop1(); 1609 let src_pos = environ.stacks.pop1(); 1610 let dst_pos = environ.stacks.pop1(); 1611 environ.translate_memory_copy(builder, src_index, dst_index, dst_pos, src_pos, len)?; 1612 } 1613 Operator::MemoryFill { mem } => { 1614 let mem = MemoryIndex::from_u32(*mem); 1615 let _heap = environ.get_or_create_heap(builder.func, mem); 1616 let len = environ.stacks.pop1(); 1617 let val = environ.stacks.pop1(); 1618 let dest = environ.stacks.pop1(); 1619 environ.translate_memory_fill(builder, mem, dest, val, len)?; 1620 } 1621 Operator::MemoryInit { data_index, mem } => { 1622 let mem = MemoryIndex::from_u32(*mem); 1623 let _heap = environ.get_or_create_heap(builder.func, mem); 1624 let len = environ.stacks.pop1(); 1625 let src = environ.stacks.pop1(); 1626 let dest = environ.stacks.pop1(); 1627 environ.translate_memory_init(builder, mem, *data_index, dest, src, len)?; 1628 } 1629 Operator::DataDrop { data_index } => { 1630 environ.translate_data_drop(builder.cursor(), *data_index)?; 1631 } 1632 Operator::TableSize { table: index } => { 1633 let result = 1634 environ.translate_table_size(builder.cursor(), TableIndex::from_u32(*index))?; 1635 environ.stacks.push1(result); 1636 } 1637 Operator::TableGrow { table: index } => { 1638 let table_index = TableIndex::from_u32(*index); 1639 let delta = environ.stacks.pop1(); 1640 let init_value = environ.stacks.pop1(); 1641 let result = environ.translate_table_grow(builder, table_index, delta, init_value)?; 1642 environ.stacks.push1(result); 1643 } 1644 Operator::TableGet { table: index } => { 1645 let table_index = TableIndex::from_u32(*index); 1646 let index = environ.stacks.pop1(); 1647 let result = environ.translate_table_get(builder, table_index, index)?; 1648 environ.stacks.push1(result); 1649 } 1650 Operator::TableSet { table: index } => { 1651 let table_index = TableIndex::from_u32(*index); 1652 let value = environ.stacks.pop1(); 1653 let index = environ.stacks.pop1(); 1654 environ.translate_table_set(builder, table_index, value, index)?; 1655 } 1656 Operator::TableCopy { 1657 dst_table: dst_table_index, 1658 src_table: src_table_index, 1659 } => { 1660 let len = environ.stacks.pop1(); 1661 let src = environ.stacks.pop1(); 1662 let dest = environ.stacks.pop1(); 1663 environ.translate_table_copy( 1664 builder, 1665 TableIndex::from_u32(*dst_table_index), 1666 TableIndex::from_u32(*src_table_index), 1667 dest, 1668 src, 1669 len, 1670 )?; 1671 } 1672 Operator::TableFill { table } => { 1673 let table_index = TableIndex::from_u32(*table); 1674 let len = environ.stacks.pop1(); 1675 let val = environ.stacks.pop1(); 1676 let dest = environ.stacks.pop1(); 1677 environ.translate_table_fill(builder, table_index, dest, val, len)?; 1678 } 1679 Operator::TableInit { 1680 elem_index, 1681 table: table_index, 1682 } => { 1683 let len = environ.stacks.pop1(); 1684 let src = environ.stacks.pop1(); 1685 let dest = environ.stacks.pop1(); 1686 environ.translate_table_init( 1687 builder, 1688 *elem_index, 1689 TableIndex::from_u32(*table_index), 1690 dest, 1691 src, 1692 len, 1693 )?; 1694 } 1695 Operator::ElemDrop { elem_index } => { 1696 environ.translate_elem_drop(builder.cursor(), *elem_index)?; 1697 } 1698 Operator::V128Const { value } => { 1699 let data = value.bytes().to_vec().into(); 1700 let handle = builder.func.dfg.constants.insert(data); 1701 let value = builder.ins().vconst(I8X16, handle); 1702 // the v128.const is typed in CLIF as a I8x16 but bitcast to a different type 1703 // before use 1704 environ.stacks.push1(value) 1705 } 1706 Operator::I8x16Splat | Operator::I16x8Splat => { 1707 let reduced = builder 1708 .ins() 1709 .ireduce(type_of(op).lane_type(), environ.stacks.pop1()); 1710 let splatted = builder.ins().splat(type_of(op), reduced); 1711 environ.stacks.push1(splatted) 1712 } 1713 Operator::I32x4Splat 1714 | Operator::I64x2Splat 1715 | Operator::F32x4Splat 1716 | Operator::F64x2Splat => { 1717 let splatted = builder.ins().splat(type_of(op), environ.stacks.pop1()); 1718 environ.stacks.push1(splatted) 1719 } 1720 Operator::V128Load8Splat { memarg } 1721 | Operator::V128Load16Splat { memarg } 1722 | Operator::V128Load32Splat { memarg } 1723 | Operator::V128Load64Splat { memarg } => { 1724 unwrap_or_return_unreachable_state!( 1725 environ, 1726 translate_load( 1727 memarg, 1728 ir::Opcode::Load, 1729 type_of(op).lane_type(), 1730 builder, 1731 environ, 1732 )? 1733 ); 1734 let splatted = builder.ins().splat(type_of(op), environ.stacks.pop1()); 1735 environ.stacks.push1(splatted) 1736 } 1737 Operator::V128Load32Zero { memarg } | Operator::V128Load64Zero { memarg } => { 1738 unwrap_or_return_unreachable_state!( 1739 environ, 1740 translate_load( 1741 memarg, 1742 ir::Opcode::Load, 1743 type_of(op).lane_type(), 1744 builder, 1745 environ, 1746 )? 1747 ); 1748 let as_vector = builder 1749 .ins() 1750 .scalar_to_vector(type_of(op), environ.stacks.pop1()); 1751 environ.stacks.push1(as_vector) 1752 } 1753 Operator::V128Load8Lane { memarg, lane } 1754 | Operator::V128Load16Lane { memarg, lane } 1755 | Operator::V128Load32Lane { memarg, lane } 1756 | Operator::V128Load64Lane { memarg, lane } => { 1757 let vector = pop1_with_bitcast(environ, type_of(op), builder); 1758 unwrap_or_return_unreachable_state!( 1759 environ, 1760 translate_load( 1761 memarg, 1762 ir::Opcode::Load, 1763 type_of(op).lane_type(), 1764 builder, 1765 environ, 1766 )? 1767 ); 1768 let replacement = environ.stacks.pop1(); 1769 environ 1770 .stacks 1771 .push1(builder.ins().insertlane(vector, replacement, *lane)) 1772 } 1773 Operator::V128Store8Lane { memarg, lane } 1774 | Operator::V128Store16Lane { memarg, lane } 1775 | Operator::V128Store32Lane { memarg, lane } 1776 | Operator::V128Store64Lane { memarg, lane } => { 1777 let vector = pop1_with_bitcast(environ, type_of(op), builder); 1778 environ 1779 .stacks 1780 .push1(builder.ins().extractlane(vector, *lane)); 1781 translate_store(memarg, ir::Opcode::Store, builder, environ)?; 1782 } 1783 Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => { 1784 let vector = pop1_with_bitcast(environ, type_of(op), builder); 1785 let extracted = builder.ins().extractlane(vector, *lane); 1786 environ.stacks.push1(builder.ins().sextend(I32, extracted)) 1787 } 1788 Operator::I8x16ExtractLaneU { lane } | Operator::I16x8ExtractLaneU { lane } => { 1789 let vector = pop1_with_bitcast(environ, type_of(op), builder); 1790 let extracted = builder.ins().extractlane(vector, *lane); 1791 environ.stacks.push1(builder.ins().uextend(I32, extracted)); 1792 // On x86, PEXTRB zeroes the upper bits of the destination register of extractlane so 1793 // uextend could be elided; for now, uextend is needed for Cranelift's type checks to 1794 // work. 1795 } 1796 Operator::I32x4ExtractLane { lane } 1797 | Operator::I64x2ExtractLane { lane } 1798 | Operator::F32x4ExtractLane { lane } 1799 | Operator::F64x2ExtractLane { lane } => { 1800 let vector = pop1_with_bitcast(environ, type_of(op), builder); 1801 environ 1802 .stacks 1803 .push1(builder.ins().extractlane(vector, *lane)) 1804 } 1805 Operator::I8x16ReplaceLane { lane } | Operator::I16x8ReplaceLane { lane } => { 1806 let (vector, replacement) = environ.stacks.pop2(); 1807 let ty = type_of(op); 1808 let reduced = builder.ins().ireduce(ty.lane_type(), replacement); 1809 let vector = optionally_bitcast_vector(vector, ty, builder); 1810 environ 1811 .stacks 1812 .push1(builder.ins().insertlane(vector, reduced, *lane)) 1813 } 1814 Operator::I32x4ReplaceLane { lane } 1815 | Operator::I64x2ReplaceLane { lane } 1816 | Operator::F32x4ReplaceLane { lane } 1817 | Operator::F64x2ReplaceLane { lane } => { 1818 let (vector, replacement) = environ.stacks.pop2(); 1819 let vector = optionally_bitcast_vector(vector, type_of(op), builder); 1820 environ 1821 .stacks 1822 .push1(builder.ins().insertlane(vector, replacement, *lane)) 1823 } 1824 Operator::I8x16Shuffle { lanes, .. } => { 1825 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 1826 let result = environ.i8x16_shuffle(builder, a, b, lanes); 1827 environ.stacks.push1(result); 1828 // At this point the original types of a and b are lost; users of this value (i.e. this 1829 // WASM-to-CLIF translator) may need to bitcast for type-correctness. This is due 1830 // to WASM using the less specific v128 type for certain operations and more specific 1831 // types (e.g. i8x16) for others. 1832 } 1833 Operator::I8x16Swizzle => { 1834 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 1835 let result = environ.swizzle(builder, a, b); 1836 environ.stacks.push1(result); 1837 } 1838 Operator::I8x16Add | Operator::I16x8Add | Operator::I32x4Add | Operator::I64x2Add => { 1839 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1840 environ.stacks.push1(builder.ins().iadd(a, b)) 1841 } 1842 Operator::I8x16AddSatS | Operator::I16x8AddSatS => { 1843 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1844 environ.stacks.push1(builder.ins().sadd_sat(a, b)) 1845 } 1846 Operator::I8x16AddSatU | Operator::I16x8AddSatU => { 1847 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1848 environ.stacks.push1(builder.ins().uadd_sat(a, b)) 1849 } 1850 Operator::I8x16Sub | Operator::I16x8Sub | Operator::I32x4Sub | Operator::I64x2Sub => { 1851 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1852 environ.stacks.push1(builder.ins().isub(a, b)) 1853 } 1854 Operator::I8x16SubSatS | Operator::I16x8SubSatS => { 1855 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1856 environ.stacks.push1(builder.ins().ssub_sat(a, b)) 1857 } 1858 Operator::I8x16SubSatU | Operator::I16x8SubSatU => { 1859 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1860 environ.stacks.push1(builder.ins().usub_sat(a, b)) 1861 } 1862 Operator::I8x16MinS | Operator::I16x8MinS | Operator::I32x4MinS => { 1863 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1864 environ.stacks.push1(builder.ins().smin(a, b)) 1865 } 1866 Operator::I8x16MinU | Operator::I16x8MinU | Operator::I32x4MinU => { 1867 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1868 environ.stacks.push1(builder.ins().umin(a, b)) 1869 } 1870 Operator::I8x16MaxS | Operator::I16x8MaxS | Operator::I32x4MaxS => { 1871 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1872 environ.stacks.push1(builder.ins().smax(a, b)) 1873 } 1874 Operator::I8x16MaxU | Operator::I16x8MaxU | Operator::I32x4MaxU => { 1875 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1876 environ.stacks.push1(builder.ins().umax(a, b)) 1877 } 1878 Operator::I8x16AvgrU | Operator::I16x8AvgrU => { 1879 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1880 environ.stacks.push1(builder.ins().avg_round(a, b)) 1881 } 1882 Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => { 1883 let a = pop1_with_bitcast(environ, type_of(op), builder); 1884 environ.stacks.push1(builder.ins().ineg(a)) 1885 } 1886 Operator::I8x16Abs | Operator::I16x8Abs | Operator::I32x4Abs | Operator::I64x2Abs => { 1887 let a = pop1_with_bitcast(environ, type_of(op), builder); 1888 environ.stacks.push1(builder.ins().iabs(a)) 1889 } 1890 Operator::I16x8Mul | Operator::I32x4Mul | Operator::I64x2Mul => { 1891 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1892 environ.stacks.push1(builder.ins().imul(a, b)) 1893 } 1894 Operator::V128Or => { 1895 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1896 environ.stacks.push1(builder.ins().bor(a, b)) 1897 } 1898 Operator::V128Xor => { 1899 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1900 environ.stacks.push1(builder.ins().bxor(a, b)) 1901 } 1902 Operator::V128And => { 1903 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1904 environ.stacks.push1(builder.ins().band(a, b)) 1905 } 1906 Operator::V128AndNot => { 1907 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 1908 environ.stacks.push1(builder.ins().band_not(a, b)) 1909 } 1910 Operator::V128Not => { 1911 let a = environ.stacks.pop1(); 1912 environ.stacks.push1(builder.ins().bnot(a)); 1913 } 1914 Operator::I8x16Shl | Operator::I16x8Shl | Operator::I32x4Shl | Operator::I64x2Shl => { 1915 let (a, b) = environ.stacks.pop2(); 1916 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1917 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1918 // with cranelift's instruction. 1919 environ.stacks.push1(builder.ins().ishl(bitcast_a, b)) 1920 } 1921 Operator::I8x16ShrU | Operator::I16x8ShrU | Operator::I32x4ShrU | Operator::I64x2ShrU => { 1922 let (a, b) = environ.stacks.pop2(); 1923 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1924 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1925 // with cranelift's instruction. 1926 environ.stacks.push1(builder.ins().ushr(bitcast_a, b)) 1927 } 1928 Operator::I8x16ShrS | Operator::I16x8ShrS | Operator::I32x4ShrS | Operator::I64x2ShrS => { 1929 let (a, b) = environ.stacks.pop2(); 1930 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1931 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1932 // with cranelift's instruction. 1933 environ.stacks.push1(builder.ins().sshr(bitcast_a, b)) 1934 } 1935 Operator::V128Bitselect => { 1936 let (a, b, c) = pop3_with_bitcast(environ, I8X16, builder); 1937 // The CLIF operand ordering is slightly different and the types of all three 1938 // operands must match (hence the bitcast). 1939 environ.stacks.push1(builder.ins().bitselect(c, a, b)) 1940 } 1941 Operator::V128AnyTrue => { 1942 let a = pop1_with_bitcast(environ, type_of(op), builder); 1943 let bool_result = builder.ins().vany_true(a); 1944 environ 1945 .stacks 1946 .push1(builder.ins().uextend(I32, bool_result)) 1947 } 1948 Operator::I8x16AllTrue 1949 | Operator::I16x8AllTrue 1950 | Operator::I32x4AllTrue 1951 | Operator::I64x2AllTrue => { 1952 let a = pop1_with_bitcast(environ, type_of(op), builder); 1953 let bool_result = builder.ins().vall_true(a); 1954 environ 1955 .stacks 1956 .push1(builder.ins().uextend(I32, bool_result)) 1957 } 1958 Operator::I8x16Bitmask 1959 | Operator::I16x8Bitmask 1960 | Operator::I32x4Bitmask 1961 | Operator::I64x2Bitmask => { 1962 let a = pop1_with_bitcast(environ, type_of(op), builder); 1963 environ.stacks.push1(builder.ins().vhigh_bits(I32, a)); 1964 } 1965 Operator::I8x16Eq | Operator::I16x8Eq | Operator::I32x4Eq | Operator::I64x2Eq => { 1966 translate_vector_icmp(IntCC::Equal, type_of(op), builder, environ) 1967 } 1968 Operator::I8x16Ne | Operator::I16x8Ne | Operator::I32x4Ne | Operator::I64x2Ne => { 1969 translate_vector_icmp(IntCC::NotEqual, type_of(op), builder, environ) 1970 } 1971 Operator::I8x16GtS | Operator::I16x8GtS | Operator::I32x4GtS | Operator::I64x2GtS => { 1972 translate_vector_icmp(IntCC::SignedGreaterThan, type_of(op), builder, environ) 1973 } 1974 Operator::I8x16LtS | Operator::I16x8LtS | Operator::I32x4LtS | Operator::I64x2LtS => { 1975 translate_vector_icmp(IntCC::SignedLessThan, type_of(op), builder, environ) 1976 } 1977 Operator::I8x16GtU | Operator::I16x8GtU | Operator::I32x4GtU => { 1978 translate_vector_icmp(IntCC::UnsignedGreaterThan, type_of(op), builder, environ) 1979 } 1980 Operator::I8x16LtU | Operator::I16x8LtU | Operator::I32x4LtU => { 1981 translate_vector_icmp(IntCC::UnsignedLessThan, type_of(op), builder, environ) 1982 } 1983 Operator::I8x16GeS | Operator::I16x8GeS | Operator::I32x4GeS | Operator::I64x2GeS => { 1984 translate_vector_icmp( 1985 IntCC::SignedGreaterThanOrEqual, 1986 type_of(op), 1987 builder, 1988 environ, 1989 ) 1990 } 1991 Operator::I8x16LeS | Operator::I16x8LeS | Operator::I32x4LeS | Operator::I64x2LeS => { 1992 translate_vector_icmp(IntCC::SignedLessThanOrEqual, type_of(op), builder, environ) 1993 } 1994 Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => translate_vector_icmp( 1995 IntCC::UnsignedGreaterThanOrEqual, 1996 type_of(op), 1997 builder, 1998 environ, 1999 ), 2000 Operator::I8x16LeU | Operator::I16x8LeU | Operator::I32x4LeU => translate_vector_icmp( 2001 IntCC::UnsignedLessThanOrEqual, 2002 type_of(op), 2003 builder, 2004 environ, 2005 ), 2006 Operator::F32x4Eq | Operator::F64x2Eq => { 2007 translate_vector_fcmp(FloatCC::Equal, type_of(op), builder, environ) 2008 } 2009 Operator::F32x4Ne | Operator::F64x2Ne => { 2010 translate_vector_fcmp(FloatCC::NotEqual, type_of(op), builder, environ) 2011 } 2012 Operator::F32x4Lt | Operator::F64x2Lt => { 2013 translate_vector_fcmp(FloatCC::LessThan, type_of(op), builder, environ) 2014 } 2015 Operator::F32x4Gt | Operator::F64x2Gt => { 2016 translate_vector_fcmp(FloatCC::GreaterThan, type_of(op), builder, environ) 2017 } 2018 Operator::F32x4Le | Operator::F64x2Le => { 2019 translate_vector_fcmp(FloatCC::LessThanOrEqual, type_of(op), builder, environ) 2020 } 2021 Operator::F32x4Ge | Operator::F64x2Ge => { 2022 translate_vector_fcmp(FloatCC::GreaterThanOrEqual, type_of(op), builder, environ) 2023 } 2024 Operator::F32x4Add | Operator::F64x2Add => { 2025 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2026 environ.stacks.push1(builder.ins().fadd(a, b)) 2027 } 2028 Operator::F32x4Sub | Operator::F64x2Sub => { 2029 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2030 environ.stacks.push1(builder.ins().fsub(a, b)) 2031 } 2032 Operator::F32x4Mul | Operator::F64x2Mul => { 2033 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2034 environ.stacks.push1(builder.ins().fmul(a, b)) 2035 } 2036 Operator::F32x4Div | Operator::F64x2Div => { 2037 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2038 environ.stacks.push1(builder.ins().fdiv(a, b)) 2039 } 2040 Operator::F32x4Max | Operator::F64x2Max => { 2041 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2042 environ.stacks.push1(builder.ins().fmax(a, b)) 2043 } 2044 Operator::F32x4Min | Operator::F64x2Min => { 2045 let (a, b) = pop2_with_bitcast(environ, type_of(op), builder); 2046 environ.stacks.push1(builder.ins().fmin(a, b)) 2047 } 2048 Operator::F32x4PMax | Operator::F64x2PMax => { 2049 // Note the careful ordering here with respect to `fcmp` and 2050 // `bitselect`. This matches the spec definition of: 2051 // 2052 // fpmax(z1, z2) = 2053 // * If z1 is less than z2 then return z2. 2054 // * Else return z1. 2055 let ty = type_of(op); 2056 let (a, b) = pop2_with_bitcast(environ, ty, builder); 2057 let cmp = builder.ins().fcmp(FloatCC::LessThan, a, b); 2058 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2059 environ.stacks.push1(builder.ins().bitselect(cmp, b, a)) 2060 } 2061 Operator::F32x4PMin | Operator::F64x2PMin => { 2062 // Note the careful ordering here which is similar to `pmax` above: 2063 // 2064 // fpmin(z1, z2) = 2065 // * If z2 is less than z1 then return z2. 2066 // * Else return z1. 2067 let ty = type_of(op); 2068 let (a, b) = pop2_with_bitcast(environ, ty, builder); 2069 let cmp = builder.ins().fcmp(FloatCC::LessThan, b, a); 2070 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2071 environ.stacks.push1(builder.ins().bitselect(cmp, b, a)) 2072 } 2073 Operator::F32x4Sqrt | Operator::F64x2Sqrt => { 2074 let a = pop1_with_bitcast(environ, type_of(op), builder); 2075 environ.stacks.push1(builder.ins().sqrt(a)) 2076 } 2077 Operator::F32x4Neg | Operator::F64x2Neg => { 2078 let a = pop1_with_bitcast(environ, type_of(op), builder); 2079 environ.stacks.push1(builder.ins().fneg(a)) 2080 } 2081 Operator::F32x4Abs | Operator::F64x2Abs => { 2082 let a = pop1_with_bitcast(environ, type_of(op), builder); 2083 environ.stacks.push1(builder.ins().fabs(a)) 2084 } 2085 Operator::F32x4ConvertI32x4S => { 2086 let a = pop1_with_bitcast(environ, I32X4, builder); 2087 environ.stacks.push1(builder.ins().fcvt_from_sint(F32X4, a)) 2088 } 2089 Operator::F32x4ConvertI32x4U => { 2090 let a = pop1_with_bitcast(environ, I32X4, builder); 2091 environ.stacks.push1(builder.ins().fcvt_from_uint(F32X4, a)) 2092 } 2093 Operator::F64x2ConvertLowI32x4S => { 2094 let a = pop1_with_bitcast(environ, I32X4, builder); 2095 let widened_a = builder.ins().swiden_low(a); 2096 environ 2097 .stacks 2098 .push1(builder.ins().fcvt_from_sint(F64X2, widened_a)); 2099 } 2100 Operator::F64x2ConvertLowI32x4U => { 2101 let a = pop1_with_bitcast(environ, I32X4, builder); 2102 let widened_a = builder.ins().uwiden_low(a); 2103 environ 2104 .stacks 2105 .push1(builder.ins().fcvt_from_uint(F64X2, widened_a)); 2106 } 2107 Operator::F64x2PromoteLowF32x4 => { 2108 let a = pop1_with_bitcast(environ, F32X4, builder); 2109 environ.stacks.push1(builder.ins().fvpromote_low(a)); 2110 } 2111 Operator::F32x4DemoteF64x2Zero => { 2112 let a = pop1_with_bitcast(environ, F64X2, builder); 2113 environ.stacks.push1(builder.ins().fvdemote(a)); 2114 } 2115 Operator::I32x4TruncSatF32x4S => { 2116 let a = pop1_with_bitcast(environ, F32X4, builder); 2117 environ 2118 .stacks 2119 .push1(builder.ins().fcvt_to_sint_sat(I32X4, a)) 2120 } 2121 Operator::I32x4TruncSatF64x2SZero => { 2122 let a = pop1_with_bitcast(environ, F64X2, builder); 2123 let converted_a = builder.ins().fcvt_to_sint_sat(I64X2, a); 2124 let handle = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2125 let zero = builder.ins().vconst(I64X2, handle); 2126 2127 environ 2128 .stacks 2129 .push1(builder.ins().snarrow(converted_a, zero)); 2130 } 2131 2132 // FIXME(#5913): the relaxed instructions here are translated the same 2133 // as the saturating instructions, even when the code generator 2134 // configuration allow for different semantics across hosts. On x86, 2135 // however, it's theoretically possible to have a slightly more optimal 2136 // lowering which accounts for NaN differently, although the lowering is 2137 // still not trivial (e.g. one instruction). At this time the 2138 // more-optimal-but-still-large lowering for x86 is not implemented so 2139 // the relaxed instructions are listed here instead of down below with 2140 // the other relaxed instructions. An x86-specific implementation (or 2141 // perhaps for other backends too) should be added and the codegen for 2142 // the relaxed instruction should conditionally be different. 2143 Operator::I32x4RelaxedTruncF32x4U | Operator::I32x4TruncSatF32x4U => { 2144 let a = pop1_with_bitcast(environ, F32X4, builder); 2145 environ 2146 .stacks 2147 .push1(builder.ins().fcvt_to_uint_sat(I32X4, a)) 2148 } 2149 Operator::I32x4RelaxedTruncF64x2UZero | Operator::I32x4TruncSatF64x2UZero => { 2150 let a = pop1_with_bitcast(environ, F64X2, builder); 2151 let zero_constant = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2152 let result = if environ.is_x86() && !environ.isa().has_round() { 2153 // On x86 the vector lowering for `fcvt_to_uint_sat` requires 2154 // SSE4.1 `round` instructions. If SSE4.1 isn't available it 2155 // falls back to a libcall which we don't want in Wasmtime. 2156 // Handle this by falling back to the scalar implementation 2157 // which does not require SSE4.1 instructions. 2158 let lane0 = builder.ins().extractlane(a, 0); 2159 let lane1 = builder.ins().extractlane(a, 1); 2160 let lane0_rounded = builder.ins().fcvt_to_uint_sat(I32, lane0); 2161 let lane1_rounded = builder.ins().fcvt_to_uint_sat(I32, lane1); 2162 let result = builder.ins().vconst(I32X4, zero_constant); 2163 let result = builder.ins().insertlane(result, lane0_rounded, 0); 2164 builder.ins().insertlane(result, lane1_rounded, 1) 2165 } else { 2166 let converted_a = builder.ins().fcvt_to_uint_sat(I64X2, a); 2167 let zero = builder.ins().vconst(I64X2, zero_constant); 2168 builder.ins().uunarrow(converted_a, zero) 2169 }; 2170 environ.stacks.push1(result); 2171 } 2172 2173 Operator::I8x16NarrowI16x8S => { 2174 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2175 environ.stacks.push1(builder.ins().snarrow(a, b)) 2176 } 2177 Operator::I16x8NarrowI32x4S => { 2178 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2179 environ.stacks.push1(builder.ins().snarrow(a, b)) 2180 } 2181 Operator::I8x16NarrowI16x8U => { 2182 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2183 environ.stacks.push1(builder.ins().unarrow(a, b)) 2184 } 2185 Operator::I16x8NarrowI32x4U => { 2186 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2187 environ.stacks.push1(builder.ins().unarrow(a, b)) 2188 } 2189 Operator::I16x8ExtendLowI8x16S => { 2190 let a = pop1_with_bitcast(environ, I8X16, builder); 2191 environ.stacks.push1(builder.ins().swiden_low(a)) 2192 } 2193 Operator::I16x8ExtendHighI8x16S => { 2194 let a = pop1_with_bitcast(environ, I8X16, builder); 2195 environ.stacks.push1(builder.ins().swiden_high(a)) 2196 } 2197 Operator::I16x8ExtendLowI8x16U => { 2198 let a = pop1_with_bitcast(environ, I8X16, builder); 2199 environ.stacks.push1(builder.ins().uwiden_low(a)) 2200 } 2201 Operator::I16x8ExtendHighI8x16U => { 2202 let a = pop1_with_bitcast(environ, I8X16, builder); 2203 environ.stacks.push1(builder.ins().uwiden_high(a)) 2204 } 2205 Operator::I32x4ExtendLowI16x8S => { 2206 let a = pop1_with_bitcast(environ, I16X8, builder); 2207 environ.stacks.push1(builder.ins().swiden_low(a)) 2208 } 2209 Operator::I32x4ExtendHighI16x8S => { 2210 let a = pop1_with_bitcast(environ, I16X8, builder); 2211 environ.stacks.push1(builder.ins().swiden_high(a)) 2212 } 2213 Operator::I32x4ExtendLowI16x8U => { 2214 let a = pop1_with_bitcast(environ, I16X8, builder); 2215 environ.stacks.push1(builder.ins().uwiden_low(a)) 2216 } 2217 Operator::I32x4ExtendHighI16x8U => { 2218 let a = pop1_with_bitcast(environ, I16X8, builder); 2219 environ.stacks.push1(builder.ins().uwiden_high(a)) 2220 } 2221 Operator::I64x2ExtendLowI32x4S => { 2222 let a = pop1_with_bitcast(environ, I32X4, builder); 2223 environ.stacks.push1(builder.ins().swiden_low(a)) 2224 } 2225 Operator::I64x2ExtendHighI32x4S => { 2226 let a = pop1_with_bitcast(environ, I32X4, builder); 2227 environ.stacks.push1(builder.ins().swiden_high(a)) 2228 } 2229 Operator::I64x2ExtendLowI32x4U => { 2230 let a = pop1_with_bitcast(environ, I32X4, builder); 2231 environ.stacks.push1(builder.ins().uwiden_low(a)) 2232 } 2233 Operator::I64x2ExtendHighI32x4U => { 2234 let a = pop1_with_bitcast(environ, I32X4, builder); 2235 environ.stacks.push1(builder.ins().uwiden_high(a)) 2236 } 2237 Operator::I16x8ExtAddPairwiseI8x16S => { 2238 let a = pop1_with_bitcast(environ, I8X16, builder); 2239 let widen_low = builder.ins().swiden_low(a); 2240 let widen_high = builder.ins().swiden_high(a); 2241 environ 2242 .stacks 2243 .push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2244 } 2245 Operator::I32x4ExtAddPairwiseI16x8S => { 2246 let a = pop1_with_bitcast(environ, I16X8, builder); 2247 let widen_low = builder.ins().swiden_low(a); 2248 let widen_high = builder.ins().swiden_high(a); 2249 environ 2250 .stacks 2251 .push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2252 } 2253 Operator::I16x8ExtAddPairwiseI8x16U => { 2254 let a = pop1_with_bitcast(environ, I8X16, builder); 2255 let widen_low = builder.ins().uwiden_low(a); 2256 let widen_high = builder.ins().uwiden_high(a); 2257 environ 2258 .stacks 2259 .push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2260 } 2261 Operator::I32x4ExtAddPairwiseI16x8U => { 2262 let a = pop1_with_bitcast(environ, I16X8, builder); 2263 let widen_low = builder.ins().uwiden_low(a); 2264 let widen_high = builder.ins().uwiden_high(a); 2265 environ 2266 .stacks 2267 .push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2268 } 2269 Operator::F32x4Ceil => { 2270 let arg = pop1_with_bitcast(environ, F32X4, builder); 2271 let result = environ.ceil_f32x4(builder, arg); 2272 environ.stacks.push1(result); 2273 } 2274 Operator::F64x2Ceil => { 2275 let arg = pop1_with_bitcast(environ, F64X2, builder); 2276 let result = environ.ceil_f64x2(builder, arg); 2277 environ.stacks.push1(result); 2278 } 2279 Operator::F32x4Floor => { 2280 let arg = pop1_with_bitcast(environ, F32X4, builder); 2281 let result = environ.floor_f32x4(builder, arg); 2282 environ.stacks.push1(result); 2283 } 2284 Operator::F64x2Floor => { 2285 let arg = pop1_with_bitcast(environ, F64X2, builder); 2286 let result = environ.floor_f64x2(builder, arg); 2287 environ.stacks.push1(result); 2288 } 2289 Operator::F32x4Trunc => { 2290 let arg = pop1_with_bitcast(environ, F32X4, builder); 2291 let result = environ.trunc_f32x4(builder, arg); 2292 environ.stacks.push1(result); 2293 } 2294 Operator::F64x2Trunc => { 2295 let arg = pop1_with_bitcast(environ, F64X2, builder); 2296 let result = environ.trunc_f64x2(builder, arg); 2297 environ.stacks.push1(result); 2298 } 2299 Operator::F32x4Nearest => { 2300 let arg = pop1_with_bitcast(environ, F32X4, builder); 2301 let result = environ.nearest_f32x4(builder, arg); 2302 environ.stacks.push1(result); 2303 } 2304 Operator::F64x2Nearest => { 2305 let arg = pop1_with_bitcast(environ, F64X2, builder); 2306 let result = environ.nearest_f64x2(builder, arg); 2307 environ.stacks.push1(result); 2308 } 2309 Operator::I32x4DotI16x8S => { 2310 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2311 let alow = builder.ins().swiden_low(a); 2312 let blow = builder.ins().swiden_low(b); 2313 let low = builder.ins().imul(alow, blow); 2314 let ahigh = builder.ins().swiden_high(a); 2315 let bhigh = builder.ins().swiden_high(b); 2316 let high = builder.ins().imul(ahigh, bhigh); 2317 environ.stacks.push1(builder.ins().iadd_pairwise(low, high)); 2318 } 2319 Operator::I8x16Popcnt => { 2320 let arg = pop1_with_bitcast(environ, type_of(op), builder); 2321 environ.stacks.push1(builder.ins().popcnt(arg)); 2322 } 2323 Operator::I16x8Q15MulrSatS => { 2324 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2325 environ.stacks.push1(builder.ins().sqmul_round_sat(a, b)) 2326 } 2327 Operator::I16x8ExtMulLowI8x16S => { 2328 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2329 let a_low = builder.ins().swiden_low(a); 2330 let b_low = builder.ins().swiden_low(b); 2331 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2332 } 2333 Operator::I16x8ExtMulHighI8x16S => { 2334 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2335 let a_high = builder.ins().swiden_high(a); 2336 let b_high = builder.ins().swiden_high(b); 2337 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2338 } 2339 Operator::I16x8ExtMulLowI8x16U => { 2340 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2341 let a_low = builder.ins().uwiden_low(a); 2342 let b_low = builder.ins().uwiden_low(b); 2343 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2344 } 2345 Operator::I16x8ExtMulHighI8x16U => { 2346 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2347 let a_high = builder.ins().uwiden_high(a); 2348 let b_high = builder.ins().uwiden_high(b); 2349 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2350 } 2351 Operator::I32x4ExtMulLowI16x8S => { 2352 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2353 let a_low = builder.ins().swiden_low(a); 2354 let b_low = builder.ins().swiden_low(b); 2355 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2356 } 2357 Operator::I32x4ExtMulHighI16x8S => { 2358 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2359 let a_high = builder.ins().swiden_high(a); 2360 let b_high = builder.ins().swiden_high(b); 2361 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2362 } 2363 Operator::I32x4ExtMulLowI16x8U => { 2364 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2365 let a_low = builder.ins().uwiden_low(a); 2366 let b_low = builder.ins().uwiden_low(b); 2367 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2368 } 2369 Operator::I32x4ExtMulHighI16x8U => { 2370 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2371 let a_high = builder.ins().uwiden_high(a); 2372 let b_high = builder.ins().uwiden_high(b); 2373 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2374 } 2375 Operator::I64x2ExtMulLowI32x4S => { 2376 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2377 let a_low = builder.ins().swiden_low(a); 2378 let b_low = builder.ins().swiden_low(b); 2379 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2380 } 2381 Operator::I64x2ExtMulHighI32x4S => { 2382 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2383 let a_high = builder.ins().swiden_high(a); 2384 let b_high = builder.ins().swiden_high(b); 2385 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2386 } 2387 Operator::I64x2ExtMulLowI32x4U => { 2388 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2389 let a_low = builder.ins().uwiden_low(a); 2390 let b_low = builder.ins().uwiden_low(b); 2391 environ.stacks.push1(builder.ins().imul(a_low, b_low)); 2392 } 2393 Operator::I64x2ExtMulHighI32x4U => { 2394 let (a, b) = pop2_with_bitcast(environ, I32X4, builder); 2395 let a_high = builder.ins().uwiden_high(a); 2396 let b_high = builder.ins().uwiden_high(b); 2397 environ.stacks.push1(builder.ins().imul(a_high, b_high)); 2398 } 2399 Operator::MemoryDiscard { .. } => { 2400 return Err(wasm_unsupported!( 2401 "proposed memory-control operator {:?}", 2402 op 2403 )); 2404 } 2405 2406 Operator::F32x4RelaxedMax | Operator::F64x2RelaxedMax => { 2407 let ty = type_of(op); 2408 let (a, b) = pop2_with_bitcast(environ, ty, builder); 2409 environ.stacks.push1( 2410 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2411 // Deterministic semantics match the `fmax` instruction, or 2412 // the `fAAxBB.max` wasm instruction. 2413 builder.ins().fmax(a, b) 2414 } else { 2415 // Note that this matches the `pmax` translation which has 2416 // careful ordering of its operands to trigger 2417 // pattern-matches in the x86 backend. 2418 let cmp = builder.ins().fcmp(FloatCC::LessThan, a, b); 2419 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2420 builder.ins().bitselect(cmp, b, a) 2421 }, 2422 ) 2423 } 2424 2425 Operator::F32x4RelaxedMin | Operator::F64x2RelaxedMin => { 2426 let ty = type_of(op); 2427 let (a, b) = pop2_with_bitcast(environ, ty, builder); 2428 environ.stacks.push1( 2429 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2430 // Deterministic semantics match the `fmin` instruction, or 2431 // the `fAAxBB.min` wasm instruction. 2432 builder.ins().fmin(a, b) 2433 } else { 2434 // Note that this matches the `pmin` translation which has 2435 // careful ordering of its operands to trigger 2436 // pattern-matches in the x86 backend. 2437 let cmp = builder.ins().fcmp(FloatCC::LessThan, b, a); 2438 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2439 builder.ins().bitselect(cmp, b, a) 2440 }, 2441 ); 2442 } 2443 2444 Operator::I8x16RelaxedSwizzle => { 2445 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2446 let result = environ.relaxed_swizzle(builder, a, b); 2447 environ.stacks.push1(result); 2448 } 2449 2450 Operator::F32x4RelaxedMadd => { 2451 let (a, b, c) = pop3_with_bitcast(environ, type_of(op), builder); 2452 let result = environ.fma_f32x4(builder, a, b, c); 2453 environ.stacks.push1(result); 2454 } 2455 Operator::F64x2RelaxedMadd => { 2456 let (a, b, c) = pop3_with_bitcast(environ, type_of(op), builder); 2457 let result = environ.fma_f64x2(builder, a, b, c); 2458 environ.stacks.push1(result); 2459 } 2460 Operator::F32x4RelaxedNmadd => { 2461 let (a, b, c) = pop3_with_bitcast(environ, type_of(op), builder); 2462 let a = builder.ins().fneg(a); 2463 let result = environ.fma_f32x4(builder, a, b, c); 2464 environ.stacks.push1(result); 2465 } 2466 Operator::F64x2RelaxedNmadd => { 2467 let (a, b, c) = pop3_with_bitcast(environ, type_of(op), builder); 2468 let a = builder.ins().fneg(a); 2469 let result = environ.fma_f64x2(builder, a, b, c); 2470 environ.stacks.push1(result); 2471 } 2472 2473 Operator::I8x16RelaxedLaneselect 2474 | Operator::I16x8RelaxedLaneselect 2475 | Operator::I32x4RelaxedLaneselect 2476 | Operator::I64x2RelaxedLaneselect => { 2477 let ty = type_of(op); 2478 let (a, b, c) = pop3_with_bitcast(environ, ty, builder); 2479 // Note that the variable swaps here are intentional due to 2480 // the difference of the order of the wasm op and the clif 2481 // op. 2482 environ.stacks.push1( 2483 if environ.relaxed_simd_deterministic() 2484 || !environ.use_blendv_for_relaxed_laneselect(ty) 2485 { 2486 // Deterministic semantics are a `bitselect` along the lines 2487 // of the wasm `v128.bitselect` instruction. 2488 builder.ins().bitselect(c, a, b) 2489 } else { 2490 builder.ins().blendv(c, a, b) 2491 }, 2492 ); 2493 } 2494 2495 Operator::I32x4RelaxedTruncF32x4S => { 2496 let a = pop1_with_bitcast(environ, F32X4, builder); 2497 environ.stacks.push1( 2498 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2499 // Deterministic semantics are to match the 2500 // `i32x4.trunc_sat_f32x4_s` instruction. 2501 builder.ins().fcvt_to_sint_sat(I32X4, a) 2502 } else { 2503 builder.ins().x86_cvtt2dq(I32X4, a) 2504 }, 2505 ) 2506 } 2507 Operator::I32x4RelaxedTruncF64x2SZero => { 2508 let a = pop1_with_bitcast(environ, F64X2, builder); 2509 let converted_a = if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2510 // Deterministic semantics are to match the 2511 // `i32x4.trunc_sat_f64x2_s_zero` instruction. 2512 builder.ins().fcvt_to_sint_sat(I64X2, a) 2513 } else { 2514 builder.ins().x86_cvtt2dq(I64X2, a) 2515 }; 2516 let handle = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2517 let zero = builder.ins().vconst(I64X2, handle); 2518 2519 environ 2520 .stacks 2521 .push1(builder.ins().snarrow(converted_a, zero)); 2522 } 2523 Operator::I16x8RelaxedQ15mulrS => { 2524 let (a, b) = pop2_with_bitcast(environ, I16X8, builder); 2525 environ.stacks.push1( 2526 if environ.relaxed_simd_deterministic() 2527 || !environ.use_x86_pmulhrsw_for_relaxed_q15mul() 2528 { 2529 // Deterministic semantics are to match the 2530 // `i16x8.q15mulr_sat_s` instruction. 2531 builder.ins().sqmul_round_sat(a, b) 2532 } else { 2533 builder.ins().x86_pmulhrsw(a, b) 2534 }, 2535 ); 2536 } 2537 Operator::I16x8RelaxedDotI8x16I7x16S => { 2538 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2539 environ.stacks.push1( 2540 if environ.relaxed_simd_deterministic() || !environ.use_x86_pmaddubsw_for_dot() { 2541 // Deterministic semantics are to treat both operands as 2542 // signed integers and perform the dot product. 2543 let alo = builder.ins().swiden_low(a); 2544 let blo = builder.ins().swiden_low(b); 2545 let lo = builder.ins().imul(alo, blo); 2546 let ahi = builder.ins().swiden_high(a); 2547 let bhi = builder.ins().swiden_high(b); 2548 let hi = builder.ins().imul(ahi, bhi); 2549 builder.ins().iadd_pairwise(lo, hi) 2550 } else { 2551 builder.ins().x86_pmaddubsw(a, b) 2552 }, 2553 ); 2554 } 2555 2556 Operator::I32x4RelaxedDotI8x16I7x16AddS => { 2557 let c = pop1_with_bitcast(environ, I32X4, builder); 2558 let (a, b) = pop2_with_bitcast(environ, I8X16, builder); 2559 let dot = 2560 if environ.relaxed_simd_deterministic() || !environ.use_x86_pmaddubsw_for_dot() { 2561 // Deterministic semantics are to treat both operands as 2562 // signed integers and perform the dot product. 2563 let alo = builder.ins().swiden_low(a); 2564 let blo = builder.ins().swiden_low(b); 2565 let lo = builder.ins().imul(alo, blo); 2566 let ahi = builder.ins().swiden_high(a); 2567 let bhi = builder.ins().swiden_high(b); 2568 let hi = builder.ins().imul(ahi, bhi); 2569 builder.ins().iadd_pairwise(lo, hi) 2570 } else { 2571 builder.ins().x86_pmaddubsw(a, b) 2572 }; 2573 let dotlo = builder.ins().swiden_low(dot); 2574 let dothi = builder.ins().swiden_high(dot); 2575 let dot32 = builder.ins().iadd_pairwise(dotlo, dothi); 2576 environ.stacks.push1(builder.ins().iadd(dot32, c)); 2577 } 2578 2579 Operator::BrOnNull { relative_depth } => { 2580 let r = environ.stacks.pop1(); 2581 let &[.., WasmValType::Ref(r_ty)] = operand_types else { 2582 unreachable!("validation") 2583 }; 2584 let is_null = environ.translate_ref_is_null(builder.cursor(), r, r_ty)?; 2585 let (br_destination, inputs) = translate_br_if_args(*relative_depth, environ); 2586 let else_block = builder.create_block(); 2587 canonicalise_brif(builder, is_null, br_destination, inputs, else_block, &[]); 2588 2589 builder.seal_block(else_block); // The only predecessor is the current block. 2590 builder.switch_to_block(else_block); 2591 environ.stacks.push1(r); 2592 } 2593 Operator::BrOnNonNull { relative_depth } => { 2594 // We write this a bit differently from the spec to avoid an extra 2595 // block/branch and the typed accounting thereof. Instead of the 2596 // spec's approach, it's described as such: 2597 // Peek the value val from the stack. 2598 // If val is ref.null ht, then: pop the value val from the stack. 2599 // Else: Execute the instruction (br relative_depth). 2600 let r = environ.stacks.peek1(); 2601 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2602 unreachable!("validation") 2603 }; 2604 let r_ty = *r_ty; 2605 let (br_destination, inputs) = translate_br_if_args(*relative_depth, environ); 2606 let inputs = inputs.to_vec(); 2607 let is_null = environ.translate_ref_is_null(builder.cursor(), r, r_ty)?; 2608 let else_block = builder.create_block(); 2609 canonicalise_brif(builder, is_null, else_block, &[], br_destination, &inputs); 2610 2611 // In the null case, pop the ref 2612 environ.stacks.pop1(); 2613 2614 builder.seal_block(else_block); // The only predecessor is the current block. 2615 2616 // The rest of the translation operates on our is null case, which is 2617 // currently an empty block 2618 builder.switch_to_block(else_block); 2619 } 2620 Operator::CallRef { type_index } => { 2621 // Get function signature 2622 // `index` is the index of the function's signature and `table_index` is the index of 2623 // the table to search the function in. 2624 let type_index = TypeIndex::from_u32(*type_index); 2625 let sigref = environ.get_or_create_sig_ref(builder.func, type_index); 2626 let num_args = environ.num_params_for_function_type(type_index); 2627 let callee = environ.stacks.pop1(); 2628 2629 // Bitcast any vector arguments to their default type, I8X16, before calling. 2630 let mut args = environ.stacks.peekn(num_args).to_vec(); 2631 bitcast_wasm_params(environ, sigref, &mut args, builder); 2632 2633 let inst_results = 2634 environ.translate_call_ref(builder, srcloc, sigref, callee, &args)?; 2635 2636 debug_assert_eq!( 2637 inst_results.len(), 2638 builder.func.dfg.signatures[sigref].returns.len(), 2639 "translate_call_ref results should match the call signature" 2640 ); 2641 environ.stacks.popn(num_args); 2642 environ.stacks.pushn(&inst_results); 2643 } 2644 Operator::RefAsNonNull => { 2645 let r = environ.stacks.pop1(); 2646 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2647 unreachable!("validation") 2648 }; 2649 let is_null = environ.translate_ref_is_null(builder.cursor(), r, *r_ty)?; 2650 environ.trapnz(builder, is_null, crate::TRAP_NULL_REFERENCE); 2651 environ.stacks.push1(r); 2652 } 2653 2654 Operator::RefI31 => { 2655 let val = environ.stacks.pop1(); 2656 let i31ref = environ.translate_ref_i31(builder.cursor(), val)?; 2657 environ.stacks.push1(i31ref); 2658 } 2659 Operator::I31GetS => { 2660 let i31ref = environ.stacks.pop1(); 2661 let val = environ.translate_i31_get_s(builder, i31ref)?; 2662 environ.stacks.push1(val); 2663 } 2664 Operator::I31GetU => { 2665 let i31ref = environ.stacks.pop1(); 2666 let val = environ.translate_i31_get_u(builder, i31ref)?; 2667 environ.stacks.push1(val); 2668 } 2669 2670 Operator::StructNew { struct_type_index } => { 2671 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2672 let arity = environ.struct_fields_len(struct_type_index)?; 2673 let fields: StructFieldsVec = environ.stacks.peekn(arity).iter().copied().collect(); 2674 environ.stacks.popn(arity); 2675 let struct_ref = environ.translate_struct_new(builder, struct_type_index, fields)?; 2676 environ.stacks.push1(struct_ref); 2677 } 2678 2679 Operator::StructNewDefault { struct_type_index } => { 2680 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2681 let struct_ref = environ.translate_struct_new_default(builder, struct_type_index)?; 2682 environ.stacks.push1(struct_ref); 2683 } 2684 2685 Operator::StructSet { 2686 struct_type_index, 2687 field_index, 2688 } => { 2689 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2690 let val = environ.stacks.pop1(); 2691 let struct_ref = environ.stacks.pop1(); 2692 environ.translate_struct_set( 2693 builder, 2694 struct_type_index, 2695 *field_index, 2696 struct_ref, 2697 val, 2698 )?; 2699 } 2700 2701 Operator::StructGetS { 2702 struct_type_index, 2703 field_index, 2704 } => { 2705 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2706 let struct_ref = environ.stacks.pop1(); 2707 let val = environ.translate_struct_get( 2708 builder, 2709 struct_type_index, 2710 *field_index, 2711 struct_ref, 2712 Some(Extension::Sign), 2713 )?; 2714 environ.stacks.push1(val); 2715 } 2716 2717 Operator::StructGetU { 2718 struct_type_index, 2719 field_index, 2720 } => { 2721 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2722 let struct_ref = environ.stacks.pop1(); 2723 let val = environ.translate_struct_get( 2724 builder, 2725 struct_type_index, 2726 *field_index, 2727 struct_ref, 2728 Some(Extension::Zero), 2729 )?; 2730 environ.stacks.push1(val); 2731 } 2732 2733 Operator::StructGet { 2734 struct_type_index, 2735 field_index, 2736 } => { 2737 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2738 let struct_ref = environ.stacks.pop1(); 2739 let val = environ.translate_struct_get( 2740 builder, 2741 struct_type_index, 2742 *field_index, 2743 struct_ref, 2744 None, 2745 )?; 2746 environ.stacks.push1(val); 2747 } 2748 2749 Operator::ArrayNew { array_type_index } => { 2750 let array_type_index = TypeIndex::from_u32(*array_type_index); 2751 let (elem, len) = environ.stacks.pop2(); 2752 let array_ref = environ.translate_array_new(builder, array_type_index, elem, len)?; 2753 environ.stacks.push1(array_ref); 2754 } 2755 Operator::ArrayNewDefault { array_type_index } => { 2756 let array_type_index = TypeIndex::from_u32(*array_type_index); 2757 let len = environ.stacks.pop1(); 2758 let array_ref = environ.translate_array_new_default(builder, array_type_index, len)?; 2759 environ.stacks.push1(array_ref); 2760 } 2761 Operator::ArrayNewFixed { 2762 array_type_index, 2763 array_size, 2764 } => { 2765 let array_type_index = TypeIndex::from_u32(*array_type_index); 2766 let array_size = usize::try_from(*array_size).unwrap(); 2767 let elems = environ.stacks.peekn(array_size).to_vec(); 2768 let array_ref = environ.translate_array_new_fixed(builder, array_type_index, &elems)?; 2769 environ.stacks.popn(array_size); 2770 environ.stacks.push1(array_ref); 2771 } 2772 Operator::ArrayNewData { 2773 array_type_index, 2774 array_data_index, 2775 } => { 2776 let array_type_index = TypeIndex::from_u32(*array_type_index); 2777 let array_data_index = DataIndex::from_u32(*array_data_index); 2778 let (data_offset, len) = environ.stacks.pop2(); 2779 let array_ref = environ.translate_array_new_data( 2780 builder, 2781 array_type_index, 2782 array_data_index, 2783 data_offset, 2784 len, 2785 )?; 2786 environ.stacks.push1(array_ref); 2787 } 2788 Operator::ArrayNewElem { 2789 array_type_index, 2790 array_elem_index, 2791 } => { 2792 let array_type_index = TypeIndex::from_u32(*array_type_index); 2793 let array_elem_index = ElemIndex::from_u32(*array_elem_index); 2794 let (elem_offset, len) = environ.stacks.pop2(); 2795 let array_ref = environ.translate_array_new_elem( 2796 builder, 2797 array_type_index, 2798 array_elem_index, 2799 elem_offset, 2800 len, 2801 )?; 2802 environ.stacks.push1(array_ref); 2803 } 2804 Operator::ArrayCopy { 2805 array_type_index_dst, 2806 array_type_index_src, 2807 } => { 2808 let array_type_index_dst = TypeIndex::from_u32(*array_type_index_dst); 2809 let array_type_index_src = TypeIndex::from_u32(*array_type_index_src); 2810 let (dst_array, dst_index, src_array, src_index, len) = environ.stacks.pop5(); 2811 environ.translate_array_copy( 2812 builder, 2813 array_type_index_dst, 2814 dst_array, 2815 dst_index, 2816 array_type_index_src, 2817 src_array, 2818 src_index, 2819 len, 2820 )?; 2821 } 2822 Operator::ArrayFill { array_type_index } => { 2823 let array_type_index = TypeIndex::from_u32(*array_type_index); 2824 let (array, index, val, len) = environ.stacks.pop4(); 2825 environ.translate_array_fill(builder, array_type_index, array, index, val, len)?; 2826 } 2827 Operator::ArrayInitData { 2828 array_type_index, 2829 array_data_index, 2830 } => { 2831 let array_type_index = TypeIndex::from_u32(*array_type_index); 2832 let array_data_index = DataIndex::from_u32(*array_data_index); 2833 let (array, dst_index, src_index, len) = environ.stacks.pop4(); 2834 environ.translate_array_init_data( 2835 builder, 2836 array_type_index, 2837 array, 2838 dst_index, 2839 array_data_index, 2840 src_index, 2841 len, 2842 )?; 2843 } 2844 Operator::ArrayInitElem { 2845 array_type_index, 2846 array_elem_index, 2847 } => { 2848 let array_type_index = TypeIndex::from_u32(*array_type_index); 2849 let array_elem_index = ElemIndex::from_u32(*array_elem_index); 2850 let (array, dst_index, src_index, len) = environ.stacks.pop4(); 2851 environ.translate_array_init_elem( 2852 builder, 2853 array_type_index, 2854 array, 2855 dst_index, 2856 array_elem_index, 2857 src_index, 2858 len, 2859 )?; 2860 } 2861 Operator::ArrayLen => { 2862 let array = environ.stacks.pop1(); 2863 let len = environ.translate_array_len(builder, array)?; 2864 environ.stacks.push1(len); 2865 } 2866 Operator::ArrayGet { array_type_index } => { 2867 let array_type_index = TypeIndex::from_u32(*array_type_index); 2868 let (array, index) = environ.stacks.pop2(); 2869 let elem = 2870 environ.translate_array_get(builder, array_type_index, array, index, None)?; 2871 environ.stacks.push1(elem); 2872 } 2873 Operator::ArrayGetS { array_type_index } => { 2874 let array_type_index = TypeIndex::from_u32(*array_type_index); 2875 let (array, index) = environ.stacks.pop2(); 2876 let elem = environ.translate_array_get( 2877 builder, 2878 array_type_index, 2879 array, 2880 index, 2881 Some(Extension::Sign), 2882 )?; 2883 environ.stacks.push1(elem); 2884 } 2885 Operator::ArrayGetU { array_type_index } => { 2886 let array_type_index = TypeIndex::from_u32(*array_type_index); 2887 let (array, index) = environ.stacks.pop2(); 2888 let elem = environ.translate_array_get( 2889 builder, 2890 array_type_index, 2891 array, 2892 index, 2893 Some(Extension::Zero), 2894 )?; 2895 environ.stacks.push1(elem); 2896 } 2897 Operator::ArraySet { array_type_index } => { 2898 let array_type_index = TypeIndex::from_u32(*array_type_index); 2899 let (array, index, elem) = environ.stacks.pop3(); 2900 environ.translate_array_set(builder, array_type_index, array, index, elem)?; 2901 } 2902 Operator::RefEq => { 2903 let (r1, r2) = environ.stacks.pop2(); 2904 let eq = builder.ins().icmp(ir::condcodes::IntCC::Equal, r1, r2); 2905 let eq = builder.ins().uextend(ir::types::I32, eq); 2906 environ.stacks.push1(eq); 2907 } 2908 Operator::RefTestNonNull { hty } => { 2909 let r = environ.stacks.pop1(); 2910 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2911 unreachable!("validation") 2912 }; 2913 let heap_type = environ.convert_heap_type(*hty)?; 2914 let result = environ.translate_ref_test( 2915 builder, 2916 WasmRefType { 2917 heap_type, 2918 nullable: false, 2919 }, 2920 r, 2921 *r_ty, 2922 )?; 2923 environ.stacks.push1(result); 2924 } 2925 Operator::RefTestNullable { hty } => { 2926 let r = environ.stacks.pop1(); 2927 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2928 unreachable!("validation") 2929 }; 2930 let heap_type = environ.convert_heap_type(*hty)?; 2931 let result = environ.translate_ref_test( 2932 builder, 2933 WasmRefType { 2934 heap_type, 2935 nullable: true, 2936 }, 2937 r, 2938 *r_ty, 2939 )?; 2940 environ.stacks.push1(result); 2941 } 2942 Operator::RefCastNonNull { hty } => { 2943 let r = environ.stacks.pop1(); 2944 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2945 unreachable!("validation") 2946 }; 2947 let heap_type = environ.convert_heap_type(*hty)?; 2948 let cast_okay = environ.translate_ref_test( 2949 builder, 2950 WasmRefType { 2951 heap_type, 2952 nullable: false, 2953 }, 2954 r, 2955 *r_ty, 2956 )?; 2957 environ.trapz(builder, cast_okay, crate::TRAP_CAST_FAILURE); 2958 environ.stacks.push1(r); 2959 } 2960 Operator::RefCastNullable { hty } => { 2961 let r = environ.stacks.pop1(); 2962 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2963 unreachable!("validation") 2964 }; 2965 let heap_type = environ.convert_heap_type(*hty)?; 2966 let cast_okay = environ.translate_ref_test( 2967 builder, 2968 WasmRefType { 2969 heap_type, 2970 nullable: true, 2971 }, 2972 r, 2973 *r_ty, 2974 )?; 2975 environ.trapz(builder, cast_okay, crate::TRAP_CAST_FAILURE); 2976 environ.stacks.push1(r); 2977 } 2978 Operator::BrOnCast { 2979 relative_depth, 2980 to_ref_type, 2981 from_ref_type: _, 2982 } => { 2983 let r = environ.stacks.peek1(); 2984 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2985 unreachable!("validation") 2986 }; 2987 2988 let to_ref_type = environ.convert_ref_type(*to_ref_type)?; 2989 let cast_is_okay = environ.translate_ref_test(builder, to_ref_type, r, *r_ty)?; 2990 2991 let (cast_succeeds_block, inputs) = translate_br_if_args(*relative_depth, environ); 2992 let cast_fails_block = builder.create_block(); 2993 canonicalise_brif( 2994 builder, 2995 cast_is_okay, 2996 cast_succeeds_block, 2997 inputs, 2998 cast_fails_block, 2999 &[ 3000 // NB: the `cast_fails_block` is dominated by the current 3001 // block, and therefore doesn't need any block params. 3002 ], 3003 ); 3004 3005 // The only predecessor is the current block. 3006 builder.seal_block(cast_fails_block); 3007 3008 // The next Wasm instruction is executed when the cast failed and we 3009 // did not branch away. 3010 builder.switch_to_block(cast_fails_block); 3011 } 3012 Operator::BrOnCastFail { 3013 relative_depth, 3014 to_ref_type, 3015 from_ref_type: _, 3016 } => { 3017 let r = environ.stacks.peek1(); 3018 let [.., WasmValType::Ref(r_ty)] = operand_types else { 3019 unreachable!("validation") 3020 }; 3021 3022 let to_ref_type = environ.convert_ref_type(*to_ref_type)?; 3023 let cast_is_okay = environ.translate_ref_test(builder, to_ref_type, r, *r_ty)?; 3024 3025 let (cast_fails_block, inputs) = translate_br_if_args(*relative_depth, environ); 3026 let cast_succeeds_block = builder.create_block(); 3027 canonicalise_brif( 3028 builder, 3029 cast_is_okay, 3030 cast_succeeds_block, 3031 &[ 3032 // NB: the `cast_succeeds_block` is dominated by the current 3033 // block, and therefore doesn't need any block params. 3034 ], 3035 cast_fails_block, 3036 inputs, 3037 ); 3038 3039 // The only predecessor is the current block. 3040 builder.seal_block(cast_succeeds_block); 3041 3042 // The next Wasm instruction is executed when the cast succeeded and 3043 // we did not branch away. 3044 builder.switch_to_block(cast_succeeds_block); 3045 } 3046 3047 Operator::AnyConvertExtern => { 3048 // Pop an `externref`, push an `anyref`. But they have the same 3049 // representation, so we don't actually need to do anything. 3050 } 3051 Operator::ExternConvertAny => { 3052 // Pop an `anyref`, push an `externref`. But they have the same 3053 // representation, so we don't actually need to do anything. 3054 } 3055 3056 Operator::ContNew { cont_type_index } => { 3057 let cont_type_index = TypeIndex::from_u32(*cont_type_index); 3058 let arg_types: SmallVec<[_; 8]> = environ 3059 .continuation_arguments(cont_type_index) 3060 .to_smallvec(); 3061 let result_types: SmallVec<[_; 8]> = 3062 environ.continuation_returns(cont_type_index).to_smallvec(); 3063 let r = environ.stacks.pop1(); 3064 let contobj = environ.translate_cont_new(builder, r, &arg_types, &result_types)?; 3065 environ.stacks.push1(contobj); 3066 } 3067 Operator::ContBind { 3068 argument_index, 3069 result_index, 3070 } => { 3071 let src_types = environ.continuation_arguments(TypeIndex::from_u32(*argument_index)); 3072 let dst_arity = environ 3073 .continuation_arguments(TypeIndex::from_u32(*result_index)) 3074 .len(); 3075 let arg_count = src_types.len() - dst_arity; 3076 3077 let arg_types = &src_types[0..arg_count]; 3078 for arg_type in arg_types { 3079 // We can't bind GC objects using cont.bind at the moment: We 3080 // don't have the necessary infrastructure to traverse the 3081 // buffers used by cont.bind when looking for GC roots. Thus, 3082 // this crude check ensures that these buffers can never contain 3083 // GC roots to begin with. 3084 if arg_type.is_vmgcref_type_and_not_i31() { 3085 return Err(wasmtime_environ::WasmError::Unsupported( 3086 "cont.bind does not support GC types at the moment".into(), 3087 )); 3088 } 3089 } 3090 3091 let (original_contobj, args) = 3092 environ.stacks.peekn(arg_count + 1).split_last().unwrap(); 3093 let original_contobj = *original_contobj; 3094 let args = args.to_vec(); 3095 3096 let new_contobj = environ.translate_cont_bind(builder, original_contobj, &args); 3097 3098 environ.stacks.popn(arg_count + 1); 3099 environ.stacks.push1(new_contobj); 3100 } 3101 Operator::Suspend { tag_index } => { 3102 let tag_index = TagIndex::from_u32(*tag_index); 3103 let param_types = environ.tag_params(tag_index).to_vec(); 3104 let return_types: SmallVec<[_; 8]> = environ 3105 .tag_returns(tag_index) 3106 .iter() 3107 .map(|ty| crate::value_type(environ.isa(), *ty)) 3108 .collect(); 3109 3110 let params = environ.stacks.peekn(param_types.len()).to_vec(); 3111 let param_count = params.len(); 3112 3113 let return_values = 3114 environ.translate_suspend(builder, tag_index.as_u32(), ¶ms, &return_types); 3115 3116 environ.stacks.popn(param_count); 3117 environ.stacks.pushn(&return_values); 3118 } 3119 Operator::Resume { 3120 cont_type_index, 3121 resume_table: wasm_resume_table, 3122 } => { 3123 // We translate the block indices in the wasm resume_table to actual Blocks. 3124 let mut clif_resume_table = vec![]; 3125 for handle in &wasm_resume_table.handlers { 3126 match handle { 3127 wasmparser::Handle::OnLabel { tag, label } => { 3128 let i = environ.stacks.control_stack.len() - 1 - (*label as usize); 3129 let frame = &mut environ.stacks.control_stack[i]; 3130 // This is side-effecting! 3131 frame.set_branched_to_exit(); 3132 clif_resume_table.push((*tag, Some(frame.br_destination()))); 3133 } 3134 wasmparser::Handle::OnSwitch { tag } => { 3135 clif_resume_table.push((*tag, None)); 3136 } 3137 } 3138 } 3139 3140 let cont_type_index = TypeIndex::from_u32(*cont_type_index); 3141 let arity = environ.continuation_arguments(cont_type_index).len(); 3142 let (contobj, call_args) = environ.stacks.peekn(arity + 1).split_last().unwrap(); 3143 let contobj = *contobj; 3144 let call_args = call_args.to_vec(); 3145 3146 let cont_return_vals = environ.translate_resume( 3147 builder, 3148 cont_type_index.as_u32(), 3149 contobj, 3150 &call_args, 3151 &clif_resume_table, 3152 )?; 3153 3154 environ.stacks.popn(arity + 1); // arguments + continuation 3155 environ.stacks.pushn(&cont_return_vals); 3156 } 3157 Operator::ResumeThrow { 3158 cont_type_index: _, 3159 tag_index: _, 3160 resume_table: _, 3161 } => { 3162 // TODO(10248) This depends on exception handling 3163 return Err(wasmtime_environ::WasmError::Unsupported( 3164 "resume.throw instructions not supported, yet".to_string(), 3165 )); 3166 } 3167 Operator::Switch { 3168 cont_type_index, 3169 tag_index, 3170 } => { 3171 // Arguments of the continuation we are going to switch to 3172 let continuation_argument_types: SmallVec<[_; 8]> = environ 3173 .continuation_arguments(TypeIndex::from_u32(*cont_type_index)) 3174 .to_smallvec(); 3175 // Arity includes the continuation argument 3176 let arity = continuation_argument_types.len(); 3177 let (contobj, switch_args) = environ.stacks.peekn(arity).split_last().unwrap(); 3178 let contobj = *contobj; 3179 let switch_args = switch_args.to_vec(); 3180 3181 // Type of the continuation we are going to create by suspending the 3182 // currently running stack 3183 let current_continuation_type = continuation_argument_types.last().unwrap(); 3184 let current_continuation_type = current_continuation_type.unwrap_ref_type(); 3185 3186 // Argument types of current_continuation_type. These will in turn 3187 // be the types of the arguments we receive when someone switches 3188 // back to this switch instruction 3189 let current_continuation_arg_types: SmallVec<[_; 8]> = 3190 match current_continuation_type.heap_type { 3191 WasmHeapType::ConcreteCont(index) => { 3192 let mti = index 3193 .as_module_type_index() 3194 .expect("Only supporting module type indices on switch for now"); 3195 3196 environ 3197 .continuation_arguments(TypeIndex::from_u32(mti.as_u32())) 3198 .iter() 3199 .map(|ty| crate::value_type(environ.isa(), *ty)) 3200 .collect() 3201 } 3202 _ => panic!("Invalid type on switch"), 3203 }; 3204 3205 let switch_return_values = environ.translate_switch( 3206 builder, 3207 *tag_index, 3208 contobj, 3209 &switch_args, 3210 ¤t_continuation_arg_types, 3211 )?; 3212 3213 environ.stacks.popn(arity); 3214 environ.stacks.pushn(&switch_return_values) 3215 } 3216 3217 Operator::GlobalAtomicGet { .. } 3218 | Operator::GlobalAtomicSet { .. } 3219 | Operator::GlobalAtomicRmwAdd { .. } 3220 | Operator::GlobalAtomicRmwSub { .. } 3221 | Operator::GlobalAtomicRmwOr { .. } 3222 | Operator::GlobalAtomicRmwXor { .. } 3223 | Operator::GlobalAtomicRmwAnd { .. } 3224 | Operator::GlobalAtomicRmwXchg { .. } 3225 | Operator::GlobalAtomicRmwCmpxchg { .. } 3226 | Operator::TableAtomicGet { .. } 3227 | Operator::TableAtomicSet { .. } 3228 | Operator::TableAtomicRmwXchg { .. } 3229 | Operator::TableAtomicRmwCmpxchg { .. } 3230 | Operator::StructAtomicGet { .. } 3231 | Operator::StructAtomicGetS { .. } 3232 | Operator::StructAtomicGetU { .. } 3233 | Operator::StructAtomicSet { .. } 3234 | Operator::StructAtomicRmwAdd { .. } 3235 | Operator::StructAtomicRmwSub { .. } 3236 | Operator::StructAtomicRmwOr { .. } 3237 | Operator::StructAtomicRmwXor { .. } 3238 | Operator::StructAtomicRmwAnd { .. } 3239 | Operator::StructAtomicRmwXchg { .. } 3240 | Operator::StructAtomicRmwCmpxchg { .. } 3241 | Operator::ArrayAtomicGet { .. } 3242 | Operator::ArrayAtomicGetS { .. } 3243 | Operator::ArrayAtomicGetU { .. } 3244 | Operator::ArrayAtomicSet { .. } 3245 | Operator::ArrayAtomicRmwAdd { .. } 3246 | Operator::ArrayAtomicRmwSub { .. } 3247 | Operator::ArrayAtomicRmwOr { .. } 3248 | Operator::ArrayAtomicRmwXor { .. } 3249 | Operator::ArrayAtomicRmwAnd { .. } 3250 | Operator::ArrayAtomicRmwXchg { .. } 3251 | Operator::ArrayAtomicRmwCmpxchg { .. } 3252 | Operator::RefI31Shared { .. } => { 3253 return Err(wasm_unsupported!( 3254 "shared-everything-threads operators are not yet implemented" 3255 )); 3256 } 3257 3258 Operator::I64MulWideS => { 3259 let (arg1, arg2) = environ.stacks.pop2(); 3260 let arg1 = builder.ins().sextend(I128, arg1); 3261 let arg2 = builder.ins().sextend(I128, arg2); 3262 let result = builder.ins().imul(arg1, arg2); 3263 let (lo, hi) = builder.ins().isplit(result); 3264 environ.stacks.push2(lo, hi); 3265 } 3266 Operator::I64MulWideU => { 3267 let (arg1, arg2) = environ.stacks.pop2(); 3268 let arg1 = builder.ins().uextend(I128, arg1); 3269 let arg2 = builder.ins().uextend(I128, arg2); 3270 let result = builder.ins().imul(arg1, arg2); 3271 let (lo, hi) = builder.ins().isplit(result); 3272 environ.stacks.push2(lo, hi); 3273 } 3274 Operator::I64Add128 => { 3275 let (arg1, arg2, arg3, arg4) = environ.stacks.pop4(); 3276 let arg1 = builder.ins().iconcat(arg1, arg2); 3277 let arg2 = builder.ins().iconcat(arg3, arg4); 3278 let result = builder.ins().iadd(arg1, arg2); 3279 let (res1, res2) = builder.ins().isplit(result); 3280 environ.stacks.push2(res1, res2); 3281 } 3282 Operator::I64Sub128 => { 3283 let (arg1, arg2, arg3, arg4) = environ.stacks.pop4(); 3284 let arg1 = builder.ins().iconcat(arg1, arg2); 3285 let arg2 = builder.ins().iconcat(arg3, arg4); 3286 let result = builder.ins().isub(arg1, arg2); 3287 let (res1, res2) = builder.ins().isplit(result); 3288 environ.stacks.push2(res1, res2); 3289 } 3290 3291 // catch-all as `Operator` is `#[non_exhaustive]` 3292 op => return Err(wasm_unsupported!("operator {op:?}")), 3293 }; 3294 Ok(()) 3295 } 3296 3297 /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them 3298 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable 3299 /// portion so the translation state must be updated accordingly. 3300 fn translate_unreachable_operator( 3301 validator: &FuncValidator<impl WasmModuleResources>, 3302 op: &Operator, 3303 builder: &mut FunctionBuilder, 3304 environ: &mut FuncEnvironment<'_>, 3305 ) -> WasmResult<()> { 3306 debug_assert!(!environ.is_reachable()); 3307 match *op { 3308 Operator::If { blockty } => { 3309 // Push a placeholder control stack entry. The if isn't reachable, 3310 // so we don't have any branches anywhere. 3311 environ.stacks.push_if( 3312 ir::Block::reserved_value(), 3313 ElseData::NoElse { 3314 branch_inst: ir::Inst::reserved_value(), 3315 placeholder: ir::Block::reserved_value(), 3316 }, 3317 0, 3318 0, 3319 blockty, 3320 ); 3321 } 3322 Operator::Loop { blockty: _ } 3323 | Operator::Block { blockty: _ } 3324 | Operator::TryTable { try_table: _ } => { 3325 environ.stacks.push_block(ir::Block::reserved_value(), 0, 0); 3326 } 3327 Operator::Else => { 3328 let i = environ.stacks.control_stack.len() - 1; 3329 let reachable = environ.is_reachable(); 3330 match environ.stacks.control_stack[i] { 3331 ControlStackFrame::If { 3332 ref else_data, 3333 head_is_reachable, 3334 ref mut consequent_ends_reachable, 3335 blocktype, 3336 .. 3337 } => { 3338 debug_assert!(consequent_ends_reachable.is_none()); 3339 *consequent_ends_reachable = Some(reachable); 3340 3341 if head_is_reachable { 3342 // We have a branch from the head of the `if` to the `else`. 3343 environ.stacks.reachable = true; 3344 3345 let else_block = match *else_data { 3346 ElseData::NoElse { 3347 branch_inst, 3348 placeholder, 3349 } => { 3350 let (params, _results) = 3351 blocktype_params_results(validator, blocktype)?; 3352 let else_block = block_with_params(builder, params, environ)?; 3353 let frame = environ.stacks.control_stack.last().unwrap(); 3354 frame.truncate_value_stack_to_else_params( 3355 &mut environ.stacks.stack, 3356 &mut environ.stacks.stack_shape, 3357 ); 3358 3359 // We change the target of the branch instruction. 3360 builder.change_jump_destination( 3361 branch_inst, 3362 placeholder, 3363 else_block, 3364 ); 3365 builder.seal_block(else_block); 3366 else_block 3367 } 3368 ElseData::WithElse { else_block } => { 3369 let frame = environ.stacks.control_stack.last().unwrap(); 3370 frame.truncate_value_stack_to_else_params( 3371 &mut environ.stacks.stack, 3372 &mut environ.stacks.stack_shape, 3373 ); 3374 else_block 3375 } 3376 }; 3377 3378 builder.switch_to_block(else_block); 3379 3380 // Again, no need to push the parameters for the `else`, 3381 // since we already did when we saw the original `if`. See 3382 // the comment for translating `Operator::Else` in 3383 // `translate_operator` for details. 3384 } 3385 } 3386 _ => unreachable!(), 3387 } 3388 } 3389 Operator::End => { 3390 let value_stack = &mut environ.stacks.stack; 3391 let stack_shape = &mut environ.stacks.stack_shape; 3392 let control_stack = &mut environ.stacks.control_stack; 3393 let frame = control_stack.pop().unwrap(); 3394 3395 frame.restore_catch_handlers(&mut environ.stacks.handlers, builder); 3396 3397 // Pop unused parameters from stack. 3398 frame.truncate_value_stack_to_original_size(value_stack, stack_shape); 3399 3400 let reachable_anyway = match frame { 3401 // If it is a loop we also have to seal the body loop block 3402 ControlStackFrame::Loop { header, .. } => { 3403 builder.seal_block(header); 3404 // And loops can't have branches to the end. 3405 false 3406 } 3407 // If we never set `consequent_ends_reachable` then that means 3408 // we are finishing the consequent now, and there was no 3409 // `else`. Whether the following block is reachable depends only 3410 // on if the head was reachable. 3411 ControlStackFrame::If { 3412 head_is_reachable, 3413 consequent_ends_reachable: None, 3414 .. 3415 } => head_is_reachable, 3416 // Since we are only in this function when in unreachable code, 3417 // we know that the alternative just ended unreachable. Whether 3418 // the following block is reachable depends on if the consequent 3419 // ended reachable or not. 3420 ControlStackFrame::If { 3421 head_is_reachable, 3422 consequent_ends_reachable: Some(consequent_ends_reachable), 3423 .. 3424 } => head_is_reachable && consequent_ends_reachable, 3425 // All other control constructs are already handled. 3426 _ => false, 3427 }; 3428 3429 if frame.exit_is_branched_to() || reachable_anyway { 3430 builder.switch_to_block(frame.following_code()); 3431 builder.seal_block(frame.following_code()); 3432 3433 // And add the return values of the block but only if the next block is reachable 3434 // (which corresponds to testing if the stack depth is 1) 3435 value_stack.extend_from_slice(builder.block_params(frame.following_code())); 3436 environ.stacks.reachable = true; 3437 } 3438 } 3439 _ => { 3440 // We don't translate because this is unreachable code 3441 } 3442 } 3443 3444 Ok(()) 3445 } 3446 3447 /// This function is a generalized helper for validating that a wasm-supplied 3448 /// heap address is in-bounds. 3449 /// 3450 /// This function takes a litany of parameters and requires that the *Wasm* 3451 /// address to be verified is at the top of the stack in `state`. This will 3452 /// generate necessary IR to validate that the heap address is correctly 3453 /// in-bounds, and various parameters are returned describing the valid *native* 3454 /// heap address if execution reaches that point. 3455 /// 3456 /// Returns `None` when the Wasm access will unconditionally trap. 3457 /// 3458 /// Returns `(flags, wasm_addr, native_addr)`. 3459 fn prepare_addr( 3460 memarg: &MemArg, 3461 access_size: u8, 3462 builder: &mut FunctionBuilder, 3463 environ: &mut FuncEnvironment<'_>, 3464 ) -> WasmResult<Reachability<(MemFlags, Value, Value)>> { 3465 let index = environ.stacks.pop1(); 3466 3467 let memory_index = MemoryIndex::from_u32(memarg.memory); 3468 let heap = environ.get_or_create_heap(builder.func, memory_index); 3469 3470 // How exactly the bounds check is performed here and what it's performed 3471 // on is a bit tricky. Generally we want to rely on access violations (e.g. 3472 // segfaults) to generate traps since that means we don't have to bounds 3473 // check anything explicitly. 3474 // 3475 // (1) If we don't have a guard page of unmapped memory, though, then we 3476 // can't rely on this trapping behavior through segfaults. Instead we need 3477 // to bounds-check the entire memory access here which is everything from 3478 // `addr32 + offset` to `addr32 + offset + width` (not inclusive). In this 3479 // scenario our adjusted offset that we're checking is `memarg.offset + 3480 // access_size`. Note that we do saturating arithmetic here to avoid 3481 // overflow. The addition here is in the 64-bit space, which means that 3482 // we'll never overflow for 32-bit wasm but for 64-bit this is an issue. If 3483 // our effective offset is u64::MAX though then it's impossible for for 3484 // that to actually be a valid offset because otherwise the wasm linear 3485 // memory would take all of the host memory! 3486 // 3487 // (2) If we have a guard page, however, then we can perform a further 3488 // optimization of the generated code by only checking multiples of the 3489 // offset-guard size to be more CSE-friendly. Knowing that we have at least 3490 // 1 page of a guard page we're then able to disregard the `width` since we 3491 // know it's always less than one page. Our bounds check will be for the 3492 // first byte which will either succeed and be guaranteed to fault if it's 3493 // actually out of bounds, or the bounds check itself will fail. In any case 3494 // we assert that the width is reasonably small for now so this assumption 3495 // can be adjusted in the future if we get larger widths. 3496 // 3497 // Put another way we can say, where `y < offset_guard_size`: 3498 // 3499 // n * offset_guard_size + y = offset 3500 // 3501 // We'll then pass `n * offset_guard_size` as the bounds check value. If 3502 // this traps then our `offset` would have trapped anyway. If this check 3503 // passes we know 3504 // 3505 // addr32 + n * offset_guard_size < bound 3506 // 3507 // which means 3508 // 3509 // addr32 + n * offset_guard_size + y < bound + offset_guard_size 3510 // 3511 // because `y < offset_guard_size`, which then means: 3512 // 3513 // addr32 + offset < bound + offset_guard_size 3514 // 3515 // Since we know that that guard size bytes are all unmapped we're 3516 // guaranteed that `offset` and the `width` bytes after it are either 3517 // in-bounds or will hit the guard page, meaning we'll get the desired 3518 // semantics we want. 3519 // 3520 // --- 3521 // 3522 // With all that in mind remember that the goal is to bounds check as few 3523 // things as possible. To facilitate this the "fast path" is expected to be 3524 // hit like so: 3525 // 3526 // * For wasm32, wasmtime defaults to 4gb "static" memories with 2gb guard 3527 // regions. This means that for all offsets <=2gb, we hit the optimized 3528 // case for `heap_addr` on static memories 4gb in size in cranelift's 3529 // legalization of `heap_addr`, eliding the bounds check entirely. 3530 // 3531 // * For wasm64 offsets <=2gb will generate a single `heap_addr` 3532 // instruction, but at this time all heaps are "dynamic" which means that 3533 // a single bounds check is forced. Ideally we'd do better here, but 3534 // that's the current state of affairs. 3535 // 3536 // Basically we assume that most configurations have a guard page and most 3537 // offsets in `memarg` are <=2gb, which means we get the fast path of one 3538 // `heap_addr` instruction plus a hardcoded i32-offset in memory-related 3539 // instructions. 3540 let heap = environ.heaps()[heap].clone(); 3541 let addr = match u32::try_from(memarg.offset) { 3542 // If our offset fits within a u32, then we can place the it into the 3543 // offset immediate of the `heap_addr` instruction. 3544 Ok(offset) => bounds_check_and_compute_addr( 3545 builder, 3546 environ, 3547 &heap, 3548 index, 3549 BoundsCheck::StaticOffset { 3550 offset, 3551 access_size, 3552 }, 3553 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3554 ), 3555 3556 // If the offset doesn't fit within a u32, then we can't pass it 3557 // directly into `heap_addr`. 3558 // 3559 // One reasonable question you might ask is "why not?". There's no 3560 // fundamental reason why `heap_addr` *must* take a 32-bit offset. The 3561 // reason this isn't done, though, is that blindly changing the offset 3562 // to a 64-bit offset increases the size of the `InstructionData` enum 3563 // in cranelift by 8 bytes (16 to 24). This can have significant 3564 // performance implications so the conclusion when this was written was 3565 // that we shouldn't do that. 3566 // 3567 // Without the ability to put the whole offset into the `heap_addr` 3568 // instruction we need to fold the offset into the address itself with 3569 // an unsigned addition. In doing so though we need to check for 3570 // overflow because that would mean the address is out-of-bounds (wasm 3571 // bounds checks happen on the effective 33 or 65 bit address once the 3572 // offset is factored in). 3573 // 3574 // Once we have the effective address, offset already folded in, then 3575 // `heap_addr` is used to verify that the address is indeed in-bounds. 3576 // 3577 // Note that this is generating what's likely to be at least two 3578 // branches, one for the overflow and one for the bounds check itself. 3579 // For now though that should hopefully be ok since 4gb+ offsets are 3580 // relatively odd/rare. In the future if needed we can look into 3581 // optimizing this more. 3582 Err(_) => { 3583 let offset = builder 3584 .ins() 3585 .iconst(heap.index_type(), memarg.offset.cast_signed()); 3586 let adjusted_index = environ.uadd_overflow_trap( 3587 builder, 3588 index, 3589 offset, 3590 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3591 ); 3592 bounds_check_and_compute_addr( 3593 builder, 3594 environ, 3595 &heap, 3596 adjusted_index, 3597 BoundsCheck::StaticOffset { 3598 offset: 0, 3599 access_size, 3600 }, 3601 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3602 ) 3603 } 3604 }; 3605 let addr = match addr { 3606 Reachability::Unreachable => return Ok(Reachability::Unreachable), 3607 Reachability::Reachable(a) => a, 3608 }; 3609 3610 // Note that we don't set `is_aligned` here, even if the load instruction's 3611 // alignment immediate may says it's aligned, because WebAssembly's 3612 // immediate field is just a hint, while Cranelift's aligned flag needs a 3613 // guarantee. WebAssembly memory accesses are always little-endian. 3614 let mut flags = MemFlags::new(); 3615 flags.set_endianness(ir::Endianness::Little); 3616 3617 if heap.pcc_memory_type.is_some() { 3618 // Proof-carrying code is enabled; check this memory access. 3619 flags.set_checked(); 3620 } 3621 3622 // The access occurs to the `heap` disjoint category of abstract 3623 // state. This may allow alias analysis to merge redundant loads, 3624 // etc. when heap accesses occur interleaved with other (table, 3625 // vmctx, stack) accesses. 3626 flags.set_alias_region(Some(ir::AliasRegion::Heap)); 3627 3628 Ok(Reachability::Reachable((flags, index, addr))) 3629 } 3630 3631 fn align_atomic_addr( 3632 memarg: &MemArg, 3633 loaded_bytes: u8, 3634 builder: &mut FunctionBuilder, 3635 environ: &mut FuncEnvironment<'_>, 3636 ) { 3637 // Atomic addresses must all be aligned correctly, and for now we check 3638 // alignment before we check out-of-bounds-ness. The order of this check may 3639 // need to be updated depending on the outcome of the official threads 3640 // proposal itself. 3641 // 3642 // Note that with an offset>0 we generate an `iadd_imm` where the result is 3643 // thrown away after the offset check. This may truncate the offset and the 3644 // result may overflow as well, but those conditions won't affect the 3645 // alignment check itself. This can probably be optimized better and we 3646 // should do so in the future as well. 3647 if loaded_bytes > 1 { 3648 let addr = environ.stacks.pop1(); // "peek" via pop then push 3649 environ.stacks.push1(addr); 3650 let effective_addr = if memarg.offset == 0 { 3651 addr 3652 } else { 3653 builder.ins().iadd_imm(addr, memarg.offset.cast_signed()) 3654 }; 3655 debug_assert!(loaded_bytes.is_power_of_two()); 3656 let misalignment = builder 3657 .ins() 3658 .band_imm(effective_addr, i64::from(loaded_bytes - 1)); 3659 let f = builder.ins().icmp_imm(IntCC::NotEqual, misalignment, 0); 3660 environ.trapnz(builder, f, crate::TRAP_HEAP_MISALIGNED); 3661 } 3662 } 3663 3664 /// Like `prepare_addr` but for atomic accesses. 3665 /// 3666 /// Returns `None` when the Wasm access will unconditionally trap. 3667 fn prepare_atomic_addr( 3668 memarg: &MemArg, 3669 loaded_bytes: u8, 3670 builder: &mut FunctionBuilder, 3671 environ: &mut FuncEnvironment<'_>, 3672 ) -> WasmResult<Reachability<(MemFlags, Value, Value)>> { 3673 align_atomic_addr(memarg, loaded_bytes, builder, environ); 3674 prepare_addr(memarg, loaded_bytes, builder, environ) 3675 } 3676 3677 /// Translate a load instruction. 3678 /// 3679 /// Returns the execution state's reachability after the load is translated. 3680 fn translate_load( 3681 memarg: &MemArg, 3682 opcode: ir::Opcode, 3683 result_ty: Type, 3684 builder: &mut FunctionBuilder, 3685 environ: &mut FuncEnvironment<'_>, 3686 ) -> WasmResult<Reachability<()>> { 3687 let mem_op_size = mem_op_size(opcode, result_ty); 3688 let (flags, wasm_index, base) = match prepare_addr(memarg, mem_op_size, builder, environ)? { 3689 Reachability::Unreachable => return Ok(Reachability::Unreachable), 3690 Reachability::Reachable((f, i, b)) => (f, i, b), 3691 }; 3692 3693 environ.before_load(builder, mem_op_size, wasm_index, memarg.offset); 3694 3695 let (load, dfg) = builder 3696 .ins() 3697 .Load(opcode, result_ty, flags, Offset32::new(0), base); 3698 environ.stacks.push1(dfg.first_result(load)); 3699 Ok(Reachability::Reachable(())) 3700 } 3701 3702 /// Translate a store instruction. 3703 fn translate_store( 3704 memarg: &MemArg, 3705 opcode: ir::Opcode, 3706 builder: &mut FunctionBuilder, 3707 environ: &mut FuncEnvironment<'_>, 3708 ) -> WasmResult<()> { 3709 let val = environ.stacks.pop1(); 3710 let val_ty = builder.func.dfg.value_type(val); 3711 let mem_op_size = mem_op_size(opcode, val_ty); 3712 3713 let (flags, wasm_index, base) = unwrap_or_return_unreachable_state!( 3714 environ, 3715 prepare_addr(memarg, mem_op_size, builder, environ)? 3716 ); 3717 3718 environ.before_store(builder, mem_op_size, wasm_index, memarg.offset); 3719 3720 builder 3721 .ins() 3722 .Store(opcode, val_ty, flags, Offset32::new(0), val, base); 3723 Ok(()) 3724 } 3725 3726 fn mem_op_size(opcode: ir::Opcode, ty: Type) -> u8 { 3727 match opcode { 3728 ir::Opcode::Istore8 | ir::Opcode::Sload8 | ir::Opcode::Uload8 => 1, 3729 ir::Opcode::Istore16 | ir::Opcode::Sload16 | ir::Opcode::Uload16 => 2, 3730 ir::Opcode::Istore32 | ir::Opcode::Sload32 | ir::Opcode::Uload32 => 4, 3731 ir::Opcode::Store | ir::Opcode::Load => u8::try_from(ty.bytes()).unwrap(), 3732 _ => panic!("unknown size of mem op for {opcode:?}"), 3733 } 3734 } 3735 3736 fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, environ: &mut FuncEnvironment<'_>) { 3737 let (arg0, arg1) = environ.stacks.pop2(); 3738 let val = builder.ins().icmp(cc, arg0, arg1); 3739 environ.stacks.push1(builder.ins().uextend(I32, val)); 3740 } 3741 3742 fn translate_atomic_rmw( 3743 widened_ty: Type, 3744 access_ty: Type, 3745 op: AtomicRmwOp, 3746 memarg: &MemArg, 3747 builder: &mut FunctionBuilder, 3748 environ: &mut FuncEnvironment<'_>, 3749 ) -> WasmResult<()> { 3750 let mut arg2 = environ.stacks.pop1(); 3751 let arg2_ty = builder.func.dfg.value_type(arg2); 3752 3753 // The operation is performed at type `access_ty`, and the old value is zero-extended 3754 // to type `widened_ty`. 3755 match access_ty { 3756 I8 | I16 | I32 | I64 => {} 3757 _ => { 3758 return Err(wasm_unsupported!( 3759 "atomic_rmw: unsupported access type {:?}", 3760 access_ty 3761 )); 3762 } 3763 }; 3764 let w_ty_ok = match widened_ty { 3765 I32 | I64 => true, 3766 _ => false, 3767 }; 3768 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3769 3770 assert!(arg2_ty.bytes() >= access_ty.bytes()); 3771 if arg2_ty.bytes() > access_ty.bytes() { 3772 arg2 = builder.ins().ireduce(access_ty, arg2); 3773 } 3774 3775 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3776 environ, 3777 prepare_atomic_addr( 3778 memarg, 3779 u8::try_from(access_ty.bytes()).unwrap(), 3780 builder, 3781 environ, 3782 )? 3783 ); 3784 3785 let mut res = builder.ins().atomic_rmw(access_ty, flags, op, addr, arg2); 3786 if access_ty != widened_ty { 3787 res = builder.ins().uextend(widened_ty, res); 3788 } 3789 environ.stacks.push1(res); 3790 Ok(()) 3791 } 3792 3793 fn translate_atomic_cas( 3794 widened_ty: Type, 3795 access_ty: Type, 3796 memarg: &MemArg, 3797 builder: &mut FunctionBuilder, 3798 environ: &mut FuncEnvironment<'_>, 3799 ) -> WasmResult<()> { 3800 let (mut expected, mut replacement) = environ.stacks.pop2(); 3801 let expected_ty = builder.func.dfg.value_type(expected); 3802 let replacement_ty = builder.func.dfg.value_type(replacement); 3803 3804 // The compare-and-swap is performed at type `access_ty`, and the old value is zero-extended 3805 // to type `widened_ty`. 3806 match access_ty { 3807 I8 | I16 | I32 | I64 => {} 3808 _ => { 3809 return Err(wasm_unsupported!( 3810 "atomic_cas: unsupported access type {:?}", 3811 access_ty 3812 )); 3813 } 3814 }; 3815 let w_ty_ok = match widened_ty { 3816 I32 | I64 => true, 3817 _ => false, 3818 }; 3819 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3820 3821 assert!(expected_ty.bytes() >= access_ty.bytes()); 3822 if expected_ty.bytes() > access_ty.bytes() { 3823 expected = builder.ins().ireduce(access_ty, expected); 3824 } 3825 assert!(replacement_ty.bytes() >= access_ty.bytes()); 3826 if replacement_ty.bytes() > access_ty.bytes() { 3827 replacement = builder.ins().ireduce(access_ty, replacement); 3828 } 3829 3830 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3831 environ, 3832 prepare_atomic_addr( 3833 memarg, 3834 u8::try_from(access_ty.bytes()).unwrap(), 3835 builder, 3836 environ, 3837 )? 3838 ); 3839 let mut res = builder.ins().atomic_cas(flags, addr, expected, replacement); 3840 if access_ty != widened_ty { 3841 res = builder.ins().uextend(widened_ty, res); 3842 } 3843 environ.stacks.push1(res); 3844 Ok(()) 3845 } 3846 3847 fn translate_atomic_load( 3848 widened_ty: Type, 3849 access_ty: Type, 3850 memarg: &MemArg, 3851 builder: &mut FunctionBuilder, 3852 environ: &mut FuncEnvironment<'_>, 3853 ) -> WasmResult<()> { 3854 // The load is performed at type `access_ty`, and the loaded value is zero extended 3855 // to `widened_ty`. 3856 match access_ty { 3857 I8 | I16 | I32 | I64 => {} 3858 _ => { 3859 return Err(wasm_unsupported!( 3860 "atomic_load: unsupported access type {:?}", 3861 access_ty 3862 )); 3863 } 3864 }; 3865 let w_ty_ok = match widened_ty { 3866 I32 | I64 => true, 3867 _ => false, 3868 }; 3869 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3870 3871 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3872 environ, 3873 prepare_atomic_addr( 3874 memarg, 3875 u8::try_from(access_ty.bytes()).unwrap(), 3876 builder, 3877 environ, 3878 )? 3879 ); 3880 let mut res = builder.ins().atomic_load(access_ty, flags, addr); 3881 if access_ty != widened_ty { 3882 res = builder.ins().uextend(widened_ty, res); 3883 } 3884 environ.stacks.push1(res); 3885 Ok(()) 3886 } 3887 3888 fn translate_atomic_store( 3889 access_ty: Type, 3890 memarg: &MemArg, 3891 builder: &mut FunctionBuilder, 3892 environ: &mut FuncEnvironment<'_>, 3893 ) -> WasmResult<()> { 3894 let mut data = environ.stacks.pop1(); 3895 let data_ty = builder.func.dfg.value_type(data); 3896 3897 // The operation is performed at type `access_ty`, and the data to be stored may first 3898 // need to be narrowed accordingly. 3899 match access_ty { 3900 I8 | I16 | I32 | I64 => {} 3901 _ => { 3902 return Err(wasm_unsupported!( 3903 "atomic_store: unsupported access type {:?}", 3904 access_ty 3905 )); 3906 } 3907 }; 3908 let d_ty_ok = match data_ty { 3909 I32 | I64 => true, 3910 _ => false, 3911 }; 3912 assert!(d_ty_ok && data_ty.bytes() >= access_ty.bytes()); 3913 3914 if data_ty.bytes() > access_ty.bytes() { 3915 data = builder.ins().ireduce(access_ty, data); 3916 } 3917 3918 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3919 environ, 3920 prepare_atomic_addr( 3921 memarg, 3922 u8::try_from(access_ty.bytes()).unwrap(), 3923 builder, 3924 environ, 3925 )? 3926 ); 3927 builder.ins().atomic_store(flags, data, addr); 3928 Ok(()) 3929 } 3930 3931 fn translate_vector_icmp( 3932 cc: IntCC, 3933 needed_type: Type, 3934 builder: &mut FunctionBuilder, 3935 env: &mut FuncEnvironment<'_>, 3936 ) { 3937 let (a, b) = env.stacks.pop2(); 3938 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 3939 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 3940 env.stacks 3941 .push1(builder.ins().icmp(cc, bitcast_a, bitcast_b)) 3942 } 3943 3944 fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, env: &mut FuncEnvironment<'_>) { 3945 let (arg0, arg1) = env.stacks.pop2(); 3946 let val = builder.ins().fcmp(cc, arg0, arg1); 3947 env.stacks.push1(builder.ins().uextend(I32, val)); 3948 } 3949 3950 fn translate_vector_fcmp( 3951 cc: FloatCC, 3952 needed_type: Type, 3953 builder: &mut FunctionBuilder, 3954 env: &mut FuncEnvironment<'_>, 3955 ) { 3956 let (a, b) = env.stacks.pop2(); 3957 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 3958 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 3959 env.stacks 3960 .push1(builder.ins().fcmp(cc, bitcast_a, bitcast_b)) 3961 } 3962 3963 fn translate_br_if( 3964 relative_depth: u32, 3965 builder: &mut FunctionBuilder, 3966 env: &mut FuncEnvironment<'_>, 3967 ) { 3968 let val = env.stacks.pop1(); 3969 let (br_destination, inputs) = translate_br_if_args(relative_depth, env); 3970 let next_block = builder.create_block(); 3971 canonicalise_brif(builder, val, br_destination, inputs, next_block, &[]); 3972 3973 builder.seal_block(next_block); // The only predecessor is the current block. 3974 builder.switch_to_block(next_block); 3975 } 3976 3977 fn translate_br_if_args<'a>( 3978 relative_depth: u32, 3979 env: &'a mut FuncEnvironment<'_>, 3980 ) -> (ir::Block, &'a mut [ir::Value]) { 3981 let i = env.stacks.control_stack.len() - 1 - (relative_depth as usize); 3982 let (return_count, br_destination) = { 3983 let frame = &mut env.stacks.control_stack[i]; 3984 // The values returned by the branch are still available for the reachable 3985 // code that comes after it 3986 frame.set_branched_to_exit(); 3987 let return_count = if frame.is_loop() { 3988 frame.num_param_values() 3989 } else { 3990 frame.num_return_values() 3991 }; 3992 (return_count, frame.br_destination()) 3993 }; 3994 let inputs = env.stacks.peekn_mut(return_count); 3995 (br_destination, inputs) 3996 } 3997 3998 /// Determine the returned value type of a WebAssembly operator 3999 fn type_of(operator: &Operator) -> Type { 4000 match operator { 4001 Operator::V128Load { .. } 4002 | Operator::V128Store { .. } 4003 | Operator::V128Const { .. } 4004 | Operator::V128Not 4005 | Operator::V128And 4006 | Operator::V128AndNot 4007 | Operator::V128Or 4008 | Operator::V128Xor 4009 | Operator::V128AnyTrue 4010 | Operator::V128Bitselect => I8X16, // default type representing V128 4011 4012 Operator::I8x16Shuffle { .. } 4013 | Operator::I8x16Splat 4014 | Operator::V128Load8Splat { .. } 4015 | Operator::V128Load8Lane { .. } 4016 | Operator::V128Store8Lane { .. } 4017 | Operator::I8x16ExtractLaneS { .. } 4018 | Operator::I8x16ExtractLaneU { .. } 4019 | Operator::I8x16ReplaceLane { .. } 4020 | Operator::I8x16Eq 4021 | Operator::I8x16Ne 4022 | Operator::I8x16LtS 4023 | Operator::I8x16LtU 4024 | Operator::I8x16GtS 4025 | Operator::I8x16GtU 4026 | Operator::I8x16LeS 4027 | Operator::I8x16LeU 4028 | Operator::I8x16GeS 4029 | Operator::I8x16GeU 4030 | Operator::I8x16Neg 4031 | Operator::I8x16Abs 4032 | Operator::I8x16AllTrue 4033 | Operator::I8x16Shl 4034 | Operator::I8x16ShrS 4035 | Operator::I8x16ShrU 4036 | Operator::I8x16Add 4037 | Operator::I8x16AddSatS 4038 | Operator::I8x16AddSatU 4039 | Operator::I8x16Sub 4040 | Operator::I8x16SubSatS 4041 | Operator::I8x16SubSatU 4042 | Operator::I8x16MinS 4043 | Operator::I8x16MinU 4044 | Operator::I8x16MaxS 4045 | Operator::I8x16MaxU 4046 | Operator::I8x16AvgrU 4047 | Operator::I8x16Bitmask 4048 | Operator::I8x16Popcnt 4049 | Operator::I8x16RelaxedLaneselect => I8X16, 4050 4051 Operator::I16x8Splat 4052 | Operator::V128Load16Splat { .. } 4053 | Operator::V128Load16Lane { .. } 4054 | Operator::V128Store16Lane { .. } 4055 | Operator::I16x8ExtractLaneS { .. } 4056 | Operator::I16x8ExtractLaneU { .. } 4057 | Operator::I16x8ReplaceLane { .. } 4058 | Operator::I16x8Eq 4059 | Operator::I16x8Ne 4060 | Operator::I16x8LtS 4061 | Operator::I16x8LtU 4062 | Operator::I16x8GtS 4063 | Operator::I16x8GtU 4064 | Operator::I16x8LeS 4065 | Operator::I16x8LeU 4066 | Operator::I16x8GeS 4067 | Operator::I16x8GeU 4068 | Operator::I16x8Neg 4069 | Operator::I16x8Abs 4070 | Operator::I16x8AllTrue 4071 | Operator::I16x8Shl 4072 | Operator::I16x8ShrS 4073 | Operator::I16x8ShrU 4074 | Operator::I16x8Add 4075 | Operator::I16x8AddSatS 4076 | Operator::I16x8AddSatU 4077 | Operator::I16x8Sub 4078 | Operator::I16x8SubSatS 4079 | Operator::I16x8SubSatU 4080 | Operator::I16x8MinS 4081 | Operator::I16x8MinU 4082 | Operator::I16x8MaxS 4083 | Operator::I16x8MaxU 4084 | Operator::I16x8AvgrU 4085 | Operator::I16x8Mul 4086 | Operator::I16x8Bitmask 4087 | Operator::I16x8RelaxedLaneselect => I16X8, 4088 4089 Operator::I32x4Splat 4090 | Operator::V128Load32Splat { .. } 4091 | Operator::V128Load32Lane { .. } 4092 | Operator::V128Store32Lane { .. } 4093 | Operator::I32x4ExtractLane { .. } 4094 | Operator::I32x4ReplaceLane { .. } 4095 | Operator::I32x4Eq 4096 | Operator::I32x4Ne 4097 | Operator::I32x4LtS 4098 | Operator::I32x4LtU 4099 | Operator::I32x4GtS 4100 | Operator::I32x4GtU 4101 | Operator::I32x4LeS 4102 | Operator::I32x4LeU 4103 | Operator::I32x4GeS 4104 | Operator::I32x4GeU 4105 | Operator::I32x4Neg 4106 | Operator::I32x4Abs 4107 | Operator::I32x4AllTrue 4108 | Operator::I32x4Shl 4109 | Operator::I32x4ShrS 4110 | Operator::I32x4ShrU 4111 | Operator::I32x4Add 4112 | Operator::I32x4Sub 4113 | Operator::I32x4Mul 4114 | Operator::I32x4MinS 4115 | Operator::I32x4MinU 4116 | Operator::I32x4MaxS 4117 | Operator::I32x4MaxU 4118 | Operator::I32x4Bitmask 4119 | Operator::I32x4TruncSatF32x4S 4120 | Operator::I32x4TruncSatF32x4U 4121 | Operator::I32x4RelaxedLaneselect 4122 | Operator::V128Load32Zero { .. } => I32X4, 4123 4124 Operator::I64x2Splat 4125 | Operator::V128Load64Splat { .. } 4126 | Operator::V128Load64Lane { .. } 4127 | Operator::V128Store64Lane { .. } 4128 | Operator::I64x2ExtractLane { .. } 4129 | Operator::I64x2ReplaceLane { .. } 4130 | Operator::I64x2Eq 4131 | Operator::I64x2Ne 4132 | Operator::I64x2LtS 4133 | Operator::I64x2GtS 4134 | Operator::I64x2LeS 4135 | Operator::I64x2GeS 4136 | Operator::I64x2Neg 4137 | Operator::I64x2Abs 4138 | Operator::I64x2AllTrue 4139 | Operator::I64x2Shl 4140 | Operator::I64x2ShrS 4141 | Operator::I64x2ShrU 4142 | Operator::I64x2Add 4143 | Operator::I64x2Sub 4144 | Operator::I64x2Mul 4145 | Operator::I64x2Bitmask 4146 | Operator::I64x2RelaxedLaneselect 4147 | Operator::V128Load64Zero { .. } => I64X2, 4148 4149 Operator::F32x4Splat 4150 | Operator::F32x4ExtractLane { .. } 4151 | Operator::F32x4ReplaceLane { .. } 4152 | Operator::F32x4Eq 4153 | Operator::F32x4Ne 4154 | Operator::F32x4Lt 4155 | Operator::F32x4Gt 4156 | Operator::F32x4Le 4157 | Operator::F32x4Ge 4158 | Operator::F32x4Abs 4159 | Operator::F32x4Neg 4160 | Operator::F32x4Sqrt 4161 | Operator::F32x4Add 4162 | Operator::F32x4Sub 4163 | Operator::F32x4Mul 4164 | Operator::F32x4Div 4165 | Operator::F32x4Min 4166 | Operator::F32x4Max 4167 | Operator::F32x4PMin 4168 | Operator::F32x4PMax 4169 | Operator::F32x4ConvertI32x4S 4170 | Operator::F32x4ConvertI32x4U 4171 | Operator::F32x4Ceil 4172 | Operator::F32x4Floor 4173 | Operator::F32x4Trunc 4174 | Operator::F32x4Nearest 4175 | Operator::F32x4RelaxedMax 4176 | Operator::F32x4RelaxedMin 4177 | Operator::F32x4RelaxedMadd 4178 | Operator::F32x4RelaxedNmadd => F32X4, 4179 4180 Operator::F64x2Splat 4181 | Operator::F64x2ExtractLane { .. } 4182 | Operator::F64x2ReplaceLane { .. } 4183 | Operator::F64x2Eq 4184 | Operator::F64x2Ne 4185 | Operator::F64x2Lt 4186 | Operator::F64x2Gt 4187 | Operator::F64x2Le 4188 | Operator::F64x2Ge 4189 | Operator::F64x2Abs 4190 | Operator::F64x2Neg 4191 | Operator::F64x2Sqrt 4192 | Operator::F64x2Add 4193 | Operator::F64x2Sub 4194 | Operator::F64x2Mul 4195 | Operator::F64x2Div 4196 | Operator::F64x2Min 4197 | Operator::F64x2Max 4198 | Operator::F64x2PMin 4199 | Operator::F64x2PMax 4200 | Operator::F64x2Ceil 4201 | Operator::F64x2Floor 4202 | Operator::F64x2Trunc 4203 | Operator::F64x2Nearest 4204 | Operator::F64x2RelaxedMax 4205 | Operator::F64x2RelaxedMin 4206 | Operator::F64x2RelaxedMadd 4207 | Operator::F64x2RelaxedNmadd => F64X2, 4208 4209 _ => unimplemented!( 4210 "Currently only SIMD instructions are mapped to their return type; the \ 4211 following instruction is not mapped: {:?}", 4212 operator 4213 ), 4214 } 4215 } 4216 4217 /// Some SIMD operations only operate on I8X16 in CLIF; this will convert them to that type by 4218 /// adding a bitcast if necessary. 4219 fn optionally_bitcast_vector( 4220 value: Value, 4221 needed_type: Type, 4222 builder: &mut FunctionBuilder, 4223 ) -> Value { 4224 if builder.func.dfg.value_type(value) != needed_type { 4225 let mut flags = MemFlags::new(); 4226 flags.set_endianness(ir::Endianness::Little); 4227 builder.ins().bitcast(needed_type, flags, value) 4228 } else { 4229 value 4230 } 4231 } 4232 4233 #[inline(always)] 4234 fn is_non_canonical_v128(ty: ir::Type) -> bool { 4235 match ty { 4236 I64X2 | I32X4 | I16X8 | F32X4 | F64X2 => true, 4237 _ => false, 4238 } 4239 } 4240 4241 /// Cast to I8X16, any vector values in `values` that are of "non-canonical" type (meaning, not 4242 /// I8X16), and return them in a slice. A pre-scan is made to determine whether any casts are 4243 /// actually necessary, and if not, the original slice is returned. Otherwise the cast values 4244 /// are returned in a slice that belongs to the caller-supplied `SmallVec`. 4245 fn canonicalise_v128_values<'a>( 4246 tmp_canonicalised: &'a mut SmallVec<[BlockArg; 16]>, 4247 builder: &mut FunctionBuilder, 4248 values: &'a [ir::Value], 4249 ) -> &'a [BlockArg] { 4250 debug_assert!(tmp_canonicalised.is_empty()); 4251 // Cast, and push the resulting `Value`s into `canonicalised`. 4252 for v in values { 4253 let value = if is_non_canonical_v128(builder.func.dfg.value_type(*v)) { 4254 let mut flags = MemFlags::new(); 4255 flags.set_endianness(ir::Endianness::Little); 4256 builder.ins().bitcast(I8X16, flags, *v) 4257 } else { 4258 *v 4259 }; 4260 tmp_canonicalised.push(BlockArg::from(value)); 4261 } 4262 tmp_canonicalised.as_slice() 4263 } 4264 4265 /// Generate a `jump` instruction, but first cast all 128-bit vector values to I8X16 if they 4266 /// don't have that type. This is done in somewhat roundabout way so as to ensure that we 4267 /// almost never have to do any heap allocation. 4268 fn canonicalise_then_jump( 4269 builder: &mut FunctionBuilder, 4270 destination: ir::Block, 4271 params: &[ir::Value], 4272 ) -> ir::Inst { 4273 let mut tmp_canonicalised = SmallVec::<[_; 16]>::new(); 4274 let canonicalised = canonicalise_v128_values(&mut tmp_canonicalised, builder, params); 4275 builder.ins().jump(destination, canonicalised) 4276 } 4277 4278 /// The same but for a `brif` instruction. 4279 fn canonicalise_brif( 4280 builder: &mut FunctionBuilder, 4281 cond: ir::Value, 4282 block_then: ir::Block, 4283 params_then: &[ir::Value], 4284 block_else: ir::Block, 4285 params_else: &[ir::Value], 4286 ) -> ir::Inst { 4287 let mut tmp_canonicalised_then = SmallVec::<[_; 16]>::new(); 4288 let canonicalised_then = 4289 canonicalise_v128_values(&mut tmp_canonicalised_then, builder, params_then); 4290 let mut tmp_canonicalised_else = SmallVec::<[_; 16]>::new(); 4291 let canonicalised_else = 4292 canonicalise_v128_values(&mut tmp_canonicalised_else, builder, params_else); 4293 builder.ins().brif( 4294 cond, 4295 block_then, 4296 canonicalised_then, 4297 block_else, 4298 canonicalised_else, 4299 ) 4300 } 4301 4302 /// A helper for popping and bitcasting a single value; since SIMD values can lose their type by 4303 /// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF 4304 /// typing issues. 4305 fn pop1_with_bitcast( 4306 env: &mut FuncEnvironment<'_>, 4307 needed_type: Type, 4308 builder: &mut FunctionBuilder, 4309 ) -> Value { 4310 optionally_bitcast_vector(env.stacks.pop1(), needed_type, builder) 4311 } 4312 4313 /// A helper for popping and bitcasting two values; since SIMD values can lose their type by 4314 /// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF 4315 /// typing issues. 4316 fn pop2_with_bitcast( 4317 env: &mut FuncEnvironment<'_>, 4318 needed_type: Type, 4319 builder: &mut FunctionBuilder, 4320 ) -> (Value, Value) { 4321 let (a, b) = env.stacks.pop2(); 4322 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 4323 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 4324 (bitcast_a, bitcast_b) 4325 } 4326 4327 fn pop3_with_bitcast( 4328 env: &mut FuncEnvironment<'_>, 4329 needed_type: Type, 4330 builder: &mut FunctionBuilder, 4331 ) -> (Value, Value, Value) { 4332 let (a, b, c) = env.stacks.pop3(); 4333 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 4334 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 4335 let bitcast_c = optionally_bitcast_vector(c, needed_type, builder); 4336 (bitcast_a, bitcast_b, bitcast_c) 4337 } 4338 4339 fn bitcast_arguments<'a>( 4340 builder: &FunctionBuilder, 4341 arguments: &'a mut [Value], 4342 params: &[ir::AbiParam], 4343 param_predicate: impl Fn(usize) -> bool, 4344 ) -> Vec<(Type, &'a mut Value)> { 4345 let filtered_param_types = params 4346 .iter() 4347 .enumerate() 4348 .filter(|(i, _)| param_predicate(*i)) 4349 .map(|(_, param)| param.value_type); 4350 4351 // zip_eq, from the itertools::Itertools trait, is like Iterator::zip but panics if one 4352 // iterator ends before the other. The `param_predicate` is required to select exactly as many 4353 // elements of `params` as there are elements in `arguments`. 4354 let pairs = filtered_param_types.zip_eq(arguments.iter_mut()); 4355 4356 // The arguments which need to be bitcasted are those which have some vector type but the type 4357 // expected by the parameter is not the same vector type as that of the provided argument. 4358 pairs 4359 .filter(|(param_type, _)| param_type.is_vector()) 4360 .filter(|(param_type, arg)| { 4361 let arg_type = builder.func.dfg.value_type(**arg); 4362 assert!( 4363 arg_type.is_vector(), 4364 "unexpected type mismatch: expected {}, argument {} was actually of type {}", 4365 param_type, 4366 *arg, 4367 arg_type 4368 ); 4369 4370 // This is the same check that would be done by `optionally_bitcast_vector`, except we 4371 // can't take a mutable borrow of the FunctionBuilder here, so we defer inserting the 4372 // bitcast instruction to the caller. 4373 arg_type != *param_type 4374 }) 4375 .collect() 4376 } 4377 4378 /// A helper for bitcasting a sequence of return values for the function currently being built. If 4379 /// a value is a vector type that does not match its expected type, this will modify the value in 4380 /// place to point to the result of a `bitcast`. This conversion is necessary to translate Wasm 4381 /// code that uses `V128` as function parameters (or implicitly in block parameters) and still use 4382 /// specific CLIF types (e.g. `I32X4`) in the function body. 4383 pub fn bitcast_wasm_returns(arguments: &mut [Value], builder: &mut FunctionBuilder) { 4384 let changes = bitcast_arguments(builder, arguments, &builder.func.signature.returns, |i| { 4385 builder.func.signature.returns[i].purpose == ir::ArgumentPurpose::Normal 4386 }); 4387 for (t, arg) in changes { 4388 let mut flags = MemFlags::new(); 4389 flags.set_endianness(ir::Endianness::Little); 4390 *arg = builder.ins().bitcast(t, flags, *arg); 4391 } 4392 } 4393 4394 /// Like `bitcast_wasm_returns`, but for the parameters being passed to a specified callee. 4395 fn bitcast_wasm_params( 4396 environ: &mut FuncEnvironment<'_>, 4397 callee_signature: ir::SigRef, 4398 arguments: &mut [Value], 4399 builder: &mut FunctionBuilder, 4400 ) { 4401 let callee_signature = &builder.func.dfg.signatures[callee_signature]; 4402 let changes = bitcast_arguments(builder, arguments, &callee_signature.params, |i| { 4403 environ.is_wasm_parameter(i) 4404 }); 4405 for (t, arg) in changes { 4406 let mut flags = MemFlags::new(); 4407 flags.set_endianness(ir::Endianness::Little); 4408 *arg = builder.ins().bitcast(t, flags, *arg); 4409 } 4410 } 4411 4412 fn create_catch_block( 4413 builder: &mut FunctionBuilder, 4414 catch: &wasmparser::Catch, 4415 environ: &mut FuncEnvironment<'_>, 4416 ) -> WasmResult<ir::Block> { 4417 let (is_ref, tag, label) = match catch { 4418 wasmparser::Catch::One { tag, label } => (false, Some(*tag), *label), 4419 wasmparser::Catch::OneRef { tag, label } => (true, Some(*tag), *label), 4420 wasmparser::Catch::All { label } => (false, None, *label), 4421 wasmparser::Catch::AllRef { label } => (true, None, *label), 4422 }; 4423 4424 // We always create a handler block with one blockparam for the 4425 // one exception payload value that we use (`exn0` block-call 4426 // argument). This one payload value is the `exnref`. Note, 4427 // however, that we carry it in a native host-pointer-sized 4428 // payload (because this is what the exception ABI in Cranelift 4429 // requires). We then generate the args for the actual branch to 4430 // the handler block: we add unboxing code to load each value in 4431 // the exception signature if a specific tag is expected (hence 4432 // signature is known), and then append the `exnref` itself if we 4433 // are compiling a `*Ref` variant. 4434 4435 let (exn_ref_ty, needs_stack_map) = environ.reference_type(WasmHeapType::Exn); 4436 let (exn_payload_wasm_ty, exn_payload_ty) = match environ.pointer_type().bits() { 4437 32 => (wasmparser::ValType::I32, I32), 4438 64 => (wasmparser::ValType::I64, I64), 4439 _ => panic!("Unsupported pointer width"), 4440 }; 4441 let block = block_with_params(builder, [exn_payload_wasm_ty], environ)?; 4442 builder.switch_to_block(block); 4443 let exn_ref = builder.func.dfg.block_params(block)[0]; 4444 debug_assert!(exn_ref_ty.bits() <= exn_payload_ty.bits()); 4445 let exn_ref = if exn_ref_ty.bits() < exn_payload_ty.bits() { 4446 builder.ins().ireduce(exn_ref_ty, exn_ref) 4447 } else { 4448 exn_ref 4449 }; 4450 4451 if needs_stack_map { 4452 builder.declare_value_needs_stack_map(exn_ref); 4453 } 4454 4455 // We encode tag indices from the module directly as Cranelift 4456 // `ExceptionTag`s. We will translate those to (instance, 4457 // defined-tag-index) pairs during the unwind walk -- necessarily 4458 // dynamic because tag imports are provided only at instantiation 4459 // time. 4460 let clif_tag = tag.map(|t| ExceptionTag::from_u32(t)); 4461 4462 environ.stacks.handlers.add_handler(clif_tag, block); 4463 4464 let mut params = vec![]; 4465 4466 if let Some(tag) = tag { 4467 let tag = TagIndex::from_u32(tag); 4468 params.extend(environ.translate_exn_unbox(builder, tag, exn_ref)?); 4469 } 4470 if is_ref { 4471 params.push(exn_ref); 4472 } 4473 4474 // Generate the branch itself. 4475 let i = environ.stacks.control_stack.len() - 1 - (label as usize); 4476 let frame = &mut environ.stacks.control_stack[i]; 4477 frame.set_branched_to_exit(); 4478 canonicalise_then_jump(builder, frame.br_destination(), ¶ms); 4479 4480 Ok(block) 4481 } 4482