1 //! This module contains the bulk of the interesting code performing the translation between 2 //! WebAssembly and Cranelift IR. 3 //! 4 //! The translation is done in one pass, opcode by opcode. Two main data structures are used during 5 //! code translations: the value stack and the control stack. The value stack mimics the execution 6 //! of the WebAssembly stack machine: each instruction result is pushed onto the stack and 7 //! instruction arguments are popped off the stack. Similarly, when encountering a control flow 8 //! block, it is pushed onto the control stack and popped off when encountering the corresponding 9 //! `End`. 10 //! 11 //! Another data structure, the translation state, records information concerning unreachable code 12 //! status and about if inserting a return at the end of the function is necessary. 13 //! 14 //! Some of the WebAssembly instructions need information about the environment for which they 15 //! are being translated: 16 //! 17 //! - the loads and stores need the memory base address; 18 //! - the `get_global` and `set_global` instructions depend on how the globals are implemented; 19 //! - `memory.size` and `memory.grow` are runtime functions; 20 //! - `call_indirect` has to translate the function index into the address of where this 21 //! is; 22 //! 23 //! That is why `translate_function_body` takes an object having the `WasmRuntime` trait as 24 //! argument. 25 //! 26 //! There is extra complexity associated with translation of 128-bit SIMD instructions. 27 //! Wasm only considers there to be a single 128-bit vector type. But CLIF's type system 28 //! distinguishes different lane configurations, so considers 8X16, 16X8, 32X4 and 64X2 to be 29 //! different types. The result is that, in wasm, it's perfectly OK to take the output of (eg) 30 //! an `add.16x8` and use that as an operand of a `sub.32x4`, without using any cast. But when 31 //! translated into CLIF, that will cause a verifier error due to the apparent type mismatch. 32 //! 33 //! This file works around that problem by liberally inserting `bitcast` instructions in many 34 //! places -- mostly, before the use of vector values, either as arguments to CLIF instructions 35 //! or as block actual parameters. These are no-op casts which nevertheless have different 36 //! input and output types, and are used (mostly) to "convert" 16X8, 32X4 and 64X2-typed vectors 37 //! to the "canonical" type, 8X16. Hence the functions `optionally_bitcast_vector`, 38 //! `bitcast_arguments`, `pop*_with_bitcast`, `canonicalise_then_jump`, 39 //! `canonicalise_then_br{z,nz}`, `is_non_canonical_v128` and `canonicalise_v128_values`. 40 //! Note that the `bitcast*` functions are occasionally used to convert to some type other than 41 //! 8X16, but the `canonicalise*` functions always convert to type 8X16. 42 //! 43 //! Be careful when adding support for new vector instructions. And when adding new jumps, even 44 //! if they are apparently don't have any connection to vectors. Never generate any kind of 45 //! (inter-block) jump directly. Instead use `canonicalise_then_jump` and 46 //! `canonicalise_then_br{z,nz}`. 47 //! 48 //! The use of bitcasts is ugly and inefficient, but currently unavoidable: 49 //! 50 //! * they make the logic in this file fragile: miss out a bitcast for any reason, and there is 51 //! the risk of the system failing in the verifier. At least for debug builds. 52 //! 53 //! * in the new backends, they potentially interfere with pattern matching on CLIF -- the 54 //! patterns need to take into account the presence of bitcast nodes. 55 //! 56 //! * in the new backends, they get translated into machine-level vector-register-copy 57 //! instructions, none of which are actually necessary. We then depend on the register 58 //! allocator to coalesce them all out. 59 //! 60 //! * they increase the total number of CLIF nodes that have to be processed, hence slowing down 61 //! the compilation pipeline. Also, the extra coalescing work generates a slowdown. 62 //! 63 //! A better solution which would avoid all four problems would be to remove the 8X16, 16X8, 64 //! 32X4 and 64X2 types from CLIF and instead have a single V128 type. 65 //! 66 //! For further background see also: 67 //! <https://github.com/bytecodealliance/wasmtime/issues/1147> 68 //! ("Too many raw_bitcasts in SIMD code") 69 //! <https://github.com/bytecodealliance/cranelift/pull/1251> 70 //! ("Add X128 type to represent WebAssembly's V128 type") 71 //! <https://github.com/bytecodealliance/cranelift/pull/1236> 72 //! ("Relax verification to allow I8X16 to act as a default vector type") 73 74 use crate::Reachability; 75 use crate::bounds_checks::{BoundsCheck, bounds_check_and_compute_addr}; 76 use crate::func_environ::{Extension, FuncEnvironment}; 77 use crate::translate::environ::{GlobalVariable, StructFieldsVec}; 78 use crate::translate::state::{ControlStackFrame, ElseData, FuncTranslationState}; 79 use crate::translate::translation_utils::{ 80 block_with_params, blocktype_params_results, f32_translation, f64_translation, 81 }; 82 use cranelift_codegen::ir::condcodes::{FloatCC, IntCC}; 83 use cranelift_codegen::ir::immediates::Offset32; 84 use cranelift_codegen::ir::{ 85 self, AtomicRmwOp, InstBuilder, JumpTableData, MemFlags, Value, ValueLabel, 86 }; 87 use cranelift_codegen::ir::{BlockArg, types::*}; 88 use cranelift_codegen::packed_option::ReservedValue; 89 use cranelift_frontend::{FunctionBuilder, Variable}; 90 use itertools::Itertools; 91 use smallvec::SmallVec; 92 use std::collections::{HashMap, hash_map}; 93 use std::vec::Vec; 94 use wasmparser::{FuncValidator, MemArg, Operator, WasmModuleResources}; 95 use wasmtime_environ::{ 96 DataIndex, ElemIndex, FuncIndex, GlobalIndex, MemoryIndex, Signed, TableIndex, TypeConvert, 97 TypeIndex, Unsigned, WasmRefType, WasmResult, WasmValType, wasm_unsupported, 98 }; 99 100 /// Given a `Reachability<T>`, unwrap the inner `T` or, when unreachable, set 101 /// `state.reachable = false` and return. 102 /// 103 /// Used in combination with calling `prepare_addr` and `prepare_atomic_addr` 104 /// when we can statically determine that a Wasm access will unconditionally 105 /// trap. 106 macro_rules! unwrap_or_return_unreachable_state { 107 ($state:ident, $value:expr) => { 108 match $value { 109 Reachability::Reachable(x) => x, 110 Reachability::Unreachable => { 111 $state.reachable = false; 112 return Ok(()); 113 } 114 } 115 }; 116 } 117 118 /// Translates wasm operators into Cranelift IR instructions. 119 pub fn translate_operator( 120 validator: &mut FuncValidator<impl WasmModuleResources>, 121 op: &Operator, 122 operand_types: Option<&[WasmValType]>, 123 builder: &mut FunctionBuilder, 124 state: &mut FuncTranslationState, 125 environ: &mut FuncEnvironment<'_>, 126 ) -> WasmResult<()> { 127 log::trace!("Translating Wasm opcode: {op:?}"); 128 129 if !state.reachable { 130 translate_unreachable_operator(validator, &op, builder, state, environ)?; 131 return Ok(()); 132 } 133 134 // Given that we believe the current block is reachable, the FunctionBuilder ought to agree. 135 debug_assert!(!builder.is_unreachable()); 136 137 let operand_types = operand_types.unwrap_or_else(|| { 138 panic!("should always have operand types available for valid, reachable ops; op = {op:?}") 139 }); 140 141 // This big match treats all Wasm code operators. 142 match op { 143 /********************************** Locals **************************************** 144 * `get_local` and `set_local` are treated as non-SSA variables and will completely 145 * disappear in the Cranelift Code 146 ***********************************************************************************/ 147 Operator::LocalGet { local_index } => { 148 let val = builder.use_var(Variable::from_u32(*local_index)); 149 state.push1(val); 150 let label = ValueLabel::from_u32(*local_index); 151 builder.set_val_label(val, label); 152 } 153 Operator::LocalSet { local_index } => { 154 let mut val = state.pop1(); 155 156 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 157 let ty = builder.func.dfg.value_type(val); 158 if ty.is_vector() { 159 val = optionally_bitcast_vector(val, I8X16, builder); 160 } 161 162 builder.def_var(Variable::from_u32(*local_index), val); 163 let label = ValueLabel::from_u32(*local_index); 164 builder.set_val_label(val, label); 165 } 166 Operator::LocalTee { local_index } => { 167 let mut val = state.peek1(); 168 169 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 170 let ty = builder.func.dfg.value_type(val); 171 if ty.is_vector() { 172 val = optionally_bitcast_vector(val, I8X16, builder); 173 } 174 175 builder.def_var(Variable::from_u32(*local_index), val); 176 let label = ValueLabel::from_u32(*local_index); 177 builder.set_val_label(val, label); 178 } 179 /********************************** Globals **************************************** 180 * `get_global` and `set_global` are handled by the environment. 181 ***********************************************************************************/ 182 Operator::GlobalGet { global_index } => { 183 let val = match state.get_global(builder.func, *global_index, environ)? { 184 GlobalVariable::Memory { gv, offset, ty } => { 185 let addr = builder.ins().global_value(environ.pointer_type(), gv); 186 let mut flags = ir::MemFlags::trusted(); 187 // Store vector globals in little-endian format to avoid 188 // byte swaps on big-endian platforms since at-rest vectors 189 // should already be in little-endian format anyway. 190 if ty.is_vector() { 191 flags.set_endianness(ir::Endianness::Little); 192 } 193 // Put globals in the "table" abstract heap category as well. 194 flags.set_alias_region(Some(ir::AliasRegion::Table)); 195 builder.ins().load(ty, flags, addr, offset) 196 } 197 GlobalVariable::Custom => environ 198 .translate_custom_global_get(builder, GlobalIndex::from_u32(*global_index))?, 199 }; 200 state.push1(val); 201 } 202 Operator::GlobalSet { global_index } => { 203 match state.get_global(builder.func, *global_index, environ)? { 204 GlobalVariable::Memory { gv, offset, ty } => { 205 let addr = builder.ins().global_value(environ.pointer_type(), gv); 206 let mut flags = ir::MemFlags::trusted(); 207 // Like `global.get`, store globals in little-endian format. 208 if ty.is_vector() { 209 flags.set_endianness(ir::Endianness::Little); 210 } 211 // Put globals in the "table" abstract heap category as well. 212 flags.set_alias_region(Some(ir::AliasRegion::Table)); 213 let mut val = state.pop1(); 214 // Ensure SIMD values are cast to their default Cranelift type, I8x16. 215 if ty.is_vector() { 216 val = optionally_bitcast_vector(val, I8X16, builder); 217 } 218 debug_assert_eq!(ty, builder.func.dfg.value_type(val)); 219 builder.ins().store(flags, val, addr, offset); 220 environ.update_global(builder, *global_index, val); 221 } 222 GlobalVariable::Custom => { 223 let val = state.pop1(); 224 environ.translate_custom_global_set( 225 builder, 226 GlobalIndex::from_u32(*global_index), 227 val, 228 )?; 229 } 230 } 231 } 232 /********************************* Stack misc *************************************** 233 * `drop`, `nop`, `unreachable` and `select`. 234 ***********************************************************************************/ 235 Operator::Drop => { 236 state.pop1(); 237 } 238 Operator::Select => { 239 let (mut arg1, mut arg2, cond) = state.pop3(); 240 if builder.func.dfg.value_type(arg1).is_vector() { 241 arg1 = optionally_bitcast_vector(arg1, I8X16, builder); 242 } 243 if builder.func.dfg.value_type(arg2).is_vector() { 244 arg2 = optionally_bitcast_vector(arg2, I8X16, builder); 245 } 246 state.push1(builder.ins().select(cond, arg1, arg2)); 247 } 248 Operator::TypedSelect { ty: _ } => { 249 // We ignore the explicit type parameter as it is only needed for 250 // validation, which we require to have been performed before 251 // translation. 252 let (mut arg1, mut arg2, cond) = state.pop3(); 253 if builder.func.dfg.value_type(arg1).is_vector() { 254 arg1 = optionally_bitcast_vector(arg1, I8X16, builder); 255 } 256 if builder.func.dfg.value_type(arg2).is_vector() { 257 arg2 = optionally_bitcast_vector(arg2, I8X16, builder); 258 } 259 state.push1(builder.ins().select(cond, arg1, arg2)); 260 } 261 Operator::Nop => { 262 // We do nothing 263 } 264 Operator::Unreachable => { 265 environ.trap(builder, crate::TRAP_UNREACHABLE); 266 state.reachable = false; 267 } 268 /***************************** Control flow blocks ********************************** 269 * When starting a control flow block, we create a new `Block` that will hold the code 270 * after the block, and we push a frame on the control stack. Depending on the type 271 * of block, we create a new `Block` for the body of the block with an associated 272 * jump instruction. 273 * 274 * The `End` instruction pops the last control frame from the control stack, seals 275 * the destination block (since `br` instructions targeting it only appear inside the 276 * block and have already been translated) and modify the value stack to use the 277 * possible `Block`'s arguments values. 278 ***********************************************************************************/ 279 Operator::Block { blockty } => { 280 let (params, results) = blocktype_params_results(validator, *blockty)?; 281 let next = block_with_params(builder, results.clone(), environ)?; 282 state.push_block(next, params.len(), results.len()); 283 } 284 Operator::Loop { blockty } => { 285 let (params, results) = blocktype_params_results(validator, *blockty)?; 286 let loop_body = block_with_params(builder, params.clone(), environ)?; 287 let next = block_with_params(builder, results.clone(), environ)?; 288 canonicalise_then_jump(builder, loop_body, state.peekn(params.len())); 289 state.push_loop(loop_body, next, params.len(), results.len()); 290 291 // Pop the initial `Block` actuals and replace them with the `Block`'s 292 // params since control flow joins at the top of the loop. 293 state.popn(params.len()); 294 state 295 .stack 296 .extend_from_slice(builder.block_params(loop_body)); 297 298 builder.switch_to_block(loop_body); 299 environ.translate_loop_header(builder)?; 300 } 301 Operator::If { blockty } => { 302 let val = state.pop1(); 303 304 let next_block = builder.create_block(); 305 let (params, results) = blocktype_params_results(validator, *blockty)?; 306 let (destination, else_data) = if params.clone().eq(results.clone()) { 307 // It is possible there is no `else` block, so we will only 308 // allocate a block for it if/when we find the `else`. For now, 309 // we if the condition isn't true, then we jump directly to the 310 // destination block following the whole `if...end`. If we do end 311 // up discovering an `else`, then we will allocate a block for it 312 // and go back and patch the jump. 313 let destination = block_with_params(builder, results.clone(), environ)?; 314 let branch_inst = canonicalise_brif( 315 builder, 316 val, 317 next_block, 318 &[], 319 destination, 320 state.peekn(params.len()), 321 ); 322 ( 323 destination, 324 ElseData::NoElse { 325 branch_inst, 326 placeholder: destination, 327 }, 328 ) 329 } else { 330 // The `if` type signature is not valid without an `else` block, 331 // so we eagerly allocate the `else` block here. 332 let destination = block_with_params(builder, results.clone(), environ)?; 333 let else_block = block_with_params(builder, params.clone(), environ)?; 334 canonicalise_brif( 335 builder, 336 val, 337 next_block, 338 &[], 339 else_block, 340 state.peekn(params.len()), 341 ); 342 builder.seal_block(else_block); 343 (destination, ElseData::WithElse { else_block }) 344 }; 345 346 builder.seal_block(next_block); // Only predecessor is the current block. 347 builder.switch_to_block(next_block); 348 349 // Here we append an argument to a Block targeted by an argumentless jump instruction 350 // But in fact there are two cases: 351 // - either the If does not have a Else clause, in that case ty = EmptyBlock 352 // and we add nothing; 353 // - either the If have an Else clause, in that case the destination of this jump 354 // instruction will be changed later when we translate the Else operator. 355 state.push_if( 356 destination, 357 else_data, 358 params.len(), 359 results.len(), 360 *blockty, 361 ); 362 } 363 Operator::Else => { 364 let i = state.control_stack.len() - 1; 365 match state.control_stack[i] { 366 ControlStackFrame::If { 367 ref else_data, 368 head_is_reachable, 369 ref mut consequent_ends_reachable, 370 num_return_values, 371 blocktype, 372 destination, 373 .. 374 } => { 375 // We finished the consequent, so record its final 376 // reachability state. 377 debug_assert!(consequent_ends_reachable.is_none()); 378 *consequent_ends_reachable = Some(state.reachable); 379 380 if head_is_reachable { 381 // We have a branch from the head of the `if` to the `else`. 382 state.reachable = true; 383 384 // Ensure we have a block for the `else` block (it may have 385 // already been pre-allocated, see `ElseData` for details). 386 let else_block = match *else_data { 387 ElseData::NoElse { 388 branch_inst, 389 placeholder, 390 } => { 391 let (params, _results) = 392 blocktype_params_results(validator, blocktype)?; 393 debug_assert_eq!(params.len(), num_return_values); 394 let else_block = 395 block_with_params(builder, params.clone(), environ)?; 396 canonicalise_then_jump( 397 builder, 398 destination, 399 state.peekn(params.len()), 400 ); 401 state.popn(params.len()); 402 403 builder.change_jump_destination( 404 branch_inst, 405 placeholder, 406 else_block, 407 ); 408 builder.seal_block(else_block); 409 else_block 410 } 411 ElseData::WithElse { else_block } => { 412 canonicalise_then_jump( 413 builder, 414 destination, 415 state.peekn(num_return_values), 416 ); 417 state.popn(num_return_values); 418 else_block 419 } 420 }; 421 422 // You might be expecting that we push the parameters for this 423 // `else` block here, something like this: 424 // 425 // state.pushn(&control_stack_frame.params); 426 // 427 // We don't do that because they are already on the top of the stack 428 // for us: we pushed the parameters twice when we saw the initial 429 // `if` so that we wouldn't have to save the parameters in the 430 // `ControlStackFrame` as another `Vec` allocation. 431 432 builder.switch_to_block(else_block); 433 434 // We don't bother updating the control frame's `ElseData` 435 // to `WithElse` because nothing else will read it. 436 } 437 } 438 _ => unreachable!(), 439 } 440 } 441 Operator::End => { 442 let frame = state.control_stack.pop().unwrap(); 443 let next_block = frame.following_code(); 444 let return_count = frame.num_return_values(); 445 let return_args = state.peekn_mut(return_count); 446 447 canonicalise_then_jump(builder, next_block, return_args); 448 // You might expect that if we just finished an `if` block that 449 // didn't have a corresponding `else` block, then we would clean 450 // up our duplicate set of parameters that we pushed earlier 451 // right here. However, we don't have to explicitly do that, 452 // since we truncate the stack back to the original height 453 // below. 454 455 builder.switch_to_block(next_block); 456 builder.seal_block(next_block); 457 458 // If it is a loop we also have to seal the body loop block 459 if let ControlStackFrame::Loop { header, .. } = frame { 460 builder.seal_block(header) 461 } 462 463 frame.truncate_value_stack_to_original_size(&mut state.stack); 464 state 465 .stack 466 .extend_from_slice(builder.block_params(next_block)); 467 } 468 /**************************** Branch instructions ********************************* 469 * The branch instructions all have as arguments a target nesting level, which 470 * corresponds to how many control stack frames do we have to pop to get the 471 * destination `Block`. 472 * 473 * Once the destination `Block` is found, we sometimes have to declare a certain depth 474 * of the stack unreachable, because some branch instructions are terminator. 475 * 476 * The `br_table` case is much more complicated because Cranelift's `br_table` instruction 477 * does not support jump arguments like all the other branch instructions. That is why, in 478 * the case where we would use jump arguments for every other branch instruction, we 479 * need to split the critical edges leaving the `br_tables` by creating one `Block` per 480 * table destination; the `br_table` will point to these newly created `Blocks` and these 481 * `Block`s contain only a jump instruction pointing to the final destination, this time with 482 * jump arguments. 483 * 484 * This system is also implemented in Cranelift's SSA construction algorithm, because 485 * `use_var` located in a destination `Block` of a `br_table` might trigger the addition 486 * of jump arguments in each predecessor branch instruction, one of which might be a 487 * `br_table`. 488 ***********************************************************************************/ 489 Operator::Br { relative_depth } => { 490 let i = state.control_stack.len() - 1 - (*relative_depth as usize); 491 let (return_count, br_destination) = { 492 let frame = &mut state.control_stack[i]; 493 // We signal that all the code that follows until the next End is unreachable 494 frame.set_branched_to_exit(); 495 let return_count = if frame.is_loop() { 496 frame.num_param_values() 497 } else { 498 frame.num_return_values() 499 }; 500 (return_count, frame.br_destination()) 501 }; 502 let destination_args = state.peekn_mut(return_count); 503 canonicalise_then_jump(builder, br_destination, destination_args); 504 state.popn(return_count); 505 state.reachable = false; 506 } 507 Operator::BrIf { relative_depth } => translate_br_if(*relative_depth, builder, state), 508 Operator::BrTable { targets } => { 509 let default = targets.default(); 510 let mut min_depth = default; 511 for depth in targets.targets() { 512 let depth = depth?; 513 if depth < min_depth { 514 min_depth = depth; 515 } 516 } 517 let jump_args_count = { 518 let i = state.control_stack.len() - 1 - (min_depth as usize); 519 let min_depth_frame = &state.control_stack[i]; 520 if min_depth_frame.is_loop() { 521 min_depth_frame.num_param_values() 522 } else { 523 min_depth_frame.num_return_values() 524 } 525 }; 526 let val = state.pop1(); 527 let mut data = Vec::with_capacity(targets.len() as usize); 528 if jump_args_count == 0 { 529 // No jump arguments 530 for depth in targets.targets() { 531 let depth = depth?; 532 let block = { 533 let i = state.control_stack.len() - 1 - (depth as usize); 534 let frame = &mut state.control_stack[i]; 535 frame.set_branched_to_exit(); 536 frame.br_destination() 537 }; 538 data.push(builder.func.dfg.block_call(block, &[])); 539 } 540 let block = { 541 let i = state.control_stack.len() - 1 - (default as usize); 542 let frame = &mut state.control_stack[i]; 543 frame.set_branched_to_exit(); 544 frame.br_destination() 545 }; 546 let block = builder.func.dfg.block_call(block, &[]); 547 let jt = builder.create_jump_table(JumpTableData::new(block, &data)); 548 builder.ins().br_table(val, jt); 549 } else { 550 // Here we have jump arguments, but Cranelift's br_table doesn't support them 551 // We then proceed to split the edges going out of the br_table 552 let return_count = jump_args_count; 553 let mut dest_block_sequence = vec![]; 554 let mut dest_block_map = HashMap::new(); 555 for depth in targets.targets() { 556 let depth = depth?; 557 let branch_block = match dest_block_map.entry(depth as usize) { 558 hash_map::Entry::Occupied(entry) => *entry.get(), 559 hash_map::Entry::Vacant(entry) => { 560 let block = builder.create_block(); 561 dest_block_sequence.push((depth as usize, block)); 562 *entry.insert(block) 563 } 564 }; 565 data.push(builder.func.dfg.block_call(branch_block, &[])); 566 } 567 let default_branch_block = match dest_block_map.entry(default as usize) { 568 hash_map::Entry::Occupied(entry) => *entry.get(), 569 hash_map::Entry::Vacant(entry) => { 570 let block = builder.create_block(); 571 dest_block_sequence.push((default as usize, block)); 572 *entry.insert(block) 573 } 574 }; 575 let default_branch_block = builder.func.dfg.block_call(default_branch_block, &[]); 576 let jt = builder.create_jump_table(JumpTableData::new(default_branch_block, &data)); 577 builder.ins().br_table(val, jt); 578 for (depth, dest_block) in dest_block_sequence { 579 builder.switch_to_block(dest_block); 580 builder.seal_block(dest_block); 581 let real_dest_block = { 582 let i = state.control_stack.len() - 1 - depth; 583 let frame = &mut state.control_stack[i]; 584 frame.set_branched_to_exit(); 585 frame.br_destination() 586 }; 587 let destination_args = state.peekn_mut(return_count); 588 canonicalise_then_jump(builder, real_dest_block, destination_args); 589 } 590 state.popn(return_count); 591 } 592 state.reachable = false; 593 } 594 Operator::Return => { 595 let return_count = { 596 let frame = &mut state.control_stack[0]; 597 frame.num_return_values() 598 }; 599 { 600 let return_args = state.peekn_mut(return_count); 601 environ.handle_before_return(&return_args, builder); 602 bitcast_wasm_returns(return_args, builder); 603 builder.ins().return_(return_args); 604 } 605 state.popn(return_count); 606 state.reachable = false; 607 } 608 /********************************** Exception handing **********************************/ 609 Operator::Try { .. } 610 | Operator::Catch { .. } 611 | Operator::Throw { .. } 612 | Operator::Rethrow { .. } 613 | Operator::Delegate { .. } 614 | Operator::CatchAll => { 615 return Err(wasm_unsupported!( 616 "proposed exception handling operator {:?}", 617 op 618 )); 619 } 620 /************************************ Calls **************************************** 621 * The call instructions pop off their arguments from the stack and append their 622 * return values to it. `call_indirect` needs environment support because there is an 623 * argument referring to an index in the external functions table of the module. 624 ************************************************************************************/ 625 Operator::Call { function_index } => { 626 let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?; 627 628 // Bitcast any vector arguments to their default type, I8X16, before calling. 629 let args = state.peekn_mut(num_args); 630 bitcast_wasm_params( 631 environ, 632 builder.func.dfg.ext_funcs[fref].signature, 633 args, 634 builder, 635 ); 636 637 let call = environ.translate_call( 638 builder, 639 FuncIndex::from_u32(*function_index), 640 fref, 641 args, 642 )?; 643 let inst_results = builder.inst_results(call); 644 debug_assert_eq!( 645 inst_results.len(), 646 builder.func.dfg.signatures[builder.func.dfg.ext_funcs[fref].signature] 647 .returns 648 .len(), 649 "translate_call results should match the call signature" 650 ); 651 state.popn(num_args); 652 state.pushn(inst_results); 653 } 654 Operator::CallIndirect { 655 type_index, 656 table_index, 657 } => { 658 // `type_index` is the index of the function's signature and 659 // `table_index` is the index of the table to search the function 660 // in. 661 let (sigref, num_args) = state.get_indirect_sig(builder.func, *type_index, environ)?; 662 let callee = state.pop1(); 663 664 // Bitcast any vector arguments to their default type, I8X16, before calling. 665 let args = state.peekn_mut(num_args); 666 bitcast_wasm_params(environ, sigref, args, builder); 667 668 let call = environ.translate_call_indirect( 669 builder, 670 validator.features(), 671 TableIndex::from_u32(*table_index), 672 TypeIndex::from_u32(*type_index), 673 sigref, 674 callee, 675 state.peekn(num_args), 676 )?; 677 let call = match call { 678 Some(call) => call, 679 None => { 680 state.reachable = false; 681 return Ok(()); 682 } 683 }; 684 let inst_results = builder.inst_results(call); 685 debug_assert_eq!( 686 inst_results.len(), 687 builder.func.dfg.signatures[sigref].returns.len(), 688 "translate_call_indirect results should match the call signature" 689 ); 690 state.popn(num_args); 691 state.pushn(inst_results); 692 } 693 /******************************* Tail Calls ****************************************** 694 * The tail call instructions pop their arguments from the stack and 695 * then permanently transfer control to their callee. The indirect 696 * version requires environment support (while the direct version can 697 * optionally be hooked but doesn't require it) it interacts with the 698 * VM's runtime state via tables. 699 ************************************************************************************/ 700 Operator::ReturnCall { function_index } => { 701 let (fref, num_args) = state.get_direct_func(builder.func, *function_index, environ)?; 702 703 // Bitcast any vector arguments to their default type, I8X16, before calling. 704 let args = state.peekn_mut(num_args); 705 bitcast_wasm_params( 706 environ, 707 builder.func.dfg.ext_funcs[fref].signature, 708 args, 709 builder, 710 ); 711 712 environ.translate_return_call( 713 builder, 714 FuncIndex::from_u32(*function_index), 715 fref, 716 args, 717 )?; 718 719 state.popn(num_args); 720 state.reachable = false; 721 } 722 Operator::ReturnCallIndirect { 723 type_index, 724 table_index, 725 } => { 726 // `type_index` is the index of the function's signature and 727 // `table_index` is the index of the table to search the function 728 // in. 729 let (sigref, num_args) = state.get_indirect_sig(builder.func, *type_index, environ)?; 730 let callee = state.pop1(); 731 732 // Bitcast any vector arguments to their default type, I8X16, before calling. 733 let args = state.peekn_mut(num_args); 734 bitcast_wasm_params(environ, sigref, args, builder); 735 736 environ.translate_return_call_indirect( 737 builder, 738 validator.features(), 739 TableIndex::from_u32(*table_index), 740 TypeIndex::from_u32(*type_index), 741 sigref, 742 callee, 743 state.peekn(num_args), 744 )?; 745 746 state.popn(num_args); 747 state.reachable = false; 748 } 749 Operator::ReturnCallRef { type_index } => { 750 // Get function signature 751 // `index` is the index of the function's signature and `table_index` is the index of 752 // the table to search the function in. 753 let (sigref, num_args) = state.get_indirect_sig(builder.func, *type_index, environ)?; 754 let callee = state.pop1(); 755 756 // Bitcast any vector arguments to their default type, I8X16, before calling. 757 let args = state.peekn_mut(num_args); 758 bitcast_wasm_params(environ, sigref, args, builder); 759 760 environ.translate_return_call_ref(builder, sigref, callee, state.peekn(num_args))?; 761 762 state.popn(num_args); 763 state.reachable = false; 764 } 765 /******************************* Memory management *********************************** 766 * Memory management is handled by environment. It is usually translated into calls to 767 * special functions. 768 ************************************************************************************/ 769 Operator::MemoryGrow { mem } => { 770 // The WebAssembly MVP only supports one linear memory, but we expect the reserved 771 // argument to be a memory index. 772 let heap_index = MemoryIndex::from_u32(*mem); 773 let heap = state.get_heap(builder.func, *mem, environ)?; 774 let val = state.pop1(); 775 environ.before_memory_grow(builder, val, heap_index); 776 state.push1(environ.translate_memory_grow(builder, heap_index, heap, val)?) 777 } 778 Operator::MemorySize { mem } => { 779 let heap_index = MemoryIndex::from_u32(*mem); 780 let heap = state.get_heap(builder.func, *mem, environ)?; 781 state.push1(environ.translate_memory_size(builder.cursor(), heap_index, heap)?); 782 } 783 /******************************* Load instructions *********************************** 784 * Wasm specifies an integer alignment flag but we drop it in Cranelift. 785 * The memory base address is provided by the environment. 786 ************************************************************************************/ 787 Operator::I32Load8U { memarg } => { 788 unwrap_or_return_unreachable_state!( 789 state, 790 translate_load(memarg, ir::Opcode::Uload8, I32, builder, state, environ)? 791 ); 792 } 793 Operator::I32Load16U { memarg } => { 794 unwrap_or_return_unreachable_state!( 795 state, 796 translate_load(memarg, ir::Opcode::Uload16, I32, builder, state, environ)? 797 ); 798 } 799 Operator::I32Load8S { memarg } => { 800 unwrap_or_return_unreachable_state!( 801 state, 802 translate_load(memarg, ir::Opcode::Sload8, I32, builder, state, environ)? 803 ); 804 } 805 Operator::I32Load16S { memarg } => { 806 unwrap_or_return_unreachable_state!( 807 state, 808 translate_load(memarg, ir::Opcode::Sload16, I32, builder, state, environ)? 809 ); 810 } 811 Operator::I64Load8U { memarg } => { 812 unwrap_or_return_unreachable_state!( 813 state, 814 translate_load(memarg, ir::Opcode::Uload8, I64, builder, state, environ)? 815 ); 816 } 817 Operator::I64Load16U { memarg } => { 818 unwrap_or_return_unreachable_state!( 819 state, 820 translate_load(memarg, ir::Opcode::Uload16, I64, builder, state, environ)? 821 ); 822 } 823 Operator::I64Load8S { memarg } => { 824 unwrap_or_return_unreachable_state!( 825 state, 826 translate_load(memarg, ir::Opcode::Sload8, I64, builder, state, environ)? 827 ); 828 } 829 Operator::I64Load16S { memarg } => { 830 unwrap_or_return_unreachable_state!( 831 state, 832 translate_load(memarg, ir::Opcode::Sload16, I64, builder, state, environ)? 833 ); 834 } 835 Operator::I64Load32S { memarg } => { 836 unwrap_or_return_unreachable_state!( 837 state, 838 translate_load(memarg, ir::Opcode::Sload32, I64, builder, state, environ)? 839 ); 840 } 841 Operator::I64Load32U { memarg } => { 842 unwrap_or_return_unreachable_state!( 843 state, 844 translate_load(memarg, ir::Opcode::Uload32, I64, builder, state, environ)? 845 ); 846 } 847 Operator::I32Load { memarg } => { 848 unwrap_or_return_unreachable_state!( 849 state, 850 translate_load(memarg, ir::Opcode::Load, I32, builder, state, environ)? 851 ); 852 } 853 Operator::F32Load { memarg } => { 854 unwrap_or_return_unreachable_state!( 855 state, 856 translate_load(memarg, ir::Opcode::Load, F32, builder, state, environ)? 857 ); 858 } 859 Operator::I64Load { memarg } => { 860 unwrap_or_return_unreachable_state!( 861 state, 862 translate_load(memarg, ir::Opcode::Load, I64, builder, state, environ)? 863 ); 864 } 865 Operator::F64Load { memarg } => { 866 unwrap_or_return_unreachable_state!( 867 state, 868 translate_load(memarg, ir::Opcode::Load, F64, builder, state, environ)? 869 ); 870 } 871 Operator::V128Load { memarg } => { 872 unwrap_or_return_unreachable_state!( 873 state, 874 translate_load(memarg, ir::Opcode::Load, I8X16, builder, state, environ)? 875 ); 876 } 877 Operator::V128Load8x8S { memarg } => { 878 //TODO(#6829): add before_load() and before_store() hooks for SIMD loads and stores. 879 let (flags, _, base) = unwrap_or_return_unreachable_state!( 880 state, 881 prepare_addr(memarg, 8, builder, state, environ)? 882 ); 883 let loaded = builder.ins().sload8x8(flags, base, 0); 884 state.push1(loaded); 885 } 886 Operator::V128Load8x8U { memarg } => { 887 let (flags, _, base) = unwrap_or_return_unreachable_state!( 888 state, 889 prepare_addr(memarg, 8, builder, state, environ)? 890 ); 891 let loaded = builder.ins().uload8x8(flags, base, 0); 892 state.push1(loaded); 893 } 894 Operator::V128Load16x4S { memarg } => { 895 let (flags, _, base) = unwrap_or_return_unreachable_state!( 896 state, 897 prepare_addr(memarg, 8, builder, state, environ)? 898 ); 899 let loaded = builder.ins().sload16x4(flags, base, 0); 900 state.push1(loaded); 901 } 902 Operator::V128Load16x4U { memarg } => { 903 let (flags, _, base) = unwrap_or_return_unreachable_state!( 904 state, 905 prepare_addr(memarg, 8, builder, state, environ)? 906 ); 907 let loaded = builder.ins().uload16x4(flags, base, 0); 908 state.push1(loaded); 909 } 910 Operator::V128Load32x2S { memarg } => { 911 let (flags, _, base) = unwrap_or_return_unreachable_state!( 912 state, 913 prepare_addr(memarg, 8, builder, state, environ)? 914 ); 915 let loaded = builder.ins().sload32x2(flags, base, 0); 916 state.push1(loaded); 917 } 918 Operator::V128Load32x2U { memarg } => { 919 let (flags, _, base) = unwrap_or_return_unreachable_state!( 920 state, 921 prepare_addr(memarg, 8, builder, state, environ)? 922 ); 923 let loaded = builder.ins().uload32x2(flags, base, 0); 924 state.push1(loaded); 925 } 926 /****************************** Store instructions *********************************** 927 * Wasm specifies an integer alignment flag but we drop it in Cranelift. 928 * The memory base address is provided by the environment. 929 ************************************************************************************/ 930 Operator::I32Store { memarg } 931 | Operator::I64Store { memarg } 932 | Operator::F32Store { memarg } 933 | Operator::F64Store { memarg } => { 934 translate_store(memarg, ir::Opcode::Store, builder, state, environ)?; 935 } 936 Operator::I32Store8 { memarg } | Operator::I64Store8 { memarg } => { 937 translate_store(memarg, ir::Opcode::Istore8, builder, state, environ)?; 938 } 939 Operator::I32Store16 { memarg } | Operator::I64Store16 { memarg } => { 940 translate_store(memarg, ir::Opcode::Istore16, builder, state, environ)?; 941 } 942 Operator::I64Store32 { memarg } => { 943 translate_store(memarg, ir::Opcode::Istore32, builder, state, environ)?; 944 } 945 Operator::V128Store { memarg } => { 946 translate_store(memarg, ir::Opcode::Store, builder, state, environ)?; 947 } 948 /****************************** Nullary Operators ************************************/ 949 Operator::I32Const { value } => { 950 state.push1(builder.ins().iconst(I32, i64::from(value.unsigned()))); 951 } 952 Operator::I64Const { value } => state.push1(builder.ins().iconst(I64, *value)), 953 Operator::F32Const { value } => { 954 state.push1(builder.ins().f32const(f32_translation(*value))); 955 } 956 Operator::F64Const { value } => { 957 state.push1(builder.ins().f64const(f64_translation(*value))); 958 } 959 /******************************* Unary Operators *************************************/ 960 Operator::I32Clz | Operator::I64Clz => { 961 let arg = state.pop1(); 962 state.push1(builder.ins().clz(arg)); 963 } 964 Operator::I32Ctz | Operator::I64Ctz => { 965 let arg = state.pop1(); 966 state.push1(builder.ins().ctz(arg)); 967 } 968 Operator::I32Popcnt | Operator::I64Popcnt => { 969 let arg = state.pop1(); 970 state.push1(builder.ins().popcnt(arg)); 971 } 972 Operator::I64ExtendI32S => { 973 let val = state.pop1(); 974 state.push1(builder.ins().sextend(I64, val)); 975 } 976 Operator::I64ExtendI32U => { 977 let val = state.pop1(); 978 state.push1(builder.ins().uextend(I64, val)); 979 } 980 Operator::I32WrapI64 => { 981 let val = state.pop1(); 982 state.push1(builder.ins().ireduce(I32, val)); 983 } 984 Operator::F32Sqrt | Operator::F64Sqrt => { 985 let arg = state.pop1(); 986 state.push1(builder.ins().sqrt(arg)); 987 } 988 Operator::F32Ceil => { 989 let arg = state.pop1(); 990 state.push1(environ.ceil_f32(builder, arg)); 991 } 992 Operator::F64Ceil => { 993 let arg = state.pop1(); 994 state.push1(environ.ceil_f64(builder, arg)); 995 } 996 Operator::F32Floor => { 997 let arg = state.pop1(); 998 state.push1(environ.floor_f32(builder, arg)); 999 } 1000 Operator::F64Floor => { 1001 let arg = state.pop1(); 1002 state.push1(environ.floor_f64(builder, arg)); 1003 } 1004 Operator::F32Trunc => { 1005 let arg = state.pop1(); 1006 state.push1(environ.trunc_f32(builder, arg)); 1007 } 1008 Operator::F64Trunc => { 1009 let arg = state.pop1(); 1010 state.push1(environ.trunc_f64(builder, arg)); 1011 } 1012 Operator::F32Nearest => { 1013 let arg = state.pop1(); 1014 state.push1(environ.nearest_f32(builder, arg)); 1015 } 1016 Operator::F64Nearest => { 1017 let arg = state.pop1(); 1018 state.push1(environ.nearest_f64(builder, arg)); 1019 } 1020 Operator::F32Abs | Operator::F64Abs => { 1021 let val = state.pop1(); 1022 state.push1(builder.ins().fabs(val)); 1023 } 1024 Operator::F32Neg | Operator::F64Neg => { 1025 let arg = state.pop1(); 1026 state.push1(builder.ins().fneg(arg)); 1027 } 1028 Operator::F64ConvertI64U | Operator::F64ConvertI32U => { 1029 let val = state.pop1(); 1030 state.push1(builder.ins().fcvt_from_uint(F64, val)); 1031 } 1032 Operator::F64ConvertI64S | Operator::F64ConvertI32S => { 1033 let val = state.pop1(); 1034 state.push1(builder.ins().fcvt_from_sint(F64, val)); 1035 } 1036 Operator::F32ConvertI64S | Operator::F32ConvertI32S => { 1037 let val = state.pop1(); 1038 state.push1(builder.ins().fcvt_from_sint(F32, val)); 1039 } 1040 Operator::F32ConvertI64U | Operator::F32ConvertI32U => { 1041 let val = state.pop1(); 1042 state.push1(builder.ins().fcvt_from_uint(F32, val)); 1043 } 1044 Operator::F64PromoteF32 => { 1045 let val = state.pop1(); 1046 state.push1(builder.ins().fpromote(F64, val)); 1047 } 1048 Operator::F32DemoteF64 => { 1049 let val = state.pop1(); 1050 state.push1(builder.ins().fdemote(F32, val)); 1051 } 1052 Operator::I64TruncF64S | Operator::I64TruncF32S => { 1053 let val = state.pop1(); 1054 state.push1(environ.translate_fcvt_to_sint(builder, I64, val)); 1055 } 1056 Operator::I32TruncF64S | Operator::I32TruncF32S => { 1057 let val = state.pop1(); 1058 state.push1(environ.translate_fcvt_to_sint(builder, I32, val)); 1059 } 1060 Operator::I64TruncF64U | Operator::I64TruncF32U => { 1061 let val = state.pop1(); 1062 state.push1(environ.translate_fcvt_to_uint(builder, I64, val)); 1063 } 1064 Operator::I32TruncF64U | Operator::I32TruncF32U => { 1065 let val = state.pop1(); 1066 state.push1(environ.translate_fcvt_to_uint(builder, I32, val)); 1067 } 1068 Operator::I64TruncSatF64S | Operator::I64TruncSatF32S => { 1069 let val = state.pop1(); 1070 state.push1(builder.ins().fcvt_to_sint_sat(I64, val)); 1071 } 1072 Operator::I32TruncSatF64S | Operator::I32TruncSatF32S => { 1073 let val = state.pop1(); 1074 state.push1(builder.ins().fcvt_to_sint_sat(I32, val)); 1075 } 1076 Operator::I64TruncSatF64U | Operator::I64TruncSatF32U => { 1077 let val = state.pop1(); 1078 state.push1(builder.ins().fcvt_to_uint_sat(I64, val)); 1079 } 1080 Operator::I32TruncSatF64U | Operator::I32TruncSatF32U => { 1081 let val = state.pop1(); 1082 state.push1(builder.ins().fcvt_to_uint_sat(I32, val)); 1083 } 1084 Operator::F32ReinterpretI32 => { 1085 let val = state.pop1(); 1086 state.push1(builder.ins().bitcast(F32, MemFlags::new(), val)); 1087 } 1088 Operator::F64ReinterpretI64 => { 1089 let val = state.pop1(); 1090 state.push1(builder.ins().bitcast(F64, MemFlags::new(), val)); 1091 } 1092 Operator::I32ReinterpretF32 => { 1093 let val = state.pop1(); 1094 state.push1(builder.ins().bitcast(I32, MemFlags::new(), val)); 1095 } 1096 Operator::I64ReinterpretF64 => { 1097 let val = state.pop1(); 1098 state.push1(builder.ins().bitcast(I64, MemFlags::new(), val)); 1099 } 1100 Operator::I32Extend8S => { 1101 let val = state.pop1(); 1102 state.push1(builder.ins().ireduce(I8, val)); 1103 let val = state.pop1(); 1104 state.push1(builder.ins().sextend(I32, val)); 1105 } 1106 Operator::I32Extend16S => { 1107 let val = state.pop1(); 1108 state.push1(builder.ins().ireduce(I16, val)); 1109 let val = state.pop1(); 1110 state.push1(builder.ins().sextend(I32, val)); 1111 } 1112 Operator::I64Extend8S => { 1113 let val = state.pop1(); 1114 state.push1(builder.ins().ireduce(I8, val)); 1115 let val = state.pop1(); 1116 state.push1(builder.ins().sextend(I64, val)); 1117 } 1118 Operator::I64Extend16S => { 1119 let val = state.pop1(); 1120 state.push1(builder.ins().ireduce(I16, val)); 1121 let val = state.pop1(); 1122 state.push1(builder.ins().sextend(I64, val)); 1123 } 1124 Operator::I64Extend32S => { 1125 let val = state.pop1(); 1126 state.push1(builder.ins().ireduce(I32, val)); 1127 let val = state.pop1(); 1128 state.push1(builder.ins().sextend(I64, val)); 1129 } 1130 /****************************** Binary Operators ************************************/ 1131 Operator::I32Add | Operator::I64Add => { 1132 let (arg1, arg2) = state.pop2(); 1133 state.push1(builder.ins().iadd(arg1, arg2)); 1134 } 1135 Operator::I32And | Operator::I64And => { 1136 let (arg1, arg2) = state.pop2(); 1137 state.push1(builder.ins().band(arg1, arg2)); 1138 } 1139 Operator::I32Or | Operator::I64Or => { 1140 let (arg1, arg2) = state.pop2(); 1141 state.push1(builder.ins().bor(arg1, arg2)); 1142 } 1143 Operator::I32Xor | Operator::I64Xor => { 1144 let (arg1, arg2) = state.pop2(); 1145 state.push1(builder.ins().bxor(arg1, arg2)); 1146 } 1147 Operator::I32Shl | Operator::I64Shl => { 1148 let (arg1, arg2) = state.pop2(); 1149 state.push1(builder.ins().ishl(arg1, arg2)); 1150 } 1151 Operator::I32ShrS | Operator::I64ShrS => { 1152 let (arg1, arg2) = state.pop2(); 1153 state.push1(builder.ins().sshr(arg1, arg2)); 1154 } 1155 Operator::I32ShrU | Operator::I64ShrU => { 1156 let (arg1, arg2) = state.pop2(); 1157 state.push1(builder.ins().ushr(arg1, arg2)); 1158 } 1159 Operator::I32Rotl | Operator::I64Rotl => { 1160 let (arg1, arg2) = state.pop2(); 1161 state.push1(builder.ins().rotl(arg1, arg2)); 1162 } 1163 Operator::I32Rotr | Operator::I64Rotr => { 1164 let (arg1, arg2) = state.pop2(); 1165 state.push1(builder.ins().rotr(arg1, arg2)); 1166 } 1167 Operator::F32Add | Operator::F64Add => { 1168 let (arg1, arg2) = state.pop2(); 1169 state.push1(builder.ins().fadd(arg1, arg2)); 1170 } 1171 Operator::I32Sub | Operator::I64Sub => { 1172 let (arg1, arg2) = state.pop2(); 1173 state.push1(builder.ins().isub(arg1, arg2)); 1174 } 1175 Operator::F32Sub | Operator::F64Sub => { 1176 let (arg1, arg2) = state.pop2(); 1177 state.push1(builder.ins().fsub(arg1, arg2)); 1178 } 1179 Operator::I32Mul | Operator::I64Mul => { 1180 let (arg1, arg2) = state.pop2(); 1181 state.push1(builder.ins().imul(arg1, arg2)); 1182 } 1183 Operator::F32Mul | Operator::F64Mul => { 1184 let (arg1, arg2) = state.pop2(); 1185 state.push1(builder.ins().fmul(arg1, arg2)); 1186 } 1187 Operator::F32Div | Operator::F64Div => { 1188 let (arg1, arg2) = state.pop2(); 1189 state.push1(builder.ins().fdiv(arg1, arg2)); 1190 } 1191 Operator::I32DivS | Operator::I64DivS => { 1192 let (arg1, arg2) = state.pop2(); 1193 state.push1(environ.translate_sdiv(builder, arg1, arg2)); 1194 } 1195 Operator::I32DivU | Operator::I64DivU => { 1196 let (arg1, arg2) = state.pop2(); 1197 state.push1(environ.translate_udiv(builder, arg1, arg2)); 1198 } 1199 Operator::I32RemS | Operator::I64RemS => { 1200 let (arg1, arg2) = state.pop2(); 1201 state.push1(environ.translate_srem(builder, arg1, arg2)); 1202 } 1203 Operator::I32RemU | Operator::I64RemU => { 1204 let (arg1, arg2) = state.pop2(); 1205 state.push1(environ.translate_urem(builder, arg1, arg2)); 1206 } 1207 Operator::F32Min | Operator::F64Min => { 1208 let (arg1, arg2) = state.pop2(); 1209 state.push1(builder.ins().fmin(arg1, arg2)); 1210 } 1211 Operator::F32Max | Operator::F64Max => { 1212 let (arg1, arg2) = state.pop2(); 1213 state.push1(builder.ins().fmax(arg1, arg2)); 1214 } 1215 Operator::F32Copysign | Operator::F64Copysign => { 1216 let (arg1, arg2) = state.pop2(); 1217 state.push1(builder.ins().fcopysign(arg1, arg2)); 1218 } 1219 /**************************** Comparison Operators **********************************/ 1220 Operator::I32LtS | Operator::I64LtS => { 1221 translate_icmp(IntCC::SignedLessThan, builder, state) 1222 } 1223 Operator::I32LtU | Operator::I64LtU => { 1224 translate_icmp(IntCC::UnsignedLessThan, builder, state) 1225 } 1226 Operator::I32LeS | Operator::I64LeS => { 1227 translate_icmp(IntCC::SignedLessThanOrEqual, builder, state) 1228 } 1229 Operator::I32LeU | Operator::I64LeU => { 1230 translate_icmp(IntCC::UnsignedLessThanOrEqual, builder, state) 1231 } 1232 Operator::I32GtS | Operator::I64GtS => { 1233 translate_icmp(IntCC::SignedGreaterThan, builder, state) 1234 } 1235 Operator::I32GtU | Operator::I64GtU => { 1236 translate_icmp(IntCC::UnsignedGreaterThan, builder, state) 1237 } 1238 Operator::I32GeS | Operator::I64GeS => { 1239 translate_icmp(IntCC::SignedGreaterThanOrEqual, builder, state) 1240 } 1241 Operator::I32GeU | Operator::I64GeU => { 1242 translate_icmp(IntCC::UnsignedGreaterThanOrEqual, builder, state) 1243 } 1244 Operator::I32Eqz | Operator::I64Eqz => { 1245 let arg = state.pop1(); 1246 let val = builder.ins().icmp_imm(IntCC::Equal, arg, 0); 1247 state.push1(builder.ins().uextend(I32, val)); 1248 } 1249 Operator::I32Eq | Operator::I64Eq => translate_icmp(IntCC::Equal, builder, state), 1250 Operator::F32Eq | Operator::F64Eq => translate_fcmp(FloatCC::Equal, builder, state), 1251 Operator::I32Ne | Operator::I64Ne => translate_icmp(IntCC::NotEqual, builder, state), 1252 Operator::F32Ne | Operator::F64Ne => translate_fcmp(FloatCC::NotEqual, builder, state), 1253 Operator::F32Gt | Operator::F64Gt => translate_fcmp(FloatCC::GreaterThan, builder, state), 1254 Operator::F32Ge | Operator::F64Ge => { 1255 translate_fcmp(FloatCC::GreaterThanOrEqual, builder, state) 1256 } 1257 Operator::F32Lt | Operator::F64Lt => translate_fcmp(FloatCC::LessThan, builder, state), 1258 Operator::F32Le | Operator::F64Le => { 1259 translate_fcmp(FloatCC::LessThanOrEqual, builder, state) 1260 } 1261 Operator::RefNull { hty } => { 1262 let hty = environ.convert_heap_type(*hty)?; 1263 state.push1(environ.translate_ref_null(builder.cursor(), hty)?) 1264 } 1265 Operator::RefIsNull => { 1266 let value = state.pop1(); 1267 let [WasmValType::Ref(ty)] = operand_types else { 1268 unreachable!("validation") 1269 }; 1270 state.push1(environ.translate_ref_is_null(builder.cursor(), value, *ty)?); 1271 } 1272 Operator::RefFunc { function_index } => { 1273 let index = FuncIndex::from_u32(*function_index); 1274 state.push1(environ.translate_ref_func(builder.cursor(), index)?); 1275 } 1276 Operator::MemoryAtomicWait32 { memarg } | Operator::MemoryAtomicWait64 { memarg } => { 1277 // The WebAssembly MVP only supports one linear memory and 1278 // wasmparser will ensure that the memory indices specified are 1279 // zero. 1280 let implied_ty = match op { 1281 Operator::MemoryAtomicWait64 { .. } => I64, 1282 Operator::MemoryAtomicWait32 { .. } => I32, 1283 _ => unreachable!(), 1284 }; 1285 let heap_index = MemoryIndex::from_u32(memarg.memory); 1286 let heap = state.get_heap(builder.func, memarg.memory, environ)?; 1287 let timeout = state.pop1(); // 64 (fixed) 1288 let expected = state.pop1(); // 32 or 64 (per the `Ixx` in `IxxAtomicWait`) 1289 assert!(builder.func.dfg.value_type(expected) == implied_ty); 1290 let addr = state.pop1(); 1291 let effective_addr = if memarg.offset == 0 { 1292 addr 1293 } else { 1294 let index_type = environ.heaps()[heap].index_type(); 1295 let offset = builder.ins().iconst(index_type, memarg.offset as i64); 1296 environ.uadd_overflow_trap(builder, addr, offset, ir::TrapCode::HEAP_OUT_OF_BOUNDS) 1297 }; 1298 // `fn translate_atomic_wait` can inspect the type of `expected` to figure out what 1299 // code it needs to generate, if it wants. 1300 let res = environ.translate_atomic_wait( 1301 builder, 1302 heap_index, 1303 heap, 1304 effective_addr, 1305 expected, 1306 timeout, 1307 )?; 1308 state.push1(res); 1309 } 1310 Operator::MemoryAtomicNotify { memarg } => { 1311 let heap_index = MemoryIndex::from_u32(memarg.memory); 1312 let heap = state.get_heap(builder.func, memarg.memory, environ)?; 1313 let count = state.pop1(); // 32 (fixed) 1314 let addr = state.pop1(); 1315 let effective_addr = if memarg.offset == 0 { 1316 addr 1317 } else { 1318 let index_type = environ.heaps()[heap].index_type(); 1319 let offset = builder.ins().iconst(index_type, memarg.offset as i64); 1320 environ.uadd_overflow_trap(builder, addr, offset, ir::TrapCode::HEAP_OUT_OF_BOUNDS) 1321 }; 1322 let res = environ.translate_atomic_notify( 1323 builder, 1324 heap_index, 1325 heap, 1326 effective_addr, 1327 count, 1328 )?; 1329 state.push1(res); 1330 } 1331 Operator::I32AtomicLoad { memarg } => { 1332 translate_atomic_load(I32, I32, memarg, builder, state, environ)? 1333 } 1334 Operator::I64AtomicLoad { memarg } => { 1335 translate_atomic_load(I64, I64, memarg, builder, state, environ)? 1336 } 1337 Operator::I32AtomicLoad8U { memarg } => { 1338 translate_atomic_load(I32, I8, memarg, builder, state, environ)? 1339 } 1340 Operator::I32AtomicLoad16U { memarg } => { 1341 translate_atomic_load(I32, I16, memarg, builder, state, environ)? 1342 } 1343 Operator::I64AtomicLoad8U { memarg } => { 1344 translate_atomic_load(I64, I8, memarg, builder, state, environ)? 1345 } 1346 Operator::I64AtomicLoad16U { memarg } => { 1347 translate_atomic_load(I64, I16, memarg, builder, state, environ)? 1348 } 1349 Operator::I64AtomicLoad32U { memarg } => { 1350 translate_atomic_load(I64, I32, memarg, builder, state, environ)? 1351 } 1352 1353 Operator::I32AtomicStore { memarg } => { 1354 translate_atomic_store(I32, memarg, builder, state, environ)? 1355 } 1356 Operator::I64AtomicStore { memarg } => { 1357 translate_atomic_store(I64, memarg, builder, state, environ)? 1358 } 1359 Operator::I32AtomicStore8 { memarg } => { 1360 translate_atomic_store(I8, memarg, builder, state, environ)? 1361 } 1362 Operator::I32AtomicStore16 { memarg } => { 1363 translate_atomic_store(I16, memarg, builder, state, environ)? 1364 } 1365 Operator::I64AtomicStore8 { memarg } => { 1366 translate_atomic_store(I8, memarg, builder, state, environ)? 1367 } 1368 Operator::I64AtomicStore16 { memarg } => { 1369 translate_atomic_store(I16, memarg, builder, state, environ)? 1370 } 1371 Operator::I64AtomicStore32 { memarg } => { 1372 translate_atomic_store(I32, memarg, builder, state, environ)? 1373 } 1374 1375 Operator::I32AtomicRmwAdd { memarg } => { 1376 translate_atomic_rmw(I32, I32, AtomicRmwOp::Add, memarg, builder, state, environ)? 1377 } 1378 Operator::I64AtomicRmwAdd { memarg } => { 1379 translate_atomic_rmw(I64, I64, AtomicRmwOp::Add, memarg, builder, state, environ)? 1380 } 1381 Operator::I32AtomicRmw8AddU { memarg } => { 1382 translate_atomic_rmw(I32, I8, AtomicRmwOp::Add, memarg, builder, state, environ)? 1383 } 1384 Operator::I32AtomicRmw16AddU { memarg } => { 1385 translate_atomic_rmw(I32, I16, AtomicRmwOp::Add, memarg, builder, state, environ)? 1386 } 1387 Operator::I64AtomicRmw8AddU { memarg } => { 1388 translate_atomic_rmw(I64, I8, AtomicRmwOp::Add, memarg, builder, state, environ)? 1389 } 1390 Operator::I64AtomicRmw16AddU { memarg } => { 1391 translate_atomic_rmw(I64, I16, AtomicRmwOp::Add, memarg, builder, state, environ)? 1392 } 1393 Operator::I64AtomicRmw32AddU { memarg } => { 1394 translate_atomic_rmw(I64, I32, AtomicRmwOp::Add, memarg, builder, state, environ)? 1395 } 1396 1397 Operator::I32AtomicRmwSub { memarg } => { 1398 translate_atomic_rmw(I32, I32, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1399 } 1400 Operator::I64AtomicRmwSub { memarg } => { 1401 translate_atomic_rmw(I64, I64, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1402 } 1403 Operator::I32AtomicRmw8SubU { memarg } => { 1404 translate_atomic_rmw(I32, I8, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1405 } 1406 Operator::I32AtomicRmw16SubU { memarg } => { 1407 translate_atomic_rmw(I32, I16, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1408 } 1409 Operator::I64AtomicRmw8SubU { memarg } => { 1410 translate_atomic_rmw(I64, I8, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1411 } 1412 Operator::I64AtomicRmw16SubU { memarg } => { 1413 translate_atomic_rmw(I64, I16, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1414 } 1415 Operator::I64AtomicRmw32SubU { memarg } => { 1416 translate_atomic_rmw(I64, I32, AtomicRmwOp::Sub, memarg, builder, state, environ)? 1417 } 1418 1419 Operator::I32AtomicRmwAnd { memarg } => { 1420 translate_atomic_rmw(I32, I32, AtomicRmwOp::And, memarg, builder, state, environ)? 1421 } 1422 Operator::I64AtomicRmwAnd { memarg } => { 1423 translate_atomic_rmw(I64, I64, AtomicRmwOp::And, memarg, builder, state, environ)? 1424 } 1425 Operator::I32AtomicRmw8AndU { memarg } => { 1426 translate_atomic_rmw(I32, I8, AtomicRmwOp::And, memarg, builder, state, environ)? 1427 } 1428 Operator::I32AtomicRmw16AndU { memarg } => { 1429 translate_atomic_rmw(I32, I16, AtomicRmwOp::And, memarg, builder, state, environ)? 1430 } 1431 Operator::I64AtomicRmw8AndU { memarg } => { 1432 translate_atomic_rmw(I64, I8, AtomicRmwOp::And, memarg, builder, state, environ)? 1433 } 1434 Operator::I64AtomicRmw16AndU { memarg } => { 1435 translate_atomic_rmw(I64, I16, AtomicRmwOp::And, memarg, builder, state, environ)? 1436 } 1437 Operator::I64AtomicRmw32AndU { memarg } => { 1438 translate_atomic_rmw(I64, I32, AtomicRmwOp::And, memarg, builder, state, environ)? 1439 } 1440 1441 Operator::I32AtomicRmwOr { memarg } => { 1442 translate_atomic_rmw(I32, I32, AtomicRmwOp::Or, memarg, builder, state, environ)? 1443 } 1444 Operator::I64AtomicRmwOr { memarg } => { 1445 translate_atomic_rmw(I64, I64, AtomicRmwOp::Or, memarg, builder, state, environ)? 1446 } 1447 Operator::I32AtomicRmw8OrU { memarg } => { 1448 translate_atomic_rmw(I32, I8, AtomicRmwOp::Or, memarg, builder, state, environ)? 1449 } 1450 Operator::I32AtomicRmw16OrU { memarg } => { 1451 translate_atomic_rmw(I32, I16, AtomicRmwOp::Or, memarg, builder, state, environ)? 1452 } 1453 Operator::I64AtomicRmw8OrU { memarg } => { 1454 translate_atomic_rmw(I64, I8, AtomicRmwOp::Or, memarg, builder, state, environ)? 1455 } 1456 Operator::I64AtomicRmw16OrU { memarg } => { 1457 translate_atomic_rmw(I64, I16, AtomicRmwOp::Or, memarg, builder, state, environ)? 1458 } 1459 Operator::I64AtomicRmw32OrU { memarg } => { 1460 translate_atomic_rmw(I64, I32, AtomicRmwOp::Or, memarg, builder, state, environ)? 1461 } 1462 1463 Operator::I32AtomicRmwXor { memarg } => { 1464 translate_atomic_rmw(I32, I32, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1465 } 1466 Operator::I64AtomicRmwXor { memarg } => { 1467 translate_atomic_rmw(I64, I64, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1468 } 1469 Operator::I32AtomicRmw8XorU { memarg } => { 1470 translate_atomic_rmw(I32, I8, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1471 } 1472 Operator::I32AtomicRmw16XorU { memarg } => { 1473 translate_atomic_rmw(I32, I16, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1474 } 1475 Operator::I64AtomicRmw8XorU { memarg } => { 1476 translate_atomic_rmw(I64, I8, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1477 } 1478 Operator::I64AtomicRmw16XorU { memarg } => { 1479 translate_atomic_rmw(I64, I16, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1480 } 1481 Operator::I64AtomicRmw32XorU { memarg } => { 1482 translate_atomic_rmw(I64, I32, AtomicRmwOp::Xor, memarg, builder, state, environ)? 1483 } 1484 1485 Operator::I32AtomicRmwXchg { memarg } => { 1486 translate_atomic_rmw(I32, I32, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1487 } 1488 Operator::I64AtomicRmwXchg { memarg } => { 1489 translate_atomic_rmw(I64, I64, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1490 } 1491 Operator::I32AtomicRmw8XchgU { memarg } => { 1492 translate_atomic_rmw(I32, I8, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1493 } 1494 Operator::I32AtomicRmw16XchgU { memarg } => { 1495 translate_atomic_rmw(I32, I16, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1496 } 1497 Operator::I64AtomicRmw8XchgU { memarg } => { 1498 translate_atomic_rmw(I64, I8, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1499 } 1500 Operator::I64AtomicRmw16XchgU { memarg } => { 1501 translate_atomic_rmw(I64, I16, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1502 } 1503 Operator::I64AtomicRmw32XchgU { memarg } => { 1504 translate_atomic_rmw(I64, I32, AtomicRmwOp::Xchg, memarg, builder, state, environ)? 1505 } 1506 1507 Operator::I32AtomicRmwCmpxchg { memarg } => { 1508 translate_atomic_cas(I32, I32, memarg, builder, state, environ)? 1509 } 1510 Operator::I64AtomicRmwCmpxchg { memarg } => { 1511 translate_atomic_cas(I64, I64, memarg, builder, state, environ)? 1512 } 1513 Operator::I32AtomicRmw8CmpxchgU { memarg } => { 1514 translate_atomic_cas(I32, I8, memarg, builder, state, environ)? 1515 } 1516 Operator::I32AtomicRmw16CmpxchgU { memarg } => { 1517 translate_atomic_cas(I32, I16, memarg, builder, state, environ)? 1518 } 1519 Operator::I64AtomicRmw8CmpxchgU { memarg } => { 1520 translate_atomic_cas(I64, I8, memarg, builder, state, environ)? 1521 } 1522 Operator::I64AtomicRmw16CmpxchgU { memarg } => { 1523 translate_atomic_cas(I64, I16, memarg, builder, state, environ)? 1524 } 1525 Operator::I64AtomicRmw32CmpxchgU { memarg } => { 1526 translate_atomic_cas(I64, I32, memarg, builder, state, environ)? 1527 } 1528 1529 Operator::AtomicFence { .. } => { 1530 builder.ins().fence(); 1531 } 1532 Operator::MemoryCopy { src_mem, dst_mem } => { 1533 let src_index = MemoryIndex::from_u32(*src_mem); 1534 let dst_index = MemoryIndex::from_u32(*dst_mem); 1535 let src_heap = state.get_heap(builder.func, *src_mem, environ)?; 1536 let dst_heap = state.get_heap(builder.func, *dst_mem, environ)?; 1537 let len = state.pop1(); 1538 let src_pos = state.pop1(); 1539 let dst_pos = state.pop1(); 1540 environ.translate_memory_copy( 1541 builder, src_index, src_heap, dst_index, dst_heap, dst_pos, src_pos, len, 1542 )?; 1543 } 1544 Operator::MemoryFill { mem } => { 1545 let heap_index = MemoryIndex::from_u32(*mem); 1546 let heap = state.get_heap(builder.func, *mem, environ)?; 1547 let len = state.pop1(); 1548 let val = state.pop1(); 1549 let dest = state.pop1(); 1550 environ.translate_memory_fill(builder, heap_index, heap, dest, val, len)?; 1551 } 1552 Operator::MemoryInit { data_index, mem } => { 1553 let heap_index = MemoryIndex::from_u32(*mem); 1554 let heap = state.get_heap(builder.func, *mem, environ)?; 1555 let len = state.pop1(); 1556 let src = state.pop1(); 1557 let dest = state.pop1(); 1558 environ.translate_memory_init( 1559 builder, 1560 heap_index, 1561 heap, 1562 *data_index, 1563 dest, 1564 src, 1565 len, 1566 )?; 1567 } 1568 Operator::DataDrop { data_index } => { 1569 environ.translate_data_drop(builder.cursor(), *data_index)?; 1570 } 1571 Operator::TableSize { table: index } => { 1572 state.push1( 1573 environ.translate_table_size(builder.cursor(), TableIndex::from_u32(*index))?, 1574 ); 1575 } 1576 Operator::TableGrow { table: index } => { 1577 let table_index = TableIndex::from_u32(*index); 1578 let delta = state.pop1(); 1579 let init_value = state.pop1(); 1580 state.push1(environ.translate_table_grow(builder, table_index, delta, init_value)?); 1581 } 1582 Operator::TableGet { table: index } => { 1583 let table_index = TableIndex::from_u32(*index); 1584 let index = state.pop1(); 1585 state.push1(environ.translate_table_get(builder, table_index, index)?); 1586 } 1587 Operator::TableSet { table: index } => { 1588 let table_index = TableIndex::from_u32(*index); 1589 let value = state.pop1(); 1590 let index = state.pop1(); 1591 environ.translate_table_set(builder, table_index, value, index)?; 1592 } 1593 Operator::TableCopy { 1594 dst_table: dst_table_index, 1595 src_table: src_table_index, 1596 } => { 1597 let len = state.pop1(); 1598 let src = state.pop1(); 1599 let dest = state.pop1(); 1600 environ.translate_table_copy( 1601 builder, 1602 TableIndex::from_u32(*dst_table_index), 1603 TableIndex::from_u32(*src_table_index), 1604 dest, 1605 src, 1606 len, 1607 )?; 1608 } 1609 Operator::TableFill { table } => { 1610 let table_index = TableIndex::from_u32(*table); 1611 let len = state.pop1(); 1612 let val = state.pop1(); 1613 let dest = state.pop1(); 1614 environ.translate_table_fill(builder, table_index, dest, val, len)?; 1615 } 1616 Operator::TableInit { 1617 elem_index, 1618 table: table_index, 1619 } => { 1620 let len = state.pop1(); 1621 let src = state.pop1(); 1622 let dest = state.pop1(); 1623 environ.translate_table_init( 1624 builder, 1625 *elem_index, 1626 TableIndex::from_u32(*table_index), 1627 dest, 1628 src, 1629 len, 1630 )?; 1631 } 1632 Operator::ElemDrop { elem_index } => { 1633 environ.translate_elem_drop(builder.cursor(), *elem_index)?; 1634 } 1635 Operator::V128Const { value } => { 1636 let data = value.bytes().to_vec().into(); 1637 let handle = builder.func.dfg.constants.insert(data); 1638 let value = builder.ins().vconst(I8X16, handle); 1639 // the v128.const is typed in CLIF as a I8x16 but bitcast to a different type 1640 // before use 1641 state.push1(value) 1642 } 1643 Operator::I8x16Splat | Operator::I16x8Splat => { 1644 let reduced = builder.ins().ireduce(type_of(op).lane_type(), state.pop1()); 1645 let splatted = builder.ins().splat(type_of(op), reduced); 1646 state.push1(splatted) 1647 } 1648 Operator::I32x4Splat 1649 | Operator::I64x2Splat 1650 | Operator::F32x4Splat 1651 | Operator::F64x2Splat => { 1652 let splatted = builder.ins().splat(type_of(op), state.pop1()); 1653 state.push1(splatted) 1654 } 1655 Operator::V128Load8Splat { memarg } 1656 | Operator::V128Load16Splat { memarg } 1657 | Operator::V128Load32Splat { memarg } 1658 | Operator::V128Load64Splat { memarg } => { 1659 unwrap_or_return_unreachable_state!( 1660 state, 1661 translate_load( 1662 memarg, 1663 ir::Opcode::Load, 1664 type_of(op).lane_type(), 1665 builder, 1666 state, 1667 environ, 1668 )? 1669 ); 1670 let splatted = builder.ins().splat(type_of(op), state.pop1()); 1671 state.push1(splatted) 1672 } 1673 Operator::V128Load32Zero { memarg } | Operator::V128Load64Zero { memarg } => { 1674 unwrap_or_return_unreachable_state!( 1675 state, 1676 translate_load( 1677 memarg, 1678 ir::Opcode::Load, 1679 type_of(op).lane_type(), 1680 builder, 1681 state, 1682 environ, 1683 )? 1684 ); 1685 let as_vector = builder.ins().scalar_to_vector(type_of(op), state.pop1()); 1686 state.push1(as_vector) 1687 } 1688 Operator::V128Load8Lane { memarg, lane } 1689 | Operator::V128Load16Lane { memarg, lane } 1690 | Operator::V128Load32Lane { memarg, lane } 1691 | Operator::V128Load64Lane { memarg, lane } => { 1692 let vector = pop1_with_bitcast(state, type_of(op), builder); 1693 unwrap_or_return_unreachable_state!( 1694 state, 1695 translate_load( 1696 memarg, 1697 ir::Opcode::Load, 1698 type_of(op).lane_type(), 1699 builder, 1700 state, 1701 environ, 1702 )? 1703 ); 1704 let replacement = state.pop1(); 1705 state.push1(builder.ins().insertlane(vector, replacement, *lane)) 1706 } 1707 Operator::V128Store8Lane { memarg, lane } 1708 | Operator::V128Store16Lane { memarg, lane } 1709 | Operator::V128Store32Lane { memarg, lane } 1710 | Operator::V128Store64Lane { memarg, lane } => { 1711 let vector = pop1_with_bitcast(state, type_of(op), builder); 1712 state.push1(builder.ins().extractlane(vector, *lane)); 1713 translate_store(memarg, ir::Opcode::Store, builder, state, environ)?; 1714 } 1715 Operator::I8x16ExtractLaneS { lane } | Operator::I16x8ExtractLaneS { lane } => { 1716 let vector = pop1_with_bitcast(state, type_of(op), builder); 1717 let extracted = builder.ins().extractlane(vector, *lane); 1718 state.push1(builder.ins().sextend(I32, extracted)) 1719 } 1720 Operator::I8x16ExtractLaneU { lane } | Operator::I16x8ExtractLaneU { lane } => { 1721 let vector = pop1_with_bitcast(state, type_of(op), builder); 1722 let extracted = builder.ins().extractlane(vector, *lane); 1723 state.push1(builder.ins().uextend(I32, extracted)); 1724 // On x86, PEXTRB zeroes the upper bits of the destination register of extractlane so 1725 // uextend could be elided; for now, uextend is needed for Cranelift's type checks to 1726 // work. 1727 } 1728 Operator::I32x4ExtractLane { lane } 1729 | Operator::I64x2ExtractLane { lane } 1730 | Operator::F32x4ExtractLane { lane } 1731 | Operator::F64x2ExtractLane { lane } => { 1732 let vector = pop1_with_bitcast(state, type_of(op), builder); 1733 state.push1(builder.ins().extractlane(vector, *lane)) 1734 } 1735 Operator::I8x16ReplaceLane { lane } | Operator::I16x8ReplaceLane { lane } => { 1736 let (vector, replacement) = state.pop2(); 1737 let ty = type_of(op); 1738 let reduced = builder.ins().ireduce(ty.lane_type(), replacement); 1739 let vector = optionally_bitcast_vector(vector, ty, builder); 1740 state.push1(builder.ins().insertlane(vector, reduced, *lane)) 1741 } 1742 Operator::I32x4ReplaceLane { lane } 1743 | Operator::I64x2ReplaceLane { lane } 1744 | Operator::F32x4ReplaceLane { lane } 1745 | Operator::F64x2ReplaceLane { lane } => { 1746 let (vector, replacement) = state.pop2(); 1747 let vector = optionally_bitcast_vector(vector, type_of(op), builder); 1748 state.push1(builder.ins().insertlane(vector, replacement, *lane)) 1749 } 1750 Operator::I8x16Shuffle { lanes, .. } => { 1751 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 1752 state.push1(environ.i8x16_shuffle(builder, a, b, lanes)); 1753 // At this point the original types of a and b are lost; users of this value (i.e. this 1754 // WASM-to-CLIF translator) may need to bitcast for type-correctness. This is due 1755 // to WASM using the less specific v128 type for certain operations and more specific 1756 // types (e.g. i8x16) for others. 1757 } 1758 Operator::I8x16Swizzle => { 1759 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 1760 state.push1(environ.swizzle(builder, a, b)); 1761 } 1762 Operator::I8x16Add | Operator::I16x8Add | Operator::I32x4Add | Operator::I64x2Add => { 1763 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1764 state.push1(builder.ins().iadd(a, b)) 1765 } 1766 Operator::I8x16AddSatS | Operator::I16x8AddSatS => { 1767 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1768 state.push1(builder.ins().sadd_sat(a, b)) 1769 } 1770 Operator::I8x16AddSatU | Operator::I16x8AddSatU => { 1771 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1772 state.push1(builder.ins().uadd_sat(a, b)) 1773 } 1774 Operator::I8x16Sub | Operator::I16x8Sub | Operator::I32x4Sub | Operator::I64x2Sub => { 1775 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1776 state.push1(builder.ins().isub(a, b)) 1777 } 1778 Operator::I8x16SubSatS | Operator::I16x8SubSatS => { 1779 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1780 state.push1(builder.ins().ssub_sat(a, b)) 1781 } 1782 Operator::I8x16SubSatU | Operator::I16x8SubSatU => { 1783 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1784 state.push1(builder.ins().usub_sat(a, b)) 1785 } 1786 Operator::I8x16MinS | Operator::I16x8MinS | Operator::I32x4MinS => { 1787 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1788 state.push1(builder.ins().smin(a, b)) 1789 } 1790 Operator::I8x16MinU | Operator::I16x8MinU | Operator::I32x4MinU => { 1791 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1792 state.push1(builder.ins().umin(a, b)) 1793 } 1794 Operator::I8x16MaxS | Operator::I16x8MaxS | Operator::I32x4MaxS => { 1795 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1796 state.push1(builder.ins().smax(a, b)) 1797 } 1798 Operator::I8x16MaxU | Operator::I16x8MaxU | Operator::I32x4MaxU => { 1799 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1800 state.push1(builder.ins().umax(a, b)) 1801 } 1802 Operator::I8x16AvgrU | Operator::I16x8AvgrU => { 1803 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1804 state.push1(builder.ins().avg_round(a, b)) 1805 } 1806 Operator::I8x16Neg | Operator::I16x8Neg | Operator::I32x4Neg | Operator::I64x2Neg => { 1807 let a = pop1_with_bitcast(state, type_of(op), builder); 1808 state.push1(builder.ins().ineg(a)) 1809 } 1810 Operator::I8x16Abs | Operator::I16x8Abs | Operator::I32x4Abs | Operator::I64x2Abs => { 1811 let a = pop1_with_bitcast(state, type_of(op), builder); 1812 state.push1(builder.ins().iabs(a)) 1813 } 1814 Operator::I16x8Mul | Operator::I32x4Mul | Operator::I64x2Mul => { 1815 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1816 state.push1(builder.ins().imul(a, b)) 1817 } 1818 Operator::V128Or => { 1819 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1820 state.push1(builder.ins().bor(a, b)) 1821 } 1822 Operator::V128Xor => { 1823 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1824 state.push1(builder.ins().bxor(a, b)) 1825 } 1826 Operator::V128And => { 1827 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1828 state.push1(builder.ins().band(a, b)) 1829 } 1830 Operator::V128AndNot => { 1831 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1832 state.push1(builder.ins().band_not(a, b)) 1833 } 1834 Operator::V128Not => { 1835 let a = state.pop1(); 1836 state.push1(builder.ins().bnot(a)); 1837 } 1838 Operator::I8x16Shl | Operator::I16x8Shl | Operator::I32x4Shl | Operator::I64x2Shl => { 1839 let (a, b) = state.pop2(); 1840 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1841 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1842 // with cranelift's instruction. 1843 state.push1(builder.ins().ishl(bitcast_a, b)) 1844 } 1845 Operator::I8x16ShrU | Operator::I16x8ShrU | Operator::I32x4ShrU | Operator::I64x2ShrU => { 1846 let (a, b) = state.pop2(); 1847 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1848 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1849 // with cranelift's instruction. 1850 state.push1(builder.ins().ushr(bitcast_a, b)) 1851 } 1852 Operator::I8x16ShrS | Operator::I16x8ShrS | Operator::I32x4ShrS | Operator::I64x2ShrS => { 1853 let (a, b) = state.pop2(); 1854 let bitcast_a = optionally_bitcast_vector(a, type_of(op), builder); 1855 // The spec expects to shift with `b mod lanewidth`; This is directly compatible 1856 // with cranelift's instruction. 1857 state.push1(builder.ins().sshr(bitcast_a, b)) 1858 } 1859 Operator::V128Bitselect => { 1860 let (a, b, c) = pop3_with_bitcast(state, I8X16, builder); 1861 // The CLIF operand ordering is slightly different and the types of all three 1862 // operands must match (hence the bitcast). 1863 state.push1(builder.ins().bitselect(c, a, b)) 1864 } 1865 Operator::V128AnyTrue => { 1866 let a = pop1_with_bitcast(state, type_of(op), builder); 1867 let bool_result = builder.ins().vany_true(a); 1868 state.push1(builder.ins().uextend(I32, bool_result)) 1869 } 1870 Operator::I8x16AllTrue 1871 | Operator::I16x8AllTrue 1872 | Operator::I32x4AllTrue 1873 | Operator::I64x2AllTrue => { 1874 let a = pop1_with_bitcast(state, type_of(op), builder); 1875 let bool_result = builder.ins().vall_true(a); 1876 state.push1(builder.ins().uextend(I32, bool_result)) 1877 } 1878 Operator::I8x16Bitmask 1879 | Operator::I16x8Bitmask 1880 | Operator::I32x4Bitmask 1881 | Operator::I64x2Bitmask => { 1882 let a = pop1_with_bitcast(state, type_of(op), builder); 1883 state.push1(builder.ins().vhigh_bits(I32, a)); 1884 } 1885 Operator::I8x16Eq | Operator::I16x8Eq | Operator::I32x4Eq | Operator::I64x2Eq => { 1886 translate_vector_icmp(IntCC::Equal, type_of(op), builder, state) 1887 } 1888 Operator::I8x16Ne | Operator::I16x8Ne | Operator::I32x4Ne | Operator::I64x2Ne => { 1889 translate_vector_icmp(IntCC::NotEqual, type_of(op), builder, state) 1890 } 1891 Operator::I8x16GtS | Operator::I16x8GtS | Operator::I32x4GtS | Operator::I64x2GtS => { 1892 translate_vector_icmp(IntCC::SignedGreaterThan, type_of(op), builder, state) 1893 } 1894 Operator::I8x16LtS | Operator::I16x8LtS | Operator::I32x4LtS | Operator::I64x2LtS => { 1895 translate_vector_icmp(IntCC::SignedLessThan, type_of(op), builder, state) 1896 } 1897 Operator::I8x16GtU | Operator::I16x8GtU | Operator::I32x4GtU => { 1898 translate_vector_icmp(IntCC::UnsignedGreaterThan, type_of(op), builder, state) 1899 } 1900 Operator::I8x16LtU | Operator::I16x8LtU | Operator::I32x4LtU => { 1901 translate_vector_icmp(IntCC::UnsignedLessThan, type_of(op), builder, state) 1902 } 1903 Operator::I8x16GeS | Operator::I16x8GeS | Operator::I32x4GeS | Operator::I64x2GeS => { 1904 translate_vector_icmp(IntCC::SignedGreaterThanOrEqual, type_of(op), builder, state) 1905 } 1906 Operator::I8x16LeS | Operator::I16x8LeS | Operator::I32x4LeS | Operator::I64x2LeS => { 1907 translate_vector_icmp(IntCC::SignedLessThanOrEqual, type_of(op), builder, state) 1908 } 1909 Operator::I8x16GeU | Operator::I16x8GeU | Operator::I32x4GeU => translate_vector_icmp( 1910 IntCC::UnsignedGreaterThanOrEqual, 1911 type_of(op), 1912 builder, 1913 state, 1914 ), 1915 Operator::I8x16LeU | Operator::I16x8LeU | Operator::I32x4LeU => { 1916 translate_vector_icmp(IntCC::UnsignedLessThanOrEqual, type_of(op), builder, state) 1917 } 1918 Operator::F32x4Eq | Operator::F64x2Eq => { 1919 translate_vector_fcmp(FloatCC::Equal, type_of(op), builder, state) 1920 } 1921 Operator::F32x4Ne | Operator::F64x2Ne => { 1922 translate_vector_fcmp(FloatCC::NotEqual, type_of(op), builder, state) 1923 } 1924 Operator::F32x4Lt | Operator::F64x2Lt => { 1925 translate_vector_fcmp(FloatCC::LessThan, type_of(op), builder, state) 1926 } 1927 Operator::F32x4Gt | Operator::F64x2Gt => { 1928 translate_vector_fcmp(FloatCC::GreaterThan, type_of(op), builder, state) 1929 } 1930 Operator::F32x4Le | Operator::F64x2Le => { 1931 translate_vector_fcmp(FloatCC::LessThanOrEqual, type_of(op), builder, state) 1932 } 1933 Operator::F32x4Ge | Operator::F64x2Ge => { 1934 translate_vector_fcmp(FloatCC::GreaterThanOrEqual, type_of(op), builder, state) 1935 } 1936 Operator::F32x4Add | Operator::F64x2Add => { 1937 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1938 state.push1(builder.ins().fadd(a, b)) 1939 } 1940 Operator::F32x4Sub | Operator::F64x2Sub => { 1941 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1942 state.push1(builder.ins().fsub(a, b)) 1943 } 1944 Operator::F32x4Mul | Operator::F64x2Mul => { 1945 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1946 state.push1(builder.ins().fmul(a, b)) 1947 } 1948 Operator::F32x4Div | Operator::F64x2Div => { 1949 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1950 state.push1(builder.ins().fdiv(a, b)) 1951 } 1952 Operator::F32x4Max | Operator::F64x2Max => { 1953 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1954 state.push1(builder.ins().fmax(a, b)) 1955 } 1956 Operator::F32x4Min | Operator::F64x2Min => { 1957 let (a, b) = pop2_with_bitcast(state, type_of(op), builder); 1958 state.push1(builder.ins().fmin(a, b)) 1959 } 1960 Operator::F32x4PMax | Operator::F64x2PMax => { 1961 // Note the careful ordering here with respect to `fcmp` and 1962 // `bitselect`. This matches the spec definition of: 1963 // 1964 // fpmax(z1, z2) = 1965 // * If z1 is less than z2 then return z2. 1966 // * Else return z1. 1967 let ty = type_of(op); 1968 let (a, b) = pop2_with_bitcast(state, ty, builder); 1969 let cmp = builder.ins().fcmp(FloatCC::LessThan, a, b); 1970 let cmp = optionally_bitcast_vector(cmp, ty, builder); 1971 state.push1(builder.ins().bitselect(cmp, b, a)) 1972 } 1973 Operator::F32x4PMin | Operator::F64x2PMin => { 1974 // Note the careful ordering here which is similar to `pmax` above: 1975 // 1976 // fpmin(z1, z2) = 1977 // * If z2 is less than z1 then return z2. 1978 // * Else return z1. 1979 let ty = type_of(op); 1980 let (a, b) = pop2_with_bitcast(state, ty, builder); 1981 let cmp = builder.ins().fcmp(FloatCC::LessThan, b, a); 1982 let cmp = optionally_bitcast_vector(cmp, ty, builder); 1983 state.push1(builder.ins().bitselect(cmp, b, a)) 1984 } 1985 Operator::F32x4Sqrt | Operator::F64x2Sqrt => { 1986 let a = pop1_with_bitcast(state, type_of(op), builder); 1987 state.push1(builder.ins().sqrt(a)) 1988 } 1989 Operator::F32x4Neg | Operator::F64x2Neg => { 1990 let a = pop1_with_bitcast(state, type_of(op), builder); 1991 state.push1(builder.ins().fneg(a)) 1992 } 1993 Operator::F32x4Abs | Operator::F64x2Abs => { 1994 let a = pop1_with_bitcast(state, type_of(op), builder); 1995 state.push1(builder.ins().fabs(a)) 1996 } 1997 Operator::F32x4ConvertI32x4S => { 1998 let a = pop1_with_bitcast(state, I32X4, builder); 1999 state.push1(builder.ins().fcvt_from_sint(F32X4, a)) 2000 } 2001 Operator::F32x4ConvertI32x4U => { 2002 let a = pop1_with_bitcast(state, I32X4, builder); 2003 state.push1(builder.ins().fcvt_from_uint(F32X4, a)) 2004 } 2005 Operator::F64x2ConvertLowI32x4S => { 2006 let a = pop1_with_bitcast(state, I32X4, builder); 2007 let widened_a = builder.ins().swiden_low(a); 2008 state.push1(builder.ins().fcvt_from_sint(F64X2, widened_a)); 2009 } 2010 Operator::F64x2ConvertLowI32x4U => { 2011 let a = pop1_with_bitcast(state, I32X4, builder); 2012 let widened_a = builder.ins().uwiden_low(a); 2013 state.push1(builder.ins().fcvt_from_uint(F64X2, widened_a)); 2014 } 2015 Operator::F64x2PromoteLowF32x4 => { 2016 let a = pop1_with_bitcast(state, F32X4, builder); 2017 state.push1(builder.ins().fvpromote_low(a)); 2018 } 2019 Operator::F32x4DemoteF64x2Zero => { 2020 let a = pop1_with_bitcast(state, F64X2, builder); 2021 state.push1(builder.ins().fvdemote(a)); 2022 } 2023 Operator::I32x4TruncSatF32x4S => { 2024 let a = pop1_with_bitcast(state, F32X4, builder); 2025 state.push1(builder.ins().fcvt_to_sint_sat(I32X4, a)) 2026 } 2027 Operator::I32x4TruncSatF64x2SZero => { 2028 let a = pop1_with_bitcast(state, F64X2, builder); 2029 let converted_a = builder.ins().fcvt_to_sint_sat(I64X2, a); 2030 let handle = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2031 let zero = builder.ins().vconst(I64X2, handle); 2032 2033 state.push1(builder.ins().snarrow(converted_a, zero)); 2034 } 2035 2036 // FIXME(#5913): the relaxed instructions here are translated the same 2037 // as the saturating instructions, even when the code generator 2038 // configuration allow for different semantics across hosts. On x86, 2039 // however, it's theoretically possible to have a slightly more optimal 2040 // lowering which accounts for NaN differently, although the lowering is 2041 // still not trivial (e.g. one instruction). At this time the 2042 // more-optimal-but-still-large lowering for x86 is not implemented so 2043 // the relaxed instructions are listed here instead of down below with 2044 // the other relaxed instructions. An x86-specific implementation (or 2045 // perhaps for other backends too) should be added and the codegen for 2046 // the relaxed instruction should conditionally be different. 2047 Operator::I32x4RelaxedTruncF32x4U | Operator::I32x4TruncSatF32x4U => { 2048 let a = pop1_with_bitcast(state, F32X4, builder); 2049 state.push1(builder.ins().fcvt_to_uint_sat(I32X4, a)) 2050 } 2051 Operator::I32x4RelaxedTruncF64x2UZero | Operator::I32x4TruncSatF64x2UZero => { 2052 let a = pop1_with_bitcast(state, F64X2, builder); 2053 let zero_constant = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2054 let result = if environ.is_x86() && !environ.isa().has_round() { 2055 // On x86 the vector lowering for `fcvt_to_uint_sat` requires 2056 // SSE4.1 `round` instructions. If SSE4.1 isn't available it 2057 // falls back to a libcall which we don't want in Wasmtime. 2058 // Handle this by falling back to the scalar implementation 2059 // which does not require SSE4.1 instructions. 2060 let lane0 = builder.ins().extractlane(a, 0); 2061 let lane1 = builder.ins().extractlane(a, 1); 2062 let lane0_rounded = builder.ins().fcvt_to_uint_sat(I32, lane0); 2063 let lane1_rounded = builder.ins().fcvt_to_uint_sat(I32, lane1); 2064 let result = builder.ins().vconst(I32X4, zero_constant); 2065 let result = builder.ins().insertlane(result, lane0_rounded, 0); 2066 builder.ins().insertlane(result, lane1_rounded, 1) 2067 } else { 2068 let converted_a = builder.ins().fcvt_to_uint_sat(I64X2, a); 2069 let zero = builder.ins().vconst(I64X2, zero_constant); 2070 builder.ins().uunarrow(converted_a, zero) 2071 }; 2072 state.push1(result); 2073 } 2074 2075 Operator::I8x16NarrowI16x8S => { 2076 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2077 state.push1(builder.ins().snarrow(a, b)) 2078 } 2079 Operator::I16x8NarrowI32x4S => { 2080 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2081 state.push1(builder.ins().snarrow(a, b)) 2082 } 2083 Operator::I8x16NarrowI16x8U => { 2084 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2085 state.push1(builder.ins().unarrow(a, b)) 2086 } 2087 Operator::I16x8NarrowI32x4U => { 2088 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2089 state.push1(builder.ins().unarrow(a, b)) 2090 } 2091 Operator::I16x8ExtendLowI8x16S => { 2092 let a = pop1_with_bitcast(state, I8X16, builder); 2093 state.push1(builder.ins().swiden_low(a)) 2094 } 2095 Operator::I16x8ExtendHighI8x16S => { 2096 let a = pop1_with_bitcast(state, I8X16, builder); 2097 state.push1(builder.ins().swiden_high(a)) 2098 } 2099 Operator::I16x8ExtendLowI8x16U => { 2100 let a = pop1_with_bitcast(state, I8X16, builder); 2101 state.push1(builder.ins().uwiden_low(a)) 2102 } 2103 Operator::I16x8ExtendHighI8x16U => { 2104 let a = pop1_with_bitcast(state, I8X16, builder); 2105 state.push1(builder.ins().uwiden_high(a)) 2106 } 2107 Operator::I32x4ExtendLowI16x8S => { 2108 let a = pop1_with_bitcast(state, I16X8, builder); 2109 state.push1(builder.ins().swiden_low(a)) 2110 } 2111 Operator::I32x4ExtendHighI16x8S => { 2112 let a = pop1_with_bitcast(state, I16X8, builder); 2113 state.push1(builder.ins().swiden_high(a)) 2114 } 2115 Operator::I32x4ExtendLowI16x8U => { 2116 let a = pop1_with_bitcast(state, I16X8, builder); 2117 state.push1(builder.ins().uwiden_low(a)) 2118 } 2119 Operator::I32x4ExtendHighI16x8U => { 2120 let a = pop1_with_bitcast(state, I16X8, builder); 2121 state.push1(builder.ins().uwiden_high(a)) 2122 } 2123 Operator::I64x2ExtendLowI32x4S => { 2124 let a = pop1_with_bitcast(state, I32X4, builder); 2125 state.push1(builder.ins().swiden_low(a)) 2126 } 2127 Operator::I64x2ExtendHighI32x4S => { 2128 let a = pop1_with_bitcast(state, I32X4, builder); 2129 state.push1(builder.ins().swiden_high(a)) 2130 } 2131 Operator::I64x2ExtendLowI32x4U => { 2132 let a = pop1_with_bitcast(state, I32X4, builder); 2133 state.push1(builder.ins().uwiden_low(a)) 2134 } 2135 Operator::I64x2ExtendHighI32x4U => { 2136 let a = pop1_with_bitcast(state, I32X4, builder); 2137 state.push1(builder.ins().uwiden_high(a)) 2138 } 2139 Operator::I16x8ExtAddPairwiseI8x16S => { 2140 let a = pop1_with_bitcast(state, I8X16, builder); 2141 let widen_low = builder.ins().swiden_low(a); 2142 let widen_high = builder.ins().swiden_high(a); 2143 state.push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2144 } 2145 Operator::I32x4ExtAddPairwiseI16x8S => { 2146 let a = pop1_with_bitcast(state, I16X8, builder); 2147 let widen_low = builder.ins().swiden_low(a); 2148 let widen_high = builder.ins().swiden_high(a); 2149 state.push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2150 } 2151 Operator::I16x8ExtAddPairwiseI8x16U => { 2152 let a = pop1_with_bitcast(state, I8X16, builder); 2153 let widen_low = builder.ins().uwiden_low(a); 2154 let widen_high = builder.ins().uwiden_high(a); 2155 state.push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2156 } 2157 Operator::I32x4ExtAddPairwiseI16x8U => { 2158 let a = pop1_with_bitcast(state, I16X8, builder); 2159 let widen_low = builder.ins().uwiden_low(a); 2160 let widen_high = builder.ins().uwiden_high(a); 2161 state.push1(builder.ins().iadd_pairwise(widen_low, widen_high)); 2162 } 2163 Operator::F32x4Ceil => { 2164 let arg = pop1_with_bitcast(state, F32X4, builder); 2165 state.push1(environ.ceil_f32x4(builder, arg)); 2166 } 2167 Operator::F64x2Ceil => { 2168 let arg = pop1_with_bitcast(state, F64X2, builder); 2169 state.push1(environ.ceil_f64x2(builder, arg)); 2170 } 2171 Operator::F32x4Floor => { 2172 let arg = pop1_with_bitcast(state, F32X4, builder); 2173 state.push1(environ.floor_f32x4(builder, arg)); 2174 } 2175 Operator::F64x2Floor => { 2176 let arg = pop1_with_bitcast(state, F64X2, builder); 2177 state.push1(environ.floor_f64x2(builder, arg)); 2178 } 2179 Operator::F32x4Trunc => { 2180 let arg = pop1_with_bitcast(state, F32X4, builder); 2181 state.push1(environ.trunc_f32x4(builder, arg)); 2182 } 2183 Operator::F64x2Trunc => { 2184 let arg = pop1_with_bitcast(state, F64X2, builder); 2185 state.push1(environ.trunc_f64x2(builder, arg)); 2186 } 2187 Operator::F32x4Nearest => { 2188 let arg = pop1_with_bitcast(state, F32X4, builder); 2189 state.push1(environ.nearest_f32x4(builder, arg)); 2190 } 2191 Operator::F64x2Nearest => { 2192 let arg = pop1_with_bitcast(state, F64X2, builder); 2193 state.push1(environ.nearest_f64x2(builder, arg)); 2194 } 2195 Operator::I32x4DotI16x8S => { 2196 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2197 let alow = builder.ins().swiden_low(a); 2198 let blow = builder.ins().swiden_low(b); 2199 let low = builder.ins().imul(alow, blow); 2200 let ahigh = builder.ins().swiden_high(a); 2201 let bhigh = builder.ins().swiden_high(b); 2202 let high = builder.ins().imul(ahigh, bhigh); 2203 state.push1(builder.ins().iadd_pairwise(low, high)); 2204 } 2205 Operator::I8x16Popcnt => { 2206 let arg = pop1_with_bitcast(state, type_of(op), builder); 2207 state.push1(builder.ins().popcnt(arg)); 2208 } 2209 Operator::I16x8Q15MulrSatS => { 2210 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2211 state.push1(builder.ins().sqmul_round_sat(a, b)) 2212 } 2213 Operator::I16x8ExtMulLowI8x16S => { 2214 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2215 let a_low = builder.ins().swiden_low(a); 2216 let b_low = builder.ins().swiden_low(b); 2217 state.push1(builder.ins().imul(a_low, b_low)); 2218 } 2219 Operator::I16x8ExtMulHighI8x16S => { 2220 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2221 let a_high = builder.ins().swiden_high(a); 2222 let b_high = builder.ins().swiden_high(b); 2223 state.push1(builder.ins().imul(a_high, b_high)); 2224 } 2225 Operator::I16x8ExtMulLowI8x16U => { 2226 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2227 let a_low = builder.ins().uwiden_low(a); 2228 let b_low = builder.ins().uwiden_low(b); 2229 state.push1(builder.ins().imul(a_low, b_low)); 2230 } 2231 Operator::I16x8ExtMulHighI8x16U => { 2232 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2233 let a_high = builder.ins().uwiden_high(a); 2234 let b_high = builder.ins().uwiden_high(b); 2235 state.push1(builder.ins().imul(a_high, b_high)); 2236 } 2237 Operator::I32x4ExtMulLowI16x8S => { 2238 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2239 let a_low = builder.ins().swiden_low(a); 2240 let b_low = builder.ins().swiden_low(b); 2241 state.push1(builder.ins().imul(a_low, b_low)); 2242 } 2243 Operator::I32x4ExtMulHighI16x8S => { 2244 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2245 let a_high = builder.ins().swiden_high(a); 2246 let b_high = builder.ins().swiden_high(b); 2247 state.push1(builder.ins().imul(a_high, b_high)); 2248 } 2249 Operator::I32x4ExtMulLowI16x8U => { 2250 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2251 let a_low = builder.ins().uwiden_low(a); 2252 let b_low = builder.ins().uwiden_low(b); 2253 state.push1(builder.ins().imul(a_low, b_low)); 2254 } 2255 Operator::I32x4ExtMulHighI16x8U => { 2256 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2257 let a_high = builder.ins().uwiden_high(a); 2258 let b_high = builder.ins().uwiden_high(b); 2259 state.push1(builder.ins().imul(a_high, b_high)); 2260 } 2261 Operator::I64x2ExtMulLowI32x4S => { 2262 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2263 let a_low = builder.ins().swiden_low(a); 2264 let b_low = builder.ins().swiden_low(b); 2265 state.push1(builder.ins().imul(a_low, b_low)); 2266 } 2267 Operator::I64x2ExtMulHighI32x4S => { 2268 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2269 let a_high = builder.ins().swiden_high(a); 2270 let b_high = builder.ins().swiden_high(b); 2271 state.push1(builder.ins().imul(a_high, b_high)); 2272 } 2273 Operator::I64x2ExtMulLowI32x4U => { 2274 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2275 let a_low = builder.ins().uwiden_low(a); 2276 let b_low = builder.ins().uwiden_low(b); 2277 state.push1(builder.ins().imul(a_low, b_low)); 2278 } 2279 Operator::I64x2ExtMulHighI32x4U => { 2280 let (a, b) = pop2_with_bitcast(state, I32X4, builder); 2281 let a_high = builder.ins().uwiden_high(a); 2282 let b_high = builder.ins().uwiden_high(b); 2283 state.push1(builder.ins().imul(a_high, b_high)); 2284 } 2285 Operator::MemoryDiscard { .. } => { 2286 return Err(wasm_unsupported!( 2287 "proposed memory-control operator {:?}", 2288 op 2289 )); 2290 } 2291 2292 Operator::F32x4RelaxedMax | Operator::F64x2RelaxedMax => { 2293 let ty = type_of(op); 2294 let (a, b) = pop2_with_bitcast(state, ty, builder); 2295 state.push1( 2296 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2297 // Deterministic semantics match the `fmax` instruction, or 2298 // the `fAAxBB.max` wasm instruction. 2299 builder.ins().fmax(a, b) 2300 } else { 2301 // Note that this matches the `pmax` translation which has 2302 // careful ordering of its operands to trigger 2303 // pattern-matches in the x86 backend. 2304 let cmp = builder.ins().fcmp(FloatCC::LessThan, a, b); 2305 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2306 builder.ins().bitselect(cmp, b, a) 2307 }, 2308 ) 2309 } 2310 2311 Operator::F32x4RelaxedMin | Operator::F64x2RelaxedMin => { 2312 let ty = type_of(op); 2313 let (a, b) = pop2_with_bitcast(state, ty, builder); 2314 state.push1( 2315 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2316 // Deterministic semantics match the `fmin` instruction, or 2317 // the `fAAxBB.min` wasm instruction. 2318 builder.ins().fmin(a, b) 2319 } else { 2320 // Note that this matches the `pmin` translation which has 2321 // careful ordering of its operands to trigger 2322 // pattern-matches in the x86 backend. 2323 let cmp = builder.ins().fcmp(FloatCC::LessThan, b, a); 2324 let cmp = optionally_bitcast_vector(cmp, ty, builder); 2325 builder.ins().bitselect(cmp, b, a) 2326 }, 2327 ); 2328 } 2329 2330 Operator::I8x16RelaxedSwizzle => { 2331 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2332 state.push1(environ.relaxed_swizzle(builder, a, b)); 2333 } 2334 2335 Operator::F32x4RelaxedMadd => { 2336 let (a, b, c) = pop3_with_bitcast(state, type_of(op), builder); 2337 state.push1(environ.fma_f32x4(builder, a, b, c)); 2338 } 2339 Operator::F64x2RelaxedMadd => { 2340 let (a, b, c) = pop3_with_bitcast(state, type_of(op), builder); 2341 state.push1(environ.fma_f64x2(builder, a, b, c)); 2342 } 2343 Operator::F32x4RelaxedNmadd => { 2344 let (a, b, c) = pop3_with_bitcast(state, type_of(op), builder); 2345 let a = builder.ins().fneg(a); 2346 state.push1(environ.fma_f32x4(builder, a, b, c)); 2347 } 2348 Operator::F64x2RelaxedNmadd => { 2349 let (a, b, c) = pop3_with_bitcast(state, type_of(op), builder); 2350 let a = builder.ins().fneg(a); 2351 state.push1(environ.fma_f64x2(builder, a, b, c)); 2352 } 2353 2354 Operator::I8x16RelaxedLaneselect 2355 | Operator::I16x8RelaxedLaneselect 2356 | Operator::I32x4RelaxedLaneselect 2357 | Operator::I64x2RelaxedLaneselect => { 2358 let ty = type_of(op); 2359 let (a, b, c) = pop3_with_bitcast(state, ty, builder); 2360 // Note that the variable swaps here are intentional due to 2361 // the difference of the order of the wasm op and the clif 2362 // op. 2363 state.push1( 2364 if environ.relaxed_simd_deterministic() 2365 || !environ.use_x86_blendv_for_relaxed_laneselect(ty) 2366 { 2367 // Deterministic semantics are a `bitselect` along the lines 2368 // of the wasm `v128.bitselect` instruction. 2369 builder.ins().bitselect(c, a, b) 2370 } else { 2371 builder.ins().x86_blendv(c, a, b) 2372 }, 2373 ); 2374 } 2375 2376 Operator::I32x4RelaxedTruncF32x4S => { 2377 let a = pop1_with_bitcast(state, F32X4, builder); 2378 state.push1( 2379 if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2380 // Deterministic semantics are to match the 2381 // `i32x4.trunc_sat_f32x4_s` instruction. 2382 builder.ins().fcvt_to_sint_sat(I32X4, a) 2383 } else { 2384 builder.ins().x86_cvtt2dq(I32X4, a) 2385 }, 2386 ) 2387 } 2388 Operator::I32x4RelaxedTruncF64x2SZero => { 2389 let a = pop1_with_bitcast(state, F64X2, builder); 2390 let converted_a = if environ.relaxed_simd_deterministic() || !environ.is_x86() { 2391 // Deterministic semantics are to match the 2392 // `i32x4.trunc_sat_f64x2_s_zero` instruction. 2393 builder.ins().fcvt_to_sint_sat(I64X2, a) 2394 } else { 2395 builder.ins().x86_cvtt2dq(I64X2, a) 2396 }; 2397 let handle = builder.func.dfg.constants.insert(vec![0u8; 16].into()); 2398 let zero = builder.ins().vconst(I64X2, handle); 2399 2400 state.push1(builder.ins().snarrow(converted_a, zero)); 2401 } 2402 Operator::I16x8RelaxedQ15mulrS => { 2403 let (a, b) = pop2_with_bitcast(state, I16X8, builder); 2404 state.push1( 2405 if environ.relaxed_simd_deterministic() 2406 || !environ.use_x86_pmulhrsw_for_relaxed_q15mul() 2407 { 2408 // Deterministic semantics are to match the 2409 // `i16x8.q15mulr_sat_s` instruction. 2410 builder.ins().sqmul_round_sat(a, b) 2411 } else { 2412 builder.ins().x86_pmulhrsw(a, b) 2413 }, 2414 ); 2415 } 2416 Operator::I16x8RelaxedDotI8x16I7x16S => { 2417 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2418 state.push1( 2419 if environ.relaxed_simd_deterministic() || !environ.use_x86_pmaddubsw_for_dot() { 2420 // Deterministic semantics are to treat both operands as 2421 // signed integers and perform the dot product. 2422 let alo = builder.ins().swiden_low(a); 2423 let blo = builder.ins().swiden_low(b); 2424 let lo = builder.ins().imul(alo, blo); 2425 let ahi = builder.ins().swiden_high(a); 2426 let bhi = builder.ins().swiden_high(b); 2427 let hi = builder.ins().imul(ahi, bhi); 2428 builder.ins().iadd_pairwise(lo, hi) 2429 } else { 2430 builder.ins().x86_pmaddubsw(a, b) 2431 }, 2432 ); 2433 } 2434 2435 Operator::I32x4RelaxedDotI8x16I7x16AddS => { 2436 let c = pop1_with_bitcast(state, I32X4, builder); 2437 let (a, b) = pop2_with_bitcast(state, I8X16, builder); 2438 let dot = 2439 if environ.relaxed_simd_deterministic() || !environ.use_x86_pmaddubsw_for_dot() { 2440 // Deterministic semantics are to treat both operands as 2441 // signed integers and perform the dot product. 2442 let alo = builder.ins().swiden_low(a); 2443 let blo = builder.ins().swiden_low(b); 2444 let lo = builder.ins().imul(alo, blo); 2445 let ahi = builder.ins().swiden_high(a); 2446 let bhi = builder.ins().swiden_high(b); 2447 let hi = builder.ins().imul(ahi, bhi); 2448 builder.ins().iadd_pairwise(lo, hi) 2449 } else { 2450 builder.ins().x86_pmaddubsw(a, b) 2451 }; 2452 let dotlo = builder.ins().swiden_low(dot); 2453 let dothi = builder.ins().swiden_high(dot); 2454 let dot32 = builder.ins().iadd_pairwise(dotlo, dothi); 2455 state.push1(builder.ins().iadd(dot32, c)); 2456 } 2457 2458 Operator::BrOnNull { relative_depth } => { 2459 let r = state.pop1(); 2460 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2461 unreachable!("validation") 2462 }; 2463 let (br_destination, inputs) = translate_br_if_args(*relative_depth, state); 2464 let is_null = environ.translate_ref_is_null(builder.cursor(), r, *r_ty)?; 2465 let else_block = builder.create_block(); 2466 canonicalise_brif(builder, is_null, br_destination, inputs, else_block, &[]); 2467 2468 builder.seal_block(else_block); // The only predecessor is the current block. 2469 builder.switch_to_block(else_block); 2470 state.push1(r); 2471 } 2472 Operator::BrOnNonNull { relative_depth } => { 2473 // We write this a bit differently from the spec to avoid an extra 2474 // block/branch and the typed accounting thereof. Instead of the 2475 // spec's approach, it's described as such: 2476 // Peek the value val from the stack. 2477 // If val is ref.null ht, then: pop the value val from the stack. 2478 // Else: Execute the instruction (br relative_depth). 2479 let r = state.peek1(); 2480 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2481 unreachable!("validation") 2482 }; 2483 let is_null = environ.translate_ref_is_null(builder.cursor(), r, *r_ty)?; 2484 let (br_destination, inputs) = translate_br_if_args(*relative_depth, state); 2485 let else_block = builder.create_block(); 2486 canonicalise_brif(builder, is_null, else_block, &[], br_destination, inputs); 2487 2488 // In the null case, pop the ref 2489 state.pop1(); 2490 2491 builder.seal_block(else_block); // The only predecessor is the current block. 2492 2493 // The rest of the translation operates on our is null case, which is 2494 // currently an empty block 2495 builder.switch_to_block(else_block); 2496 } 2497 Operator::CallRef { type_index } => { 2498 // Get function signature 2499 // `index` is the index of the function's signature and `table_index` is the index of 2500 // the table to search the function in. 2501 let (sigref, num_args) = state.get_indirect_sig(builder.func, *type_index, environ)?; 2502 let callee = state.pop1(); 2503 2504 // Bitcast any vector arguments to their default type, I8X16, before calling. 2505 let args = state.peekn_mut(num_args); 2506 bitcast_wasm_params(environ, sigref, args, builder); 2507 2508 let call = 2509 environ.translate_call_ref(builder, sigref, callee, state.peekn(num_args))?; 2510 2511 let inst_results = builder.inst_results(call); 2512 debug_assert_eq!( 2513 inst_results.len(), 2514 builder.func.dfg.signatures[sigref].returns.len(), 2515 "translate_call_ref results should match the call signature" 2516 ); 2517 state.popn(num_args); 2518 state.pushn(inst_results); 2519 } 2520 Operator::RefAsNonNull => { 2521 let r = state.pop1(); 2522 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2523 unreachable!("validation") 2524 }; 2525 let is_null = environ.translate_ref_is_null(builder.cursor(), r, *r_ty)?; 2526 environ.trapnz(builder, is_null, crate::TRAP_NULL_REFERENCE); 2527 state.push1(r); 2528 } 2529 2530 Operator::RefI31 => { 2531 let val = state.pop1(); 2532 let i31ref = environ.translate_ref_i31(builder.cursor(), val)?; 2533 state.push1(i31ref); 2534 } 2535 Operator::I31GetS => { 2536 let i31ref = state.pop1(); 2537 let val = environ.translate_i31_get_s(builder, i31ref)?; 2538 state.push1(val); 2539 } 2540 Operator::I31GetU => { 2541 let i31ref = state.pop1(); 2542 let val = environ.translate_i31_get_u(builder, i31ref)?; 2543 state.push1(val); 2544 } 2545 2546 Operator::StructNew { struct_type_index } => { 2547 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2548 let arity = environ.struct_fields_len(struct_type_index)?; 2549 let fields: StructFieldsVec = state.peekn(arity).iter().copied().collect(); 2550 state.popn(arity); 2551 let struct_ref = environ.translate_struct_new(builder, struct_type_index, fields)?; 2552 state.push1(struct_ref); 2553 } 2554 2555 Operator::StructNewDefault { struct_type_index } => { 2556 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2557 let struct_ref = environ.translate_struct_new_default(builder, struct_type_index)?; 2558 state.push1(struct_ref); 2559 } 2560 2561 Operator::StructSet { 2562 struct_type_index, 2563 field_index, 2564 } => { 2565 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2566 let val = state.pop1(); 2567 let struct_ref = state.pop1(); 2568 environ.translate_struct_set( 2569 builder, 2570 struct_type_index, 2571 *field_index, 2572 struct_ref, 2573 val, 2574 )?; 2575 } 2576 2577 Operator::StructGetS { 2578 struct_type_index, 2579 field_index, 2580 } => { 2581 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2582 let struct_ref = state.pop1(); 2583 let val = environ.translate_struct_get( 2584 builder, 2585 struct_type_index, 2586 *field_index, 2587 struct_ref, 2588 Some(Extension::Sign), 2589 )?; 2590 state.push1(val); 2591 } 2592 2593 Operator::StructGetU { 2594 struct_type_index, 2595 field_index, 2596 } => { 2597 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2598 let struct_ref = state.pop1(); 2599 let val = environ.translate_struct_get( 2600 builder, 2601 struct_type_index, 2602 *field_index, 2603 struct_ref, 2604 Some(Extension::Zero), 2605 )?; 2606 state.push1(val); 2607 } 2608 2609 Operator::StructGet { 2610 struct_type_index, 2611 field_index, 2612 } => { 2613 let struct_type_index = TypeIndex::from_u32(*struct_type_index); 2614 let struct_ref = state.pop1(); 2615 let val = environ.translate_struct_get( 2616 builder, 2617 struct_type_index, 2618 *field_index, 2619 struct_ref, 2620 None, 2621 )?; 2622 state.push1(val); 2623 } 2624 2625 Operator::TryTable { .. } | Operator::ThrowRef => { 2626 return Err(wasm_unsupported!( 2627 "exception operators are not yet implemented" 2628 )); 2629 } 2630 2631 Operator::ArrayNew { array_type_index } => { 2632 let array_type_index = TypeIndex::from_u32(*array_type_index); 2633 let (elem, len) = state.pop2(); 2634 let array_ref = environ.translate_array_new(builder, array_type_index, elem, len)?; 2635 state.push1(array_ref); 2636 } 2637 Operator::ArrayNewDefault { array_type_index } => { 2638 let array_type_index = TypeIndex::from_u32(*array_type_index); 2639 let len = state.pop1(); 2640 let array_ref = environ.translate_array_new_default(builder, array_type_index, len)?; 2641 state.push1(array_ref); 2642 } 2643 Operator::ArrayNewFixed { 2644 array_type_index, 2645 array_size, 2646 } => { 2647 let array_type_index = TypeIndex::from_u32(*array_type_index); 2648 let array_size = usize::try_from(*array_size).unwrap(); 2649 let elems = state.peekn(array_size); 2650 let array_ref = environ.translate_array_new_fixed(builder, array_type_index, elems)?; 2651 state.popn(array_size); 2652 state.push1(array_ref); 2653 } 2654 Operator::ArrayNewData { 2655 array_type_index, 2656 array_data_index, 2657 } => { 2658 let array_type_index = TypeIndex::from_u32(*array_type_index); 2659 let array_data_index = DataIndex::from_u32(*array_data_index); 2660 let (data_offset, len) = state.pop2(); 2661 let array_ref = environ.translate_array_new_data( 2662 builder, 2663 array_type_index, 2664 array_data_index, 2665 data_offset, 2666 len, 2667 )?; 2668 state.push1(array_ref); 2669 } 2670 Operator::ArrayNewElem { 2671 array_type_index, 2672 array_elem_index, 2673 } => { 2674 let array_type_index = TypeIndex::from_u32(*array_type_index); 2675 let array_elem_index = ElemIndex::from_u32(*array_elem_index); 2676 let (elem_offset, len) = state.pop2(); 2677 let array_ref = environ.translate_array_new_elem( 2678 builder, 2679 array_type_index, 2680 array_elem_index, 2681 elem_offset, 2682 len, 2683 )?; 2684 state.push1(array_ref); 2685 } 2686 Operator::ArrayCopy { 2687 array_type_index_dst, 2688 array_type_index_src, 2689 } => { 2690 let array_type_index_dst = TypeIndex::from_u32(*array_type_index_dst); 2691 let array_type_index_src = TypeIndex::from_u32(*array_type_index_src); 2692 let (dst_array, dst_index, src_array, src_index, len) = state.pop5(); 2693 environ.translate_array_copy( 2694 builder, 2695 array_type_index_dst, 2696 dst_array, 2697 dst_index, 2698 array_type_index_src, 2699 src_array, 2700 src_index, 2701 len, 2702 )?; 2703 } 2704 Operator::ArrayFill { array_type_index } => { 2705 let array_type_index = TypeIndex::from_u32(*array_type_index); 2706 let (array, index, val, len) = state.pop4(); 2707 environ.translate_array_fill(builder, array_type_index, array, index, val, len)?; 2708 } 2709 Operator::ArrayInitData { 2710 array_type_index, 2711 array_data_index, 2712 } => { 2713 let array_type_index = TypeIndex::from_u32(*array_type_index); 2714 let array_data_index = DataIndex::from_u32(*array_data_index); 2715 let (array, dst_index, src_index, len) = state.pop4(); 2716 environ.translate_array_init_data( 2717 builder, 2718 array_type_index, 2719 array, 2720 dst_index, 2721 array_data_index, 2722 src_index, 2723 len, 2724 )?; 2725 } 2726 Operator::ArrayInitElem { 2727 array_type_index, 2728 array_elem_index, 2729 } => { 2730 let array_type_index = TypeIndex::from_u32(*array_type_index); 2731 let array_elem_index = ElemIndex::from_u32(*array_elem_index); 2732 let (array, dst_index, src_index, len) = state.pop4(); 2733 environ.translate_array_init_elem( 2734 builder, 2735 array_type_index, 2736 array, 2737 dst_index, 2738 array_elem_index, 2739 src_index, 2740 len, 2741 )?; 2742 } 2743 Operator::ArrayLen => { 2744 let array = state.pop1(); 2745 let len = environ.translate_array_len(builder, array)?; 2746 state.push1(len); 2747 } 2748 Operator::ArrayGet { array_type_index } => { 2749 let array_type_index = TypeIndex::from_u32(*array_type_index); 2750 let (array, index) = state.pop2(); 2751 let elem = 2752 environ.translate_array_get(builder, array_type_index, array, index, None)?; 2753 state.push1(elem); 2754 } 2755 Operator::ArrayGetS { array_type_index } => { 2756 let array_type_index = TypeIndex::from_u32(*array_type_index); 2757 let (array, index) = state.pop2(); 2758 let elem = environ.translate_array_get( 2759 builder, 2760 array_type_index, 2761 array, 2762 index, 2763 Some(Extension::Sign), 2764 )?; 2765 state.push1(elem); 2766 } 2767 Operator::ArrayGetU { array_type_index } => { 2768 let array_type_index = TypeIndex::from_u32(*array_type_index); 2769 let (array, index) = state.pop2(); 2770 let elem = environ.translate_array_get( 2771 builder, 2772 array_type_index, 2773 array, 2774 index, 2775 Some(Extension::Zero), 2776 )?; 2777 state.push1(elem); 2778 } 2779 Operator::ArraySet { array_type_index } => { 2780 let array_type_index = TypeIndex::from_u32(*array_type_index); 2781 let (array, index, elem) = state.pop3(); 2782 environ.translate_array_set(builder, array_type_index, array, index, elem)?; 2783 } 2784 Operator::RefEq => { 2785 let (r1, r2) = state.pop2(); 2786 let eq = builder.ins().icmp(ir::condcodes::IntCC::Equal, r1, r2); 2787 let eq = builder.ins().uextend(ir::types::I32, eq); 2788 state.push1(eq); 2789 } 2790 Operator::RefTestNonNull { hty } => { 2791 let r = state.pop1(); 2792 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2793 unreachable!("validation") 2794 }; 2795 let heap_type = environ.convert_heap_type(*hty)?; 2796 let result = environ.translate_ref_test( 2797 builder, 2798 WasmRefType { 2799 heap_type, 2800 nullable: false, 2801 }, 2802 r, 2803 *r_ty, 2804 )?; 2805 state.push1(result); 2806 } 2807 Operator::RefTestNullable { hty } => { 2808 let r = state.pop1(); 2809 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2810 unreachable!("validation") 2811 }; 2812 let heap_type = environ.convert_heap_type(*hty)?; 2813 let result = environ.translate_ref_test( 2814 builder, 2815 WasmRefType { 2816 heap_type, 2817 nullable: true, 2818 }, 2819 r, 2820 *r_ty, 2821 )?; 2822 state.push1(result); 2823 } 2824 Operator::RefCastNonNull { hty } => { 2825 let r = state.pop1(); 2826 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2827 unreachable!("validation") 2828 }; 2829 let heap_type = environ.convert_heap_type(*hty)?; 2830 let cast_okay = environ.translate_ref_test( 2831 builder, 2832 WasmRefType { 2833 heap_type, 2834 nullable: false, 2835 }, 2836 r, 2837 *r_ty, 2838 )?; 2839 environ.trapz(builder, cast_okay, crate::TRAP_CAST_FAILURE); 2840 state.push1(r); 2841 } 2842 Operator::RefCastNullable { hty } => { 2843 let r = state.pop1(); 2844 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2845 unreachable!("validation") 2846 }; 2847 let heap_type = environ.convert_heap_type(*hty)?; 2848 let cast_okay = environ.translate_ref_test( 2849 builder, 2850 WasmRefType { 2851 heap_type, 2852 nullable: true, 2853 }, 2854 r, 2855 *r_ty, 2856 )?; 2857 environ.trapz(builder, cast_okay, crate::TRAP_CAST_FAILURE); 2858 state.push1(r); 2859 } 2860 Operator::BrOnCast { 2861 relative_depth, 2862 to_ref_type, 2863 from_ref_type: _, 2864 } => { 2865 let r = state.peek1(); 2866 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2867 unreachable!("validation") 2868 }; 2869 2870 let to_ref_type = environ.convert_ref_type(*to_ref_type)?; 2871 let cast_is_okay = environ.translate_ref_test(builder, to_ref_type, r, *r_ty)?; 2872 2873 let (cast_succeeds_block, inputs) = translate_br_if_args(*relative_depth, state); 2874 let cast_fails_block = builder.create_block(); 2875 canonicalise_brif( 2876 builder, 2877 cast_is_okay, 2878 cast_succeeds_block, 2879 inputs, 2880 cast_fails_block, 2881 &[ 2882 // NB: the `cast_fails_block` is dominated by the current 2883 // block, and therefore doesn't need any block params. 2884 ], 2885 ); 2886 2887 // The only predecessor is the current block. 2888 builder.seal_block(cast_fails_block); 2889 2890 // The next Wasm instruction is executed when the cast failed and we 2891 // did not branch away. 2892 builder.switch_to_block(cast_fails_block); 2893 } 2894 Operator::BrOnCastFail { 2895 relative_depth, 2896 to_ref_type, 2897 from_ref_type: _, 2898 } => { 2899 let r = state.peek1(); 2900 let [.., WasmValType::Ref(r_ty)] = operand_types else { 2901 unreachable!("validation") 2902 }; 2903 2904 let to_ref_type = environ.convert_ref_type(*to_ref_type)?; 2905 let cast_is_okay = environ.translate_ref_test(builder, to_ref_type, r, *r_ty)?; 2906 2907 let (cast_fails_block, inputs) = translate_br_if_args(*relative_depth, state); 2908 let cast_succeeds_block = builder.create_block(); 2909 canonicalise_brif( 2910 builder, 2911 cast_is_okay, 2912 cast_succeeds_block, 2913 &[ 2914 // NB: the `cast_succeeds_block` is dominated by the current 2915 // block, and therefore doesn't need any block params. 2916 ], 2917 cast_fails_block, 2918 inputs, 2919 ); 2920 2921 // The only predecessor is the current block. 2922 builder.seal_block(cast_succeeds_block); 2923 2924 // The next Wasm instruction is executed when the cast succeeded and 2925 // we did not branch away. 2926 builder.switch_to_block(cast_succeeds_block); 2927 } 2928 2929 Operator::AnyConvertExtern => { 2930 // Pop an `externref`, push an `anyref`. But they have the same 2931 // representation, so we don't actually need to do anything. 2932 } 2933 Operator::ExternConvertAny => { 2934 // Pop an `anyref`, push an `externref`. But they have the same 2935 // representation, so we don't actually need to do anything. 2936 } 2937 2938 Operator::ContNew { cont_type_index: _ } => { 2939 // TODO(10248) This is added in a follow-up PR 2940 return Err(wasmtime_environ::WasmError::Unsupported( 2941 "codegen for stack switching instructions not implemented, yet".to_string(), 2942 )); 2943 } 2944 Operator::ContBind { 2945 argument_index: _, 2946 result_index: _, 2947 } => { 2948 // TODO(10248) This is added in a follow-up PR 2949 return Err(wasmtime_environ::WasmError::Unsupported( 2950 "codegen for stack switching instructions not implemented, yet".to_string(), 2951 )); 2952 } 2953 Operator::Suspend { tag_index: _ } => { 2954 // TODO(10248) This is added in a follow-up PR 2955 return Err(wasmtime_environ::WasmError::Unsupported( 2956 "codegen for stack switching instructions not implemented, yet".to_string(), 2957 )); 2958 } 2959 Operator::Resume { 2960 cont_type_index: _, 2961 resume_table: _, 2962 } => { 2963 // TODO(10248) This is added in a follow-up PR 2964 return Err(wasmtime_environ::WasmError::Unsupported( 2965 "codegen for stack switching instructions not implemented, yet".to_string(), 2966 )); 2967 } 2968 Operator::ResumeThrow { 2969 cont_type_index: _, 2970 tag_index: _, 2971 resume_table: _, 2972 } => { 2973 // TODO(10248) This depends on exception handling 2974 return Err(wasmtime_environ::WasmError::Unsupported( 2975 "resume.throw instructions not supported, yet".to_string(), 2976 )); 2977 } 2978 Operator::Switch { 2979 cont_type_index: _, 2980 tag_index: _, 2981 } => { 2982 // TODO(10248) This is added in a follow-up PR 2983 return Err(wasmtime_environ::WasmError::Unsupported( 2984 "codegen for stack switching instructions not implemented, yet".to_string(), 2985 )); 2986 } 2987 2988 Operator::GlobalAtomicGet { .. } 2989 | Operator::GlobalAtomicSet { .. } 2990 | Operator::GlobalAtomicRmwAdd { .. } 2991 | Operator::GlobalAtomicRmwSub { .. } 2992 | Operator::GlobalAtomicRmwOr { .. } 2993 | Operator::GlobalAtomicRmwXor { .. } 2994 | Operator::GlobalAtomicRmwAnd { .. } 2995 | Operator::GlobalAtomicRmwXchg { .. } 2996 | Operator::GlobalAtomicRmwCmpxchg { .. } 2997 | Operator::TableAtomicGet { .. } 2998 | Operator::TableAtomicSet { .. } 2999 | Operator::TableAtomicRmwXchg { .. } 3000 | Operator::TableAtomicRmwCmpxchg { .. } 3001 | Operator::StructAtomicGet { .. } 3002 | Operator::StructAtomicGetS { .. } 3003 | Operator::StructAtomicGetU { .. } 3004 | Operator::StructAtomicSet { .. } 3005 | Operator::StructAtomicRmwAdd { .. } 3006 | Operator::StructAtomicRmwSub { .. } 3007 | Operator::StructAtomicRmwOr { .. } 3008 | Operator::StructAtomicRmwXor { .. } 3009 | Operator::StructAtomicRmwAnd { .. } 3010 | Operator::StructAtomicRmwXchg { .. } 3011 | Operator::StructAtomicRmwCmpxchg { .. } 3012 | Operator::ArrayAtomicGet { .. } 3013 | Operator::ArrayAtomicGetS { .. } 3014 | Operator::ArrayAtomicGetU { .. } 3015 | Operator::ArrayAtomicSet { .. } 3016 | Operator::ArrayAtomicRmwAdd { .. } 3017 | Operator::ArrayAtomicRmwSub { .. } 3018 | Operator::ArrayAtomicRmwOr { .. } 3019 | Operator::ArrayAtomicRmwXor { .. } 3020 | Operator::ArrayAtomicRmwAnd { .. } 3021 | Operator::ArrayAtomicRmwXchg { .. } 3022 | Operator::ArrayAtomicRmwCmpxchg { .. } 3023 | Operator::RefI31Shared { .. } => { 3024 return Err(wasm_unsupported!( 3025 "shared-everything-threads operators are not yet implemented" 3026 )); 3027 } 3028 3029 Operator::I64MulWideS => { 3030 let (arg1, arg2) = state.pop2(); 3031 let arg1 = builder.ins().sextend(I128, arg1); 3032 let arg2 = builder.ins().sextend(I128, arg2); 3033 let result = builder.ins().imul(arg1, arg2); 3034 let (lo, hi) = builder.ins().isplit(result); 3035 state.push2(lo, hi); 3036 } 3037 Operator::I64MulWideU => { 3038 let (arg1, arg2) = state.pop2(); 3039 let arg1 = builder.ins().uextend(I128, arg1); 3040 let arg2 = builder.ins().uextend(I128, arg2); 3041 let result = builder.ins().imul(arg1, arg2); 3042 let (lo, hi) = builder.ins().isplit(result); 3043 state.push2(lo, hi); 3044 } 3045 Operator::I64Add128 => { 3046 let (arg1, arg2, arg3, arg4) = state.pop4(); 3047 let arg1 = builder.ins().iconcat(arg1, arg2); 3048 let arg2 = builder.ins().iconcat(arg3, arg4); 3049 let result = builder.ins().iadd(arg1, arg2); 3050 let (res1, res2) = builder.ins().isplit(result); 3051 state.push2(res1, res2); 3052 } 3053 Operator::I64Sub128 => { 3054 let (arg1, arg2, arg3, arg4) = state.pop4(); 3055 let arg1 = builder.ins().iconcat(arg1, arg2); 3056 let arg2 = builder.ins().iconcat(arg3, arg4); 3057 let result = builder.ins().isub(arg1, arg2); 3058 let (res1, res2) = builder.ins().isplit(result); 3059 state.push2(res1, res2); 3060 } 3061 3062 // catch-all as `Operator` is `#[non_exhaustive]` 3063 op => return Err(wasm_unsupported!("operator {op:?}")), 3064 }; 3065 Ok(()) 3066 } 3067 3068 /// Deals with a Wasm instruction located in an unreachable portion of the code. Most of them 3069 /// are dropped but special ones like `End` or `Else` signal the potential end of the unreachable 3070 /// portion so the translation state must be updated accordingly. 3071 fn translate_unreachable_operator( 3072 validator: &FuncValidator<impl WasmModuleResources>, 3073 op: &Operator, 3074 builder: &mut FunctionBuilder, 3075 state: &mut FuncTranslationState, 3076 environ: &mut FuncEnvironment<'_>, 3077 ) -> WasmResult<()> { 3078 debug_assert!(!state.reachable); 3079 match *op { 3080 Operator::If { blockty } => { 3081 // Push a placeholder control stack entry. The if isn't reachable, 3082 // so we don't have any branches anywhere. 3083 state.push_if( 3084 ir::Block::reserved_value(), 3085 ElseData::NoElse { 3086 branch_inst: ir::Inst::reserved_value(), 3087 placeholder: ir::Block::reserved_value(), 3088 }, 3089 0, 3090 0, 3091 blockty, 3092 ); 3093 } 3094 Operator::Loop { blockty: _ } | Operator::Block { blockty: _ } => { 3095 state.push_block(ir::Block::reserved_value(), 0, 0); 3096 } 3097 Operator::Else => { 3098 let i = state.control_stack.len() - 1; 3099 match state.control_stack[i] { 3100 ControlStackFrame::If { 3101 ref else_data, 3102 head_is_reachable, 3103 ref mut consequent_ends_reachable, 3104 blocktype, 3105 .. 3106 } => { 3107 debug_assert!(consequent_ends_reachable.is_none()); 3108 *consequent_ends_reachable = Some(state.reachable); 3109 3110 if head_is_reachable { 3111 // We have a branch from the head of the `if` to the `else`. 3112 state.reachable = true; 3113 3114 let else_block = match *else_data { 3115 ElseData::NoElse { 3116 branch_inst, 3117 placeholder, 3118 } => { 3119 let (params, _results) = 3120 blocktype_params_results(validator, blocktype)?; 3121 let else_block = block_with_params(builder, params, environ)?; 3122 let frame = state.control_stack.last().unwrap(); 3123 frame.truncate_value_stack_to_else_params(&mut state.stack); 3124 3125 // We change the target of the branch instruction. 3126 builder.change_jump_destination( 3127 branch_inst, 3128 placeholder, 3129 else_block, 3130 ); 3131 builder.seal_block(else_block); 3132 else_block 3133 } 3134 ElseData::WithElse { else_block } => { 3135 let frame = state.control_stack.last().unwrap(); 3136 frame.truncate_value_stack_to_else_params(&mut state.stack); 3137 else_block 3138 } 3139 }; 3140 3141 builder.switch_to_block(else_block); 3142 3143 // Again, no need to push the parameters for the `else`, 3144 // since we already did when we saw the original `if`. See 3145 // the comment for translating `Operator::Else` in 3146 // `translate_operator` for details. 3147 } 3148 } 3149 _ => unreachable!(), 3150 } 3151 } 3152 Operator::End => { 3153 let stack = &mut state.stack; 3154 let control_stack = &mut state.control_stack; 3155 let frame = control_stack.pop().unwrap(); 3156 3157 // Pop unused parameters from stack. 3158 frame.truncate_value_stack_to_original_size(stack); 3159 3160 let reachable_anyway = match frame { 3161 // If it is a loop we also have to seal the body loop block 3162 ControlStackFrame::Loop { header, .. } => { 3163 builder.seal_block(header); 3164 // And loops can't have branches to the end. 3165 false 3166 } 3167 // If we never set `consequent_ends_reachable` then that means 3168 // we are finishing the consequent now, and there was no 3169 // `else`. Whether the following block is reachable depends only 3170 // on if the head was reachable. 3171 ControlStackFrame::If { 3172 head_is_reachable, 3173 consequent_ends_reachable: None, 3174 .. 3175 } => head_is_reachable, 3176 // Since we are only in this function when in unreachable code, 3177 // we know that the alternative just ended unreachable. Whether 3178 // the following block is reachable depends on if the consequent 3179 // ended reachable or not. 3180 ControlStackFrame::If { 3181 head_is_reachable, 3182 consequent_ends_reachable: Some(consequent_ends_reachable), 3183 .. 3184 } => head_is_reachable && consequent_ends_reachable, 3185 // All other control constructs are already handled. 3186 _ => false, 3187 }; 3188 3189 if frame.exit_is_branched_to() || reachable_anyway { 3190 builder.switch_to_block(frame.following_code()); 3191 builder.seal_block(frame.following_code()); 3192 3193 // And add the return values of the block but only if the next block is reachable 3194 // (which corresponds to testing if the stack depth is 1) 3195 stack.extend_from_slice(builder.block_params(frame.following_code())); 3196 state.reachable = true; 3197 } 3198 } 3199 _ => { 3200 // We don't translate because this is unreachable code 3201 } 3202 } 3203 3204 Ok(()) 3205 } 3206 3207 /// This function is a generalized helper for validating that a wasm-supplied 3208 /// heap address is in-bounds. 3209 /// 3210 /// This function takes a litany of parameters and requires that the *Wasm* 3211 /// address to be verified is at the top of the stack in `state`. This will 3212 /// generate necessary IR to validate that the heap address is correctly 3213 /// in-bounds, and various parameters are returned describing the valid *native* 3214 /// heap address if execution reaches that point. 3215 /// 3216 /// Returns `None` when the Wasm access will unconditionally trap. 3217 /// 3218 /// Returns `(flags, wasm_addr, native_addr)`. 3219 fn prepare_addr( 3220 memarg: &MemArg, 3221 access_size: u8, 3222 builder: &mut FunctionBuilder, 3223 state: &mut FuncTranslationState, 3224 environ: &mut FuncEnvironment<'_>, 3225 ) -> WasmResult<Reachability<(MemFlags, Value, Value)>> { 3226 let index = state.pop1(); 3227 let heap = state.get_heap(builder.func, memarg.memory, environ)?; 3228 3229 // How exactly the bounds check is performed here and what it's performed 3230 // on is a bit tricky. Generally we want to rely on access violations (e.g. 3231 // segfaults) to generate traps since that means we don't have to bounds 3232 // check anything explicitly. 3233 // 3234 // (1) If we don't have a guard page of unmapped memory, though, then we 3235 // can't rely on this trapping behavior through segfaults. Instead we need 3236 // to bounds-check the entire memory access here which is everything from 3237 // `addr32 + offset` to `addr32 + offset + width` (not inclusive). In this 3238 // scenario our adjusted offset that we're checking is `memarg.offset + 3239 // access_size`. Note that we do saturating arithmetic here to avoid 3240 // overflow. The addition here is in the 64-bit space, which means that 3241 // we'll never overflow for 32-bit wasm but for 64-bit this is an issue. If 3242 // our effective offset is u64::MAX though then it's impossible for for 3243 // that to actually be a valid offset because otherwise the wasm linear 3244 // memory would take all of the host memory! 3245 // 3246 // (2) If we have a guard page, however, then we can perform a further 3247 // optimization of the generated code by only checking multiples of the 3248 // offset-guard size to be more CSE-friendly. Knowing that we have at least 3249 // 1 page of a guard page we're then able to disregard the `width` since we 3250 // know it's always less than one page. Our bounds check will be for the 3251 // first byte which will either succeed and be guaranteed to fault if it's 3252 // actually out of bounds, or the bounds check itself will fail. In any case 3253 // we assert that the width is reasonably small for now so this assumption 3254 // can be adjusted in the future if we get larger widths. 3255 // 3256 // Put another way we can say, where `y < offset_guard_size`: 3257 // 3258 // n * offset_guard_size + y = offset 3259 // 3260 // We'll then pass `n * offset_guard_size` as the bounds check value. If 3261 // this traps then our `offset` would have trapped anyway. If this check 3262 // passes we know 3263 // 3264 // addr32 + n * offset_guard_size < bound 3265 // 3266 // which means 3267 // 3268 // addr32 + n * offset_guard_size + y < bound + offset_guard_size 3269 // 3270 // because `y < offset_guard_size`, which then means: 3271 // 3272 // addr32 + offset < bound + offset_guard_size 3273 // 3274 // Since we know that that guard size bytes are all unmapped we're 3275 // guaranteed that `offset` and the `width` bytes after it are either 3276 // in-bounds or will hit the guard page, meaning we'll get the desired 3277 // semantics we want. 3278 // 3279 // --- 3280 // 3281 // With all that in mind remember that the goal is to bounds check as few 3282 // things as possible. To facilitate this the "fast path" is expected to be 3283 // hit like so: 3284 // 3285 // * For wasm32, wasmtime defaults to 4gb "static" memories with 2gb guard 3286 // regions. This means that for all offsets <=2gb, we hit the optimized 3287 // case for `heap_addr` on static memories 4gb in size in cranelift's 3288 // legalization of `heap_addr`, eliding the bounds check entirely. 3289 // 3290 // * For wasm64 offsets <=2gb will generate a single `heap_addr` 3291 // instruction, but at this time all heaps are "dynamic" which means that 3292 // a single bounds check is forced. Ideally we'd do better here, but 3293 // that's the current state of affairs. 3294 // 3295 // Basically we assume that most configurations have a guard page and most 3296 // offsets in `memarg` are <=2gb, which means we get the fast path of one 3297 // `heap_addr` instruction plus a hardcoded i32-offset in memory-related 3298 // instructions. 3299 let heap = environ.heaps()[heap].clone(); 3300 let addr = match u32::try_from(memarg.offset) { 3301 // If our offset fits within a u32, then we can place the it into the 3302 // offset immediate of the `heap_addr` instruction. 3303 Ok(offset) => bounds_check_and_compute_addr( 3304 builder, 3305 environ, 3306 &heap, 3307 index, 3308 BoundsCheck::StaticOffset { 3309 offset, 3310 access_size, 3311 }, 3312 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3313 ), 3314 3315 // If the offset doesn't fit within a u32, then we can't pass it 3316 // directly into `heap_addr`. 3317 // 3318 // One reasonable question you might ask is "why not?". There's no 3319 // fundamental reason why `heap_addr` *must* take a 32-bit offset. The 3320 // reason this isn't done, though, is that blindly changing the offset 3321 // to a 64-bit offset increases the size of the `InstructionData` enum 3322 // in cranelift by 8 bytes (16 to 24). This can have significant 3323 // performance implications so the conclusion when this was written was 3324 // that we shouldn't do that. 3325 // 3326 // Without the ability to put the whole offset into the `heap_addr` 3327 // instruction we need to fold the offset into the address itself with 3328 // an unsigned addition. In doing so though we need to check for 3329 // overflow because that would mean the address is out-of-bounds (wasm 3330 // bounds checks happen on the effective 33 or 65 bit address once the 3331 // offset is factored in). 3332 // 3333 // Once we have the effective address, offset already folded in, then 3334 // `heap_addr` is used to verify that the address is indeed in-bounds. 3335 // 3336 // Note that this is generating what's likely to be at least two 3337 // branches, one for the overflow and one for the bounds check itself. 3338 // For now though that should hopefully be ok since 4gb+ offsets are 3339 // relatively odd/rare. In the future if needed we can look into 3340 // optimizing this more. 3341 Err(_) => { 3342 let offset = builder 3343 .ins() 3344 .iconst(heap.index_type(), memarg.offset.signed()); 3345 let adjusted_index = environ.uadd_overflow_trap( 3346 builder, 3347 index, 3348 offset, 3349 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3350 ); 3351 bounds_check_and_compute_addr( 3352 builder, 3353 environ, 3354 &heap, 3355 adjusted_index, 3356 BoundsCheck::StaticOffset { 3357 offset: 0, 3358 access_size, 3359 }, 3360 ir::TrapCode::HEAP_OUT_OF_BOUNDS, 3361 ) 3362 } 3363 }; 3364 let addr = match addr { 3365 Reachability::Unreachable => return Ok(Reachability::Unreachable), 3366 Reachability::Reachable(a) => a, 3367 }; 3368 3369 // Note that we don't set `is_aligned` here, even if the load instruction's 3370 // alignment immediate may says it's aligned, because WebAssembly's 3371 // immediate field is just a hint, while Cranelift's aligned flag needs a 3372 // guarantee. WebAssembly memory accesses are always little-endian. 3373 let mut flags = MemFlags::new(); 3374 flags.set_endianness(ir::Endianness::Little); 3375 3376 if heap.pcc_memory_type.is_some() { 3377 // Proof-carrying code is enabled; check this memory access. 3378 flags.set_checked(); 3379 } 3380 3381 // The access occurs to the `heap` disjoint category of abstract 3382 // state. This may allow alias analysis to merge redundant loads, 3383 // etc. when heap accesses occur interleaved with other (table, 3384 // vmctx, stack) accesses. 3385 flags.set_alias_region(Some(ir::AliasRegion::Heap)); 3386 3387 Ok(Reachability::Reachable((flags, index, addr))) 3388 } 3389 3390 fn align_atomic_addr( 3391 memarg: &MemArg, 3392 loaded_bytes: u8, 3393 builder: &mut FunctionBuilder, 3394 state: &mut FuncTranslationState, 3395 environ: &mut FuncEnvironment<'_>, 3396 ) { 3397 // Atomic addresses must all be aligned correctly, and for now we check 3398 // alignment before we check out-of-bounds-ness. The order of this check may 3399 // need to be updated depending on the outcome of the official threads 3400 // proposal itself. 3401 // 3402 // Note that with an offset>0 we generate an `iadd_imm` where the result is 3403 // thrown away after the offset check. This may truncate the offset and the 3404 // result may overflow as well, but those conditions won't affect the 3405 // alignment check itself. This can probably be optimized better and we 3406 // should do so in the future as well. 3407 if loaded_bytes > 1 { 3408 let addr = state.pop1(); // "peek" via pop then push 3409 state.push1(addr); 3410 let effective_addr = if memarg.offset == 0 { 3411 addr 3412 } else { 3413 builder.ins().iadd_imm(addr, memarg.offset.signed()) 3414 }; 3415 debug_assert!(loaded_bytes.is_power_of_two()); 3416 let misalignment = builder 3417 .ins() 3418 .band_imm(effective_addr, i64::from(loaded_bytes - 1)); 3419 let f = builder.ins().icmp_imm(IntCC::NotEqual, misalignment, 0); 3420 environ.trapnz(builder, f, crate::TRAP_HEAP_MISALIGNED); 3421 } 3422 } 3423 3424 /// Like `prepare_addr` but for atomic accesses. 3425 /// 3426 /// Returns `None` when the Wasm access will unconditionally trap. 3427 fn prepare_atomic_addr( 3428 memarg: &MemArg, 3429 loaded_bytes: u8, 3430 builder: &mut FunctionBuilder, 3431 state: &mut FuncTranslationState, 3432 environ: &mut FuncEnvironment<'_>, 3433 ) -> WasmResult<Reachability<(MemFlags, Value, Value)>> { 3434 align_atomic_addr(memarg, loaded_bytes, builder, state, environ); 3435 prepare_addr(memarg, loaded_bytes, builder, state, environ) 3436 } 3437 3438 /// Translate a load instruction. 3439 /// 3440 /// Returns the execution state's reachability after the load is translated. 3441 fn translate_load( 3442 memarg: &MemArg, 3443 opcode: ir::Opcode, 3444 result_ty: Type, 3445 builder: &mut FunctionBuilder, 3446 state: &mut FuncTranslationState, 3447 environ: &mut FuncEnvironment<'_>, 3448 ) -> WasmResult<Reachability<()>> { 3449 let mem_op_size = mem_op_size(opcode, result_ty); 3450 let (flags, wasm_index, base) = 3451 match prepare_addr(memarg, mem_op_size, builder, state, environ)? { 3452 Reachability::Unreachable => return Ok(Reachability::Unreachable), 3453 Reachability::Reachable((f, i, b)) => (f, i, b), 3454 }; 3455 3456 environ.before_load(builder, mem_op_size, wasm_index, memarg.offset); 3457 3458 let (load, dfg) = builder 3459 .ins() 3460 .Load(opcode, result_ty, flags, Offset32::new(0), base); 3461 state.push1(dfg.first_result(load)); 3462 Ok(Reachability::Reachable(())) 3463 } 3464 3465 /// Translate a store instruction. 3466 fn translate_store( 3467 memarg: &MemArg, 3468 opcode: ir::Opcode, 3469 builder: &mut FunctionBuilder, 3470 state: &mut FuncTranslationState, 3471 environ: &mut FuncEnvironment<'_>, 3472 ) -> WasmResult<()> { 3473 let val = state.pop1(); 3474 let val_ty = builder.func.dfg.value_type(val); 3475 let mem_op_size = mem_op_size(opcode, val_ty); 3476 3477 let (flags, wasm_index, base) = unwrap_or_return_unreachable_state!( 3478 state, 3479 prepare_addr(memarg, mem_op_size, builder, state, environ)? 3480 ); 3481 3482 environ.before_store(builder, mem_op_size, wasm_index, memarg.offset); 3483 3484 builder 3485 .ins() 3486 .Store(opcode, val_ty, flags, Offset32::new(0), val, base); 3487 Ok(()) 3488 } 3489 3490 fn mem_op_size(opcode: ir::Opcode, ty: Type) -> u8 { 3491 match opcode { 3492 ir::Opcode::Istore8 | ir::Opcode::Sload8 | ir::Opcode::Uload8 => 1, 3493 ir::Opcode::Istore16 | ir::Opcode::Sload16 | ir::Opcode::Uload16 => 2, 3494 ir::Opcode::Istore32 | ir::Opcode::Sload32 | ir::Opcode::Uload32 => 4, 3495 ir::Opcode::Store | ir::Opcode::Load => u8::try_from(ty.bytes()).unwrap(), 3496 _ => panic!("unknown size of mem op for {opcode:?}"), 3497 } 3498 } 3499 3500 fn translate_icmp(cc: IntCC, builder: &mut FunctionBuilder, state: &mut FuncTranslationState) { 3501 let (arg0, arg1) = state.pop2(); 3502 let val = builder.ins().icmp(cc, arg0, arg1); 3503 state.push1(builder.ins().uextend(I32, val)); 3504 } 3505 3506 fn translate_atomic_rmw( 3507 widened_ty: Type, 3508 access_ty: Type, 3509 op: AtomicRmwOp, 3510 memarg: &MemArg, 3511 builder: &mut FunctionBuilder, 3512 state: &mut FuncTranslationState, 3513 environ: &mut FuncEnvironment<'_>, 3514 ) -> WasmResult<()> { 3515 let mut arg2 = state.pop1(); 3516 let arg2_ty = builder.func.dfg.value_type(arg2); 3517 3518 // The operation is performed at type `access_ty`, and the old value is zero-extended 3519 // to type `widened_ty`. 3520 match access_ty { 3521 I8 | I16 | I32 | I64 => {} 3522 _ => { 3523 return Err(wasm_unsupported!( 3524 "atomic_rmw: unsupported access type {:?}", 3525 access_ty 3526 )); 3527 } 3528 }; 3529 let w_ty_ok = match widened_ty { 3530 I32 | I64 => true, 3531 _ => false, 3532 }; 3533 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3534 3535 assert!(arg2_ty.bytes() >= access_ty.bytes()); 3536 if arg2_ty.bytes() > access_ty.bytes() { 3537 arg2 = builder.ins().ireduce(access_ty, arg2); 3538 } 3539 3540 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3541 state, 3542 prepare_atomic_addr( 3543 memarg, 3544 u8::try_from(access_ty.bytes()).unwrap(), 3545 builder, 3546 state, 3547 environ, 3548 )? 3549 ); 3550 3551 let mut res = builder.ins().atomic_rmw(access_ty, flags, op, addr, arg2); 3552 if access_ty != widened_ty { 3553 res = builder.ins().uextend(widened_ty, res); 3554 } 3555 state.push1(res); 3556 Ok(()) 3557 } 3558 3559 fn translate_atomic_cas( 3560 widened_ty: Type, 3561 access_ty: Type, 3562 memarg: &MemArg, 3563 builder: &mut FunctionBuilder, 3564 state: &mut FuncTranslationState, 3565 environ: &mut FuncEnvironment<'_>, 3566 ) -> WasmResult<()> { 3567 let (mut expected, mut replacement) = state.pop2(); 3568 let expected_ty = builder.func.dfg.value_type(expected); 3569 let replacement_ty = builder.func.dfg.value_type(replacement); 3570 3571 // The compare-and-swap is performed at type `access_ty`, and the old value is zero-extended 3572 // to type `widened_ty`. 3573 match access_ty { 3574 I8 | I16 | I32 | I64 => {} 3575 _ => { 3576 return Err(wasm_unsupported!( 3577 "atomic_cas: unsupported access type {:?}", 3578 access_ty 3579 )); 3580 } 3581 }; 3582 let w_ty_ok = match widened_ty { 3583 I32 | I64 => true, 3584 _ => false, 3585 }; 3586 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3587 3588 assert!(expected_ty.bytes() >= access_ty.bytes()); 3589 if expected_ty.bytes() > access_ty.bytes() { 3590 expected = builder.ins().ireduce(access_ty, expected); 3591 } 3592 assert!(replacement_ty.bytes() >= access_ty.bytes()); 3593 if replacement_ty.bytes() > access_ty.bytes() { 3594 replacement = builder.ins().ireduce(access_ty, replacement); 3595 } 3596 3597 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3598 state, 3599 prepare_atomic_addr( 3600 memarg, 3601 u8::try_from(access_ty.bytes()).unwrap(), 3602 builder, 3603 state, 3604 environ, 3605 )? 3606 ); 3607 let mut res = builder.ins().atomic_cas(flags, addr, expected, replacement); 3608 if access_ty != widened_ty { 3609 res = builder.ins().uextend(widened_ty, res); 3610 } 3611 state.push1(res); 3612 Ok(()) 3613 } 3614 3615 fn translate_atomic_load( 3616 widened_ty: Type, 3617 access_ty: Type, 3618 memarg: &MemArg, 3619 builder: &mut FunctionBuilder, 3620 state: &mut FuncTranslationState, 3621 environ: &mut FuncEnvironment<'_>, 3622 ) -> WasmResult<()> { 3623 // The load is performed at type `access_ty`, and the loaded value is zero extended 3624 // to `widened_ty`. 3625 match access_ty { 3626 I8 | I16 | I32 | I64 => {} 3627 _ => { 3628 return Err(wasm_unsupported!( 3629 "atomic_load: unsupported access type {:?}", 3630 access_ty 3631 )); 3632 } 3633 }; 3634 let w_ty_ok = match widened_ty { 3635 I32 | I64 => true, 3636 _ => false, 3637 }; 3638 assert!(w_ty_ok && widened_ty.bytes() >= access_ty.bytes()); 3639 3640 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3641 state, 3642 prepare_atomic_addr( 3643 memarg, 3644 u8::try_from(access_ty.bytes()).unwrap(), 3645 builder, 3646 state, 3647 environ, 3648 )? 3649 ); 3650 let mut res = builder.ins().atomic_load(access_ty, flags, addr); 3651 if access_ty != widened_ty { 3652 res = builder.ins().uextend(widened_ty, res); 3653 } 3654 state.push1(res); 3655 Ok(()) 3656 } 3657 3658 fn translate_atomic_store( 3659 access_ty: Type, 3660 memarg: &MemArg, 3661 builder: &mut FunctionBuilder, 3662 state: &mut FuncTranslationState, 3663 environ: &mut FuncEnvironment<'_>, 3664 ) -> WasmResult<()> { 3665 let mut data = state.pop1(); 3666 let data_ty = builder.func.dfg.value_type(data); 3667 3668 // The operation is performed at type `access_ty`, and the data to be stored may first 3669 // need to be narrowed accordingly. 3670 match access_ty { 3671 I8 | I16 | I32 | I64 => {} 3672 _ => { 3673 return Err(wasm_unsupported!( 3674 "atomic_store: unsupported access type {:?}", 3675 access_ty 3676 )); 3677 } 3678 }; 3679 let d_ty_ok = match data_ty { 3680 I32 | I64 => true, 3681 _ => false, 3682 }; 3683 assert!(d_ty_ok && data_ty.bytes() >= access_ty.bytes()); 3684 3685 if data_ty.bytes() > access_ty.bytes() { 3686 data = builder.ins().ireduce(access_ty, data); 3687 } 3688 3689 let (flags, _, addr) = unwrap_or_return_unreachable_state!( 3690 state, 3691 prepare_atomic_addr( 3692 memarg, 3693 u8::try_from(access_ty.bytes()).unwrap(), 3694 builder, 3695 state, 3696 environ, 3697 )? 3698 ); 3699 builder.ins().atomic_store(flags, data, addr); 3700 Ok(()) 3701 } 3702 3703 fn translate_vector_icmp( 3704 cc: IntCC, 3705 needed_type: Type, 3706 builder: &mut FunctionBuilder, 3707 state: &mut FuncTranslationState, 3708 ) { 3709 let (a, b) = state.pop2(); 3710 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 3711 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 3712 state.push1(builder.ins().icmp(cc, bitcast_a, bitcast_b)) 3713 } 3714 3715 fn translate_fcmp(cc: FloatCC, builder: &mut FunctionBuilder, state: &mut FuncTranslationState) { 3716 let (arg0, arg1) = state.pop2(); 3717 let val = builder.ins().fcmp(cc, arg0, arg1); 3718 state.push1(builder.ins().uextend(I32, val)); 3719 } 3720 3721 fn translate_vector_fcmp( 3722 cc: FloatCC, 3723 needed_type: Type, 3724 builder: &mut FunctionBuilder, 3725 state: &mut FuncTranslationState, 3726 ) { 3727 let (a, b) = state.pop2(); 3728 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 3729 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 3730 state.push1(builder.ins().fcmp(cc, bitcast_a, bitcast_b)) 3731 } 3732 3733 fn translate_br_if( 3734 relative_depth: u32, 3735 builder: &mut FunctionBuilder, 3736 state: &mut FuncTranslationState, 3737 ) { 3738 let val = state.pop1(); 3739 let (br_destination, inputs) = translate_br_if_args(relative_depth, state); 3740 let next_block = builder.create_block(); 3741 canonicalise_brif(builder, val, br_destination, inputs, next_block, &[]); 3742 3743 builder.seal_block(next_block); // The only predecessor is the current block. 3744 builder.switch_to_block(next_block); 3745 } 3746 3747 fn translate_br_if_args( 3748 relative_depth: u32, 3749 state: &mut FuncTranslationState, 3750 ) -> (ir::Block, &mut [ir::Value]) { 3751 let i = state.control_stack.len() - 1 - (relative_depth as usize); 3752 let (return_count, br_destination) = { 3753 let frame = &mut state.control_stack[i]; 3754 // The values returned by the branch are still available for the reachable 3755 // code that comes after it 3756 frame.set_branched_to_exit(); 3757 let return_count = if frame.is_loop() { 3758 frame.num_param_values() 3759 } else { 3760 frame.num_return_values() 3761 }; 3762 (return_count, frame.br_destination()) 3763 }; 3764 let inputs = state.peekn_mut(return_count); 3765 (br_destination, inputs) 3766 } 3767 3768 /// Determine the returned value type of a WebAssembly operator 3769 fn type_of(operator: &Operator) -> Type { 3770 match operator { 3771 Operator::V128Load { .. } 3772 | Operator::V128Store { .. } 3773 | Operator::V128Const { .. } 3774 | Operator::V128Not 3775 | Operator::V128And 3776 | Operator::V128AndNot 3777 | Operator::V128Or 3778 | Operator::V128Xor 3779 | Operator::V128AnyTrue 3780 | Operator::V128Bitselect => I8X16, // default type representing V128 3781 3782 Operator::I8x16Shuffle { .. } 3783 | Operator::I8x16Splat 3784 | Operator::V128Load8Splat { .. } 3785 | Operator::V128Load8Lane { .. } 3786 | Operator::V128Store8Lane { .. } 3787 | Operator::I8x16ExtractLaneS { .. } 3788 | Operator::I8x16ExtractLaneU { .. } 3789 | Operator::I8x16ReplaceLane { .. } 3790 | Operator::I8x16Eq 3791 | Operator::I8x16Ne 3792 | Operator::I8x16LtS 3793 | Operator::I8x16LtU 3794 | Operator::I8x16GtS 3795 | Operator::I8x16GtU 3796 | Operator::I8x16LeS 3797 | Operator::I8x16LeU 3798 | Operator::I8x16GeS 3799 | Operator::I8x16GeU 3800 | Operator::I8x16Neg 3801 | Operator::I8x16Abs 3802 | Operator::I8x16AllTrue 3803 | Operator::I8x16Shl 3804 | Operator::I8x16ShrS 3805 | Operator::I8x16ShrU 3806 | Operator::I8x16Add 3807 | Operator::I8x16AddSatS 3808 | Operator::I8x16AddSatU 3809 | Operator::I8x16Sub 3810 | Operator::I8x16SubSatS 3811 | Operator::I8x16SubSatU 3812 | Operator::I8x16MinS 3813 | Operator::I8x16MinU 3814 | Operator::I8x16MaxS 3815 | Operator::I8x16MaxU 3816 | Operator::I8x16AvgrU 3817 | Operator::I8x16Bitmask 3818 | Operator::I8x16Popcnt 3819 | Operator::I8x16RelaxedLaneselect => I8X16, 3820 3821 Operator::I16x8Splat 3822 | Operator::V128Load16Splat { .. } 3823 | Operator::V128Load16Lane { .. } 3824 | Operator::V128Store16Lane { .. } 3825 | Operator::I16x8ExtractLaneS { .. } 3826 | Operator::I16x8ExtractLaneU { .. } 3827 | Operator::I16x8ReplaceLane { .. } 3828 | Operator::I16x8Eq 3829 | Operator::I16x8Ne 3830 | Operator::I16x8LtS 3831 | Operator::I16x8LtU 3832 | Operator::I16x8GtS 3833 | Operator::I16x8GtU 3834 | Operator::I16x8LeS 3835 | Operator::I16x8LeU 3836 | Operator::I16x8GeS 3837 | Operator::I16x8GeU 3838 | Operator::I16x8Neg 3839 | Operator::I16x8Abs 3840 | Operator::I16x8AllTrue 3841 | Operator::I16x8Shl 3842 | Operator::I16x8ShrS 3843 | Operator::I16x8ShrU 3844 | Operator::I16x8Add 3845 | Operator::I16x8AddSatS 3846 | Operator::I16x8AddSatU 3847 | Operator::I16x8Sub 3848 | Operator::I16x8SubSatS 3849 | Operator::I16x8SubSatU 3850 | Operator::I16x8MinS 3851 | Operator::I16x8MinU 3852 | Operator::I16x8MaxS 3853 | Operator::I16x8MaxU 3854 | Operator::I16x8AvgrU 3855 | Operator::I16x8Mul 3856 | Operator::I16x8Bitmask 3857 | Operator::I16x8RelaxedLaneselect => I16X8, 3858 3859 Operator::I32x4Splat 3860 | Operator::V128Load32Splat { .. } 3861 | Operator::V128Load32Lane { .. } 3862 | Operator::V128Store32Lane { .. } 3863 | Operator::I32x4ExtractLane { .. } 3864 | Operator::I32x4ReplaceLane { .. } 3865 | Operator::I32x4Eq 3866 | Operator::I32x4Ne 3867 | Operator::I32x4LtS 3868 | Operator::I32x4LtU 3869 | Operator::I32x4GtS 3870 | Operator::I32x4GtU 3871 | Operator::I32x4LeS 3872 | Operator::I32x4LeU 3873 | Operator::I32x4GeS 3874 | Operator::I32x4GeU 3875 | Operator::I32x4Neg 3876 | Operator::I32x4Abs 3877 | Operator::I32x4AllTrue 3878 | Operator::I32x4Shl 3879 | Operator::I32x4ShrS 3880 | Operator::I32x4ShrU 3881 | Operator::I32x4Add 3882 | Operator::I32x4Sub 3883 | Operator::I32x4Mul 3884 | Operator::I32x4MinS 3885 | Operator::I32x4MinU 3886 | Operator::I32x4MaxS 3887 | Operator::I32x4MaxU 3888 | Operator::I32x4Bitmask 3889 | Operator::I32x4TruncSatF32x4S 3890 | Operator::I32x4TruncSatF32x4U 3891 | Operator::I32x4RelaxedLaneselect 3892 | Operator::V128Load32Zero { .. } => I32X4, 3893 3894 Operator::I64x2Splat 3895 | Operator::V128Load64Splat { .. } 3896 | Operator::V128Load64Lane { .. } 3897 | Operator::V128Store64Lane { .. } 3898 | Operator::I64x2ExtractLane { .. } 3899 | Operator::I64x2ReplaceLane { .. } 3900 | Operator::I64x2Eq 3901 | Operator::I64x2Ne 3902 | Operator::I64x2LtS 3903 | Operator::I64x2GtS 3904 | Operator::I64x2LeS 3905 | Operator::I64x2GeS 3906 | Operator::I64x2Neg 3907 | Operator::I64x2Abs 3908 | Operator::I64x2AllTrue 3909 | Operator::I64x2Shl 3910 | Operator::I64x2ShrS 3911 | Operator::I64x2ShrU 3912 | Operator::I64x2Add 3913 | Operator::I64x2Sub 3914 | Operator::I64x2Mul 3915 | Operator::I64x2Bitmask 3916 | Operator::I64x2RelaxedLaneselect 3917 | Operator::V128Load64Zero { .. } => I64X2, 3918 3919 Operator::F32x4Splat 3920 | Operator::F32x4ExtractLane { .. } 3921 | Operator::F32x4ReplaceLane { .. } 3922 | Operator::F32x4Eq 3923 | Operator::F32x4Ne 3924 | Operator::F32x4Lt 3925 | Operator::F32x4Gt 3926 | Operator::F32x4Le 3927 | Operator::F32x4Ge 3928 | Operator::F32x4Abs 3929 | Operator::F32x4Neg 3930 | Operator::F32x4Sqrt 3931 | Operator::F32x4Add 3932 | Operator::F32x4Sub 3933 | Operator::F32x4Mul 3934 | Operator::F32x4Div 3935 | Operator::F32x4Min 3936 | Operator::F32x4Max 3937 | Operator::F32x4PMin 3938 | Operator::F32x4PMax 3939 | Operator::F32x4ConvertI32x4S 3940 | Operator::F32x4ConvertI32x4U 3941 | Operator::F32x4Ceil 3942 | Operator::F32x4Floor 3943 | Operator::F32x4Trunc 3944 | Operator::F32x4Nearest 3945 | Operator::F32x4RelaxedMax 3946 | Operator::F32x4RelaxedMin 3947 | Operator::F32x4RelaxedMadd 3948 | Operator::F32x4RelaxedNmadd => F32X4, 3949 3950 Operator::F64x2Splat 3951 | Operator::F64x2ExtractLane { .. } 3952 | Operator::F64x2ReplaceLane { .. } 3953 | Operator::F64x2Eq 3954 | Operator::F64x2Ne 3955 | Operator::F64x2Lt 3956 | Operator::F64x2Gt 3957 | Operator::F64x2Le 3958 | Operator::F64x2Ge 3959 | Operator::F64x2Abs 3960 | Operator::F64x2Neg 3961 | Operator::F64x2Sqrt 3962 | Operator::F64x2Add 3963 | Operator::F64x2Sub 3964 | Operator::F64x2Mul 3965 | Operator::F64x2Div 3966 | Operator::F64x2Min 3967 | Operator::F64x2Max 3968 | Operator::F64x2PMin 3969 | Operator::F64x2PMax 3970 | Operator::F64x2Ceil 3971 | Operator::F64x2Floor 3972 | Operator::F64x2Trunc 3973 | Operator::F64x2Nearest 3974 | Operator::F64x2RelaxedMax 3975 | Operator::F64x2RelaxedMin 3976 | Operator::F64x2RelaxedMadd 3977 | Operator::F64x2RelaxedNmadd => F64X2, 3978 3979 _ => unimplemented!( 3980 "Currently only SIMD instructions are mapped to their return type; the \ 3981 following instruction is not mapped: {:?}", 3982 operator 3983 ), 3984 } 3985 } 3986 3987 /// Some SIMD operations only operate on I8X16 in CLIF; this will convert them to that type by 3988 /// adding a bitcast if necessary. 3989 fn optionally_bitcast_vector( 3990 value: Value, 3991 needed_type: Type, 3992 builder: &mut FunctionBuilder, 3993 ) -> Value { 3994 if builder.func.dfg.value_type(value) != needed_type { 3995 let mut flags = MemFlags::new(); 3996 flags.set_endianness(ir::Endianness::Little); 3997 builder.ins().bitcast(needed_type, flags, value) 3998 } else { 3999 value 4000 } 4001 } 4002 4003 #[inline(always)] 4004 fn is_non_canonical_v128(ty: ir::Type) -> bool { 4005 match ty { 4006 I64X2 | I32X4 | I16X8 | F32X4 | F64X2 => true, 4007 _ => false, 4008 } 4009 } 4010 4011 /// Cast to I8X16, any vector values in `values` that are of "non-canonical" type (meaning, not 4012 /// I8X16), and return them in a slice. A pre-scan is made to determine whether any casts are 4013 /// actually necessary, and if not, the original slice is returned. Otherwise the cast values 4014 /// are returned in a slice that belongs to the caller-supplied `SmallVec`. 4015 fn canonicalise_v128_values<'a>( 4016 tmp_canonicalised: &'a mut SmallVec<[BlockArg; 16]>, 4017 builder: &mut FunctionBuilder, 4018 values: &'a [ir::Value], 4019 ) -> &'a [BlockArg] { 4020 debug_assert!(tmp_canonicalised.is_empty()); 4021 // Cast, and push the resulting `Value`s into `canonicalised`. 4022 for v in values { 4023 let value = if is_non_canonical_v128(builder.func.dfg.value_type(*v)) { 4024 let mut flags = MemFlags::new(); 4025 flags.set_endianness(ir::Endianness::Little); 4026 builder.ins().bitcast(I8X16, flags, *v) 4027 } else { 4028 *v 4029 }; 4030 tmp_canonicalised.push(BlockArg::from(value)); 4031 } 4032 tmp_canonicalised.as_slice() 4033 } 4034 4035 /// Generate a `jump` instruction, but first cast all 128-bit vector values to I8X16 if they 4036 /// don't have that type. This is done in somewhat roundabout way so as to ensure that we 4037 /// almost never have to do any heap allocation. 4038 fn canonicalise_then_jump( 4039 builder: &mut FunctionBuilder, 4040 destination: ir::Block, 4041 params: &[ir::Value], 4042 ) -> ir::Inst { 4043 let mut tmp_canonicalised = SmallVec::<[_; 16]>::new(); 4044 let canonicalised = canonicalise_v128_values(&mut tmp_canonicalised, builder, params); 4045 builder.ins().jump(destination, canonicalised) 4046 } 4047 4048 /// The same but for a `brif` instruction. 4049 fn canonicalise_brif( 4050 builder: &mut FunctionBuilder, 4051 cond: ir::Value, 4052 block_then: ir::Block, 4053 params_then: &[ir::Value], 4054 block_else: ir::Block, 4055 params_else: &[ir::Value], 4056 ) -> ir::Inst { 4057 let mut tmp_canonicalised_then = SmallVec::<[_; 16]>::new(); 4058 let canonicalised_then = 4059 canonicalise_v128_values(&mut tmp_canonicalised_then, builder, params_then); 4060 let mut tmp_canonicalised_else = SmallVec::<[_; 16]>::new(); 4061 let canonicalised_else = 4062 canonicalise_v128_values(&mut tmp_canonicalised_else, builder, params_else); 4063 builder.ins().brif( 4064 cond, 4065 block_then, 4066 canonicalised_then, 4067 block_else, 4068 canonicalised_else, 4069 ) 4070 } 4071 4072 /// A helper for popping and bitcasting a single value; since SIMD values can lose their type by 4073 /// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF 4074 /// typing issues. 4075 fn pop1_with_bitcast( 4076 state: &mut FuncTranslationState, 4077 needed_type: Type, 4078 builder: &mut FunctionBuilder, 4079 ) -> Value { 4080 optionally_bitcast_vector(state.pop1(), needed_type, builder) 4081 } 4082 4083 /// A helper for popping and bitcasting two values; since SIMD values can lose their type by 4084 /// using v128 (i.e. CLIF's I8x16) we must re-type the values using a bitcast to avoid CLIF 4085 /// typing issues. 4086 fn pop2_with_bitcast( 4087 state: &mut FuncTranslationState, 4088 needed_type: Type, 4089 builder: &mut FunctionBuilder, 4090 ) -> (Value, Value) { 4091 let (a, b) = state.pop2(); 4092 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 4093 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 4094 (bitcast_a, bitcast_b) 4095 } 4096 4097 fn pop3_with_bitcast( 4098 state: &mut FuncTranslationState, 4099 needed_type: Type, 4100 builder: &mut FunctionBuilder, 4101 ) -> (Value, Value, Value) { 4102 let (a, b, c) = state.pop3(); 4103 let bitcast_a = optionally_bitcast_vector(a, needed_type, builder); 4104 let bitcast_b = optionally_bitcast_vector(b, needed_type, builder); 4105 let bitcast_c = optionally_bitcast_vector(c, needed_type, builder); 4106 (bitcast_a, bitcast_b, bitcast_c) 4107 } 4108 4109 fn bitcast_arguments<'a>( 4110 builder: &FunctionBuilder, 4111 arguments: &'a mut [Value], 4112 params: &[ir::AbiParam], 4113 param_predicate: impl Fn(usize) -> bool, 4114 ) -> Vec<(Type, &'a mut Value)> { 4115 let filtered_param_types = params 4116 .iter() 4117 .enumerate() 4118 .filter(|(i, _)| param_predicate(*i)) 4119 .map(|(_, param)| param.value_type); 4120 4121 // zip_eq, from the itertools::Itertools trait, is like Iterator::zip but panics if one 4122 // iterator ends before the other. The `param_predicate` is required to select exactly as many 4123 // elements of `params` as there are elements in `arguments`. 4124 let pairs = filtered_param_types.zip_eq(arguments.iter_mut()); 4125 4126 // The arguments which need to be bitcasted are those which have some vector type but the type 4127 // expected by the parameter is not the same vector type as that of the provided argument. 4128 pairs 4129 .filter(|(param_type, _)| param_type.is_vector()) 4130 .filter(|(param_type, arg)| { 4131 let arg_type = builder.func.dfg.value_type(**arg); 4132 assert!( 4133 arg_type.is_vector(), 4134 "unexpected type mismatch: expected {}, argument {} was actually of type {}", 4135 param_type, 4136 *arg, 4137 arg_type 4138 ); 4139 4140 // This is the same check that would be done by `optionally_bitcast_vector`, except we 4141 // can't take a mutable borrow of the FunctionBuilder here, so we defer inserting the 4142 // bitcast instruction to the caller. 4143 arg_type != *param_type 4144 }) 4145 .collect() 4146 } 4147 4148 /// A helper for bitcasting a sequence of return values for the function currently being built. If 4149 /// a value is a vector type that does not match its expected type, this will modify the value in 4150 /// place to point to the result of a `bitcast`. This conversion is necessary to translate Wasm 4151 /// code that uses `V128` as function parameters (or implicitly in block parameters) and still use 4152 /// specific CLIF types (e.g. `I32X4`) in the function body. 4153 pub fn bitcast_wasm_returns(arguments: &mut [Value], builder: &mut FunctionBuilder) { 4154 let changes = bitcast_arguments(builder, arguments, &builder.func.signature.returns, |i| { 4155 builder.func.signature.returns[i].purpose == ir::ArgumentPurpose::Normal 4156 }); 4157 for (t, arg) in changes { 4158 let mut flags = MemFlags::new(); 4159 flags.set_endianness(ir::Endianness::Little); 4160 *arg = builder.ins().bitcast(t, flags, *arg); 4161 } 4162 } 4163 4164 /// Like `bitcast_wasm_returns`, but for the parameters being passed to a specified callee. 4165 fn bitcast_wasm_params( 4166 environ: &mut FuncEnvironment<'_>, 4167 callee_signature: ir::SigRef, 4168 arguments: &mut [Value], 4169 builder: &mut FunctionBuilder, 4170 ) { 4171 let callee_signature = &builder.func.dfg.signatures[callee_signature]; 4172 let changes = bitcast_arguments(builder, arguments, &callee_signature.params, |i| { 4173 environ.is_wasm_parameter(&callee_signature, i) 4174 }); 4175 for (t, arg) in changes { 4176 let mut flags = MemFlags::new(); 4177 flags.set_endianness(ir::Endianness::Little); 4178 *arg = builder.ins().bitcast(t, flags, *arg); 4179 } 4180 } 4181