1# 2005 November 30 2# 3# The author disclaims copyright to this source code. In place of 4# a legal notice, here is a blessing: 5# 6# May you do good and not evil. 7# May you find forgiveness for yourself and forgive others. 8# May you share freely, never taking more than you give. 9# 10#*********************************************************************** 11# 12# This file contains tests to ensure that the library handles malloc() failures 13# correctly. The emphasis of these tests are the _prepare(), _step() and 14# _finalize() calls. 15# 16# $Id: malloc3.test,v 1.24 2008/10/14 15:54:08 drh Exp $ 17 18set testdir [file dirname $argv0] 19source $testdir/tester.tcl 20source $testdir/malloc_common.tcl 21 22# Only run these tests if memory debugging is turned on. 23# 24if {!$MEMDEBUG} { 25 puts "Skipping malloc3 tests: not compiled with -DSQLITE_MEMDEBUG..." 26 finish_test 27 return 28} 29 30#-------------------------------------------------------------------------- 31# NOTES ON RECOVERING FROM A MALLOC FAILURE 32# 33# The tests in this file test the behaviours described in the following 34# paragraphs. These tests test the behaviour of the system when malloc() fails 35# inside of a call to _prepare(), _step(), _finalize() or _reset(). The 36# handling of malloc() failures within ancillary procedures is tested 37# elsewhere. 38# 39# Overview: 40# 41# Executing a statement is done in three stages (prepare, step and finalize). A 42# malloc() failure may occur within any stage. If a memory allocation fails 43# during statement preparation, no statement handle is returned. From the users 44# point of view the system state is as if _prepare() had never been called. 45# 46# If the memory allocation fails during the _step() or _finalize() calls, then 47# the database may be left in one of two states (after finalize() has been 48# called): 49# 50# * As if the neither _step() nor _finalize() had ever been called on 51# the statement handle (i.e. any changes made by the statement are 52# rolled back). 53# * The current transaction may be rolled back. In this case a hot-journal 54# may or may not actually be present in the filesystem. 55# 56# The caller can tell the difference between these two scenarios by invoking 57# _get_autocommit(). 58# 59# 60# Handling of sqlite3_reset(): 61# 62# If a malloc() fails while executing an sqlite3_reset() call, this is handled 63# in the same way as a failure within _finalize(). The statement handle 64# is not deleted and must be passed to _finalize() for resource deallocation. 65# Attempting to _step() or _reset() the statement after a failed _reset() will 66# always return SQLITE_NOMEM. 67# 68# 69# Other active SQL statements: 70# 71# The effect of a malloc failure on concurrently executing SQL statements, 72# particularly when the statement is executing with READ_UNCOMMITTED set and 73# the malloc() failure mandates statement rollback only. Currently, if 74# transaction rollback is required, all other vdbe's are aborted. 75# 76# Non-transient mallocs in btree.c: 77# * The Btree structure itself 78# * Each BtCursor structure 79# 80# Mallocs in pager.c: 81# readMasterJournal() - Space to read the master journal name 82# pager_delmaster() - Space for the entire master journal file 83# 84# sqlite3pager_open() - The pager structure itself 85# sqlite3_pagerget() - Space for a new page 86# pager_open_journal() - Pager.aInJournal[] bitmap 87# sqlite3pager_write() - For in-memory databases only: history page and 88# statement history page. 89# pager_stmt_begin() - Pager.aInStmt[] bitmap 90# 91# None of the above are a huge problem. The most troublesome failures are the 92# transient malloc() calls in btree.c, which can occur during the tree-balance 93# operation. This means the tree being balanced will be internally inconsistent 94# after the malloc() fails. To avoid the corrupt tree being read by a 95# READ_UNCOMMITTED query, we have to make sure the transaction or statement 96# rollback occurs before sqlite3_step() returns, not during a subsequent 97# sqlite3_finalize(). 98#-------------------------------------------------------------------------- 99 100#-------------------------------------------------------------------------- 101# NOTES ON TEST IMPLEMENTATION 102# 103# The tests in this file are implemented differently from those in other 104# files. Instead, tests are specified using three primitives: SQL, PREP and 105# TEST. Each primitive has a single argument. Primitives are processed in 106# the order they are specified in the file. 107# 108# A TEST primitive specifies a TCL script as its argument. When a TEST 109# directive is encountered the Tcl script is evaluated. Usually, this Tcl 110# script contains one or more calls to [do_test]. 111# 112# A PREP primitive specifies an SQL script as its argument. When a PREP 113# directive is encountered the SQL is evaluated using database connection 114# [db]. 115# 116# The SQL primitives are where the action happens. An SQL primitive must 117# contain a single, valid SQL statement as its argument. When an SQL 118# primitive is encountered, it is evaluated one or more times to test the 119# behaviour of the system when malloc() fails during preparation or 120# execution of said statement. The Nth time the statement is executed, 121# the Nth malloc is said to fail. The statement is executed until it 122# succeeds, i.e. (M+1) times, where M is the number of mallocs() required 123# to prepare and execute the statement. 124# 125# Each time an SQL statement fails, the driver program (see proc [run_test] 126# below) figures out if a transaction has been automatically rolled back. 127# If not, it executes any TEST block immediately proceeding the SQL 128# statement, then reexecutes the SQL statement with the next value of N. 129# 130# If a transaction has been automatically rolled back, then the driver 131# program executes all the SQL specified as part of SQL or PREP primitives 132# between the current SQL statement and the most recent "BEGIN". Any 133# TEST block immediately proceeding the SQL statement is evaluated, and 134# then the SQL statement reexecuted with the incremented N value. 135# 136# That make any sense? If not, read the code in [run_test] and it might. 137# 138# Extra restriction imposed by the implementation: 139# 140# * If a PREP block starts a transaction, it must finish it. 141# * A PREP block may not close a transaction it did not start. 142# 143#-------------------------------------------------------------------------- 144 145 146# These procs are used to build up a "program" in global variable 147# ::run_test_script. At the end of this file, the proc [run_test] is used 148# to execute the program (and all test cases contained therein). 149# 150set ::run_test_sql_id 0 151set ::run_test_script [list] 152proc TEST {id t} {lappend ::run_test_script -test [list $id $t]} 153proc PREP {p} {lappend ::run_test_script -prep [string trim $p]} 154proc DEBUG {s} {lappend ::run_test_script -debug $s} 155 156# SQL -- 157# 158# SQL ?-norollback? <sql-text> 159# 160# Add an 'SQL' primitive to the program (see notes above). If the -norollback 161# switch is present, then the statement is not allowed to automatically roll 162# back any active transaction if malloc() fails. It must rollback the statement 163# transaction only. 164# 165proc SQL {a1 {a2 ""}} { 166 # An SQL primitive parameter is a list of three elements, an id, a boolean 167 # value indicating if the statement may cause transaction rollback when 168 # malloc() fails, and the sql statement itself. 169 set id [incr ::run_test_sql_id] 170 if {$a2 == ""} { 171 lappend ::run_test_script -sql [list $id true [string trim $a1]] 172 } else { 173 lappend ::run_test_script -sql [list $id false [string trim $a2]] 174 } 175} 176 177# TEST_AUTOCOMMIT -- 178# 179# A shorthand test to see if a transaction is active or not. The first 180# argument - $id - is the integer number of the test case. The second 181# argument is either 1 or 0, the expected value of the auto-commit flag. 182# 183proc TEST_AUTOCOMMIT {id a} { 184 TEST $id "do_test \$testid { sqlite3_get_autocommit \$::DB } {$a}" 185} 186 187#-------------------------------------------------------------------------- 188# Start of test program declaration 189# 190 191 192# Warm body test. A malloc() fails in the middle of a CREATE TABLE statement 193# in a single-statement transaction on an empty database. Not too much can go 194# wrong here. 195# 196TEST 1 { 197 do_test $testid { 198 execsql {SELECT tbl_name FROM sqlite_master;} 199 } {} 200} 201SQL { 202 CREATE TABLE IF NOT EXISTS abc(a, b, c); 203} 204TEST 2 { 205 do_test $testid.1 { 206 execsql {SELECT tbl_name FROM sqlite_master;} 207 } {abc} 208} 209 210# Insert a couple of rows into the table. each insert is in its own 211# transaction. test that the table is unpopulated before running the inserts 212# (and hence after each failure of the first insert), and that it has been 213# populated correctly after the final insert succeeds. 214# 215TEST 3 { 216 do_test $testid.2 { 217 execsql {SELECT * FROM abc} 218 } {} 219} 220SQL {INSERT INTO abc VALUES(1, 2, 3);} 221SQL {INSERT INTO abc VALUES(4, 5, 6);} 222SQL {INSERT INTO abc VALUES(7, 8, 9);} 223TEST 4 { 224 do_test $testid { 225 execsql {SELECT * FROM abc} 226 } {1 2 3 4 5 6 7 8 9} 227} 228 229# Test a CREATE INDEX statement. Because the table 'abc' is so small, the index 230# will all fit on a single page, so this doesn't test too much that the CREATE 231# TABLE statement didn't test. A few of the transient malloc()s in btree.c 232# perhaps. 233# 234SQL {CREATE INDEX abc_i ON abc(a, b, c);} 235TEST 4 { 236 do_test $testid { 237 execsql { 238 SELECT * FROM abc ORDER BY a DESC; 239 } 240 } {7 8 9 4 5 6 1 2 3} 241} 242 243# Test a DELETE statement. Also create a trigger and a view, just to make sure 244# these statements don't have any obvious malloc() related bugs in them. Note 245# that the test above will be executed each time the DELETE fails, so we're 246# also testing rollback of a DELETE from a table with an index on it. 247# 248SQL {DELETE FROM abc WHERE a > 2;} 249SQL {CREATE TRIGGER abc_t AFTER INSERT ON abc BEGIN SELECT 'trigger!'; END;} 250SQL {CREATE VIEW abc_v AS SELECT * FROM abc;} 251TEST 5 { 252 do_test $testid { 253 execsql { 254 SELECT name, tbl_name FROM sqlite_master ORDER BY name; 255 SELECT * FROM abc; 256 } 257 } {abc abc abc_i abc abc_t abc abc_v abc_v 1 2 3} 258} 259 260set sql { 261 BEGIN;DELETE FROM abc; 262} 263for {set i 1} {$i < 100} {incr i} { 264 set a $i 265 set b "String value $i" 266 set c [string repeat X $i] 267 append sql "INSERT INTO abc VALUES ($a, '$b', '$c');" 268} 269append sql {COMMIT;} 270PREP $sql 271 272SQL { 273 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5); 274} 275TEST 6 { 276 do_test $testid.1 { 277 execsql {SELECT count(*) FROM abc} 278 } {94} 279 do_test $testid.2 { 280 execsql { 281 SELECT min( 282 (oid == a) AND 'String value ' || a == b AND a == length(c) 283 ) FROM abc; 284 } 285 } {1} 286} 287SQL { 288 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5); 289} 290TEST 7 { 291 do_test $testid { 292 execsql {SELECT count(*) FROM abc} 293 } {89} 294 do_test $testid { 295 execsql { 296 SELECT min( 297 (oid == a) AND 'String value ' || a == b AND a == length(c) 298 ) FROM abc; 299 } 300 } {1} 301} 302SQL { 303 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5); 304} 305TEST 9 { 306 do_test $testid { 307 execsql {SELECT count(*) FROM abc} 308 } {84} 309 do_test $testid { 310 execsql { 311 SELECT min( 312 (oid == a) AND 'String value ' || a == b AND a == length(c) 313 ) FROM abc; 314 } 315 } {1} 316} 317 318set padding [string repeat X 500] 319PREP [subst { 320 DROP TABLE abc; 321 CREATE TABLE abc(a PRIMARY KEY, padding, b, c); 322 INSERT INTO abc VALUES(0, '$padding', 2, 2); 323 INSERT INTO abc VALUES(3, '$padding', 5, 5); 324 INSERT INTO abc VALUES(6, '$padding', 8, 8); 325}] 326 327TEST 10 { 328 do_test $testid { 329 execsql {SELECT a, b, c FROM abc} 330 } {0 2 2 3 5 5 6 8 8} 331} 332 333SQL {BEGIN;} 334SQL {INSERT INTO abc VALUES(9, 'XXXXX', 11, 12);} 335TEST_AUTOCOMMIT 11 0 336SQL -norollback {UPDATE abc SET a = a + 1, c = c + 1;} 337TEST_AUTOCOMMIT 12 0 338SQL {DELETE FROM abc WHERE a = 10;} 339TEST_AUTOCOMMIT 13 0 340SQL {COMMIT;} 341 342TEST 14 { 343 do_test $testid.1 { 344 sqlite3_get_autocommit $::DB 345 } {1} 346 do_test $testid.2 { 347 execsql {SELECT a, b, c FROM abc} 348 } {1 2 3 4 5 6 7 8 9} 349} 350 351PREP [subst { 352 DROP TABLE abc; 353 CREATE TABLE abc(a, padding, b, c); 354 INSERT INTO abc VALUES(1, '$padding', 2, 3); 355 INSERT INTO abc VALUES(4, '$padding', 5, 6); 356 INSERT INTO abc VALUES(7, '$padding', 8, 9); 357 CREATE INDEX abc_i ON abc(a, padding, b, c); 358}] 359 360TEST 15 { 361 db eval {PRAGMA cache_size = 10} 362} 363 364SQL {BEGIN;} 365SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc} 366TEST 16 { 367 do_test $testid { 368 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 369 } {1 2 4 2 7 2} 370} 371SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc} 372TEST 17 { 373 do_test $testid { 374 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 375 } {1 4 4 4 7 4} 376} 377SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc} 378TEST 18 { 379 do_test $testid { 380 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 381 } {1 8 4 8 7 8} 382} 383SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc} 384TEST 19 { 385 do_test $testid { 386 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 387 } {1 16 4 16 7 16} 388} 389SQL {COMMIT;} 390TEST 21 { 391 do_test $testid { 392 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 393 } {1 16 4 16 7 16} 394} 395 396SQL {BEGIN;} 397SQL {DELETE FROM abc WHERE oid %2} 398TEST 22 { 399 do_test $testid { 400 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 401 } {1 8 4 8 7 8} 402} 403SQL {DELETE FROM abc} 404TEST 23 { 405 do_test $testid { 406 execsql {SELECT * FROM abc} 407 } {} 408} 409SQL {ROLLBACK;} 410TEST 24 { 411 do_test $testid { 412 execsql {SELECT a, count(*) FROM abc GROUP BY a;} 413 } {1 16 4 16 7 16} 414} 415 416# Test some schema modifications inside of a transaction. These should all 417# cause transaction rollback if they fail. Also query a view, to cover a bit 418# more code. 419# 420PREP {DROP VIEW abc_v;} 421TEST 25 { 422 do_test $testid { 423 execsql { 424 SELECT name, tbl_name FROM sqlite_master; 425 } 426 } {abc abc abc_i abc} 427} 428SQL {BEGIN;} 429SQL {CREATE TABLE def(d, e, f);} 430SQL {CREATE TABLE ghi(g, h, i);} 431TEST 26 { 432 do_test $testid { 433 execsql { 434 SELECT name, tbl_name FROM sqlite_master; 435 } 436 } {abc abc abc_i abc def def ghi ghi} 437} 438SQL {CREATE VIEW v1 AS SELECT * FROM def, ghi} 439SQL {CREATE UNIQUE INDEX ghi_i1 ON ghi(g);} 440TEST 27 { 441 do_test $testid { 442 execsql { 443 SELECT name, tbl_name FROM sqlite_master; 444 } 445 } {abc abc abc_i abc def def ghi ghi v1 v1 ghi_i1 ghi} 446} 447SQL {INSERT INTO def VALUES('a', 'b', 'c')} 448SQL {INSERT INTO def VALUES(1, 2, 3)} 449SQL -norollback {INSERT INTO ghi SELECT * FROM def} 450TEST 28 { 451 do_test $testid { 452 execsql { 453 SELECT * FROM def, ghi WHERE d = g; 454 } 455 } {a b c a b c 1 2 3 1 2 3} 456} 457SQL {COMMIT} 458TEST 29 { 459 do_test $testid { 460 execsql { 461 SELECT * FROM v1 WHERE d = g; 462 } 463 } {a b c a b c 1 2 3 1 2 3} 464} 465 466# Test a simple multi-file transaction 467# 468forcedelete test2.db 469ifcapable attach { 470 SQL {ATTACH 'test2.db' AS aux;} 471 SQL {BEGIN} 472 SQL {CREATE TABLE aux.tbl2(x, y, z)} 473 SQL {INSERT INTO tbl2 VALUES(1, 2, 3)} 474 SQL {INSERT INTO def VALUES(4, 5, 6)} 475 TEST 30 { 476 do_test $testid { 477 execsql { 478 SELECT * FROM tbl2, def WHERE d = x; 479 } 480 } {1 2 3 1 2 3} 481 } 482 SQL {COMMIT} 483 TEST 31 { 484 do_test $testid { 485 execsql { 486 SELECT * FROM tbl2, def WHERE d = x; 487 } 488 } {1 2 3 1 2 3} 489 } 490} 491 492# Test what happens when a malloc() fails while there are other active 493# statements. This changes the way sqlite3VdbeHalt() works. 494TEST 32 { 495 if {![info exists ::STMT32]} { 496 set sql "SELECT name FROM sqlite_master" 497 set ::STMT32 [sqlite3_prepare $::DB $sql -1 DUMMY] 498 do_test $testid { 499 sqlite3_step $::STMT32 500 } {SQLITE_ROW} 501 } 502} 503SQL BEGIN 504TEST 33 { 505 do_test $testid { 506 execsql {SELECT * FROM ghi} 507 } {a b c 1 2 3} 508} 509SQL -norollback { 510 -- There is a unique index on ghi(g), so this statement may not cause 511 -- an automatic ROLLBACK. Hence the "-norollback" switch. 512 INSERT INTO ghi SELECT '2'||g, h, i FROM ghi; 513} 514TEST 34 { 515 if {[info exists ::STMT32]} { 516 do_test $testid { 517 sqlite3_finalize $::STMT32 518 } {SQLITE_OK} 519 unset ::STMT32 520 } 521} 522SQL COMMIT 523 524# 525# End of test program declaration 526#-------------------------------------------------------------------------- 527 528proc run_test {arglist iRepeat {pcstart 0} {iFailStart 1}} { 529 if {[llength $arglist] %2} { 530 error "Uneven number of arguments to TEST" 531 } 532 533 for {set i 0} {$i < $pcstart} {incr i} { 534 set k2 [lindex $arglist [expr {2 * $i}]] 535 set v2 [lindex $arglist [expr {2 * $i + 1}]] 536 set ac [sqlite3_get_autocommit $::DB] ;# Auto-Commit 537 switch -- $k2 { 538 -sql {db eval [lindex $v2 2]} 539 -prep {db eval $v2} 540 -debug {eval $v2} 541 } 542 set nac [sqlite3_get_autocommit $::DB] ;# New Auto-Commit 543 if {$ac && !$nac} {set begin_pc $i} 544 } 545 546 db rollback_hook [list incr ::rollback_hook_count] 547 548 set iFail $iFailStart 549 set pc $pcstart 550 while {$pc*2 < [llength $arglist]} { 551 # Fetch the current instruction type and payload. 552 set k [lindex $arglist [expr {2 * $pc}]] 553 set v [lindex $arglist [expr {2 * $pc + 1}]] 554 555 # Id of this iteration: 556 set iterid "pc=$pc.iFail=$iFail$k" 557 558 switch -- $k { 559 560 -test { 561 foreach {id script} $v {} 562 set testid "malloc3-(test $id).$iterid" 563 eval $script 564 incr pc 565 } 566 567 -sql { 568 set ::rollback_hook_count 0 569 570 set id [lindex $v 0] 571 set testid "malloc3-(integrity $id).$iterid" 572 573 set ac [sqlite3_get_autocommit $::DB] ;# Auto-Commit 574 sqlite3_memdebug_fail $iFail -repeat 0 575 set rc [catch {db eval [lindex $v 2]} msg] ;# True error occurs 576 set nac [sqlite3_get_autocommit $::DB] ;# New Auto-Commit 577 578 if {$rc != 0 && $nac && !$ac} { 579 # Before [db eval] the auto-commit flag was clear. Now it 580 # is set. Since an error occured we assume this was not a 581 # commit - therefore a rollback occured. Check that the 582 # rollback-hook was invoked. 583 do_test malloc3-rollback_hook_count.$iterid { 584 set ::rollback_hook_count 585 } {1} 586 } 587 588 set nFail [sqlite3_memdebug_fail -1 -benigncnt nBenign] 589 if {$rc == 0} { 590 # Successful execution of sql. The number of failed malloc() 591 # calls should be equal to the number of benign failures. 592 # Otherwise a malloc() failed and the error was not reported. 593 # 594 set expr {$nFail!=$nBenign} 595 if {[expr $expr]} { 596 error "Unreported malloc() failure, test \"$testid\", $expr" 597 } 598 599 if {$ac && !$nac} { 600 # Before the [db eval] the auto-commit flag was set, now it 601 # is clear. We can deduce that a "BEGIN" statement has just 602 # been successfully executed. 603 set begin_pc $pc 604 } 605 606 incr pc 607 set iFail 1 608 integrity_check $testid 609 } elseif {[regexp {.*out of memory} $msg] || [db errorcode] == 3082} { 610 # Out of memory error, as expected. 611 # 612 integrity_check $testid 613 incr iFail 614 if {$nac && !$ac} { 615 if {![lindex $v 1] && [db errorcode] != 3082} { 616 # error "Statement \"[lindex $v 2]\" caused a rollback" 617 } 618 619 for {set i $begin_pc} {$i < $pc} {incr i} { 620 set k2 [lindex $arglist [expr {2 * $i}]] 621 set v2 [lindex $arglist [expr {2 * $i + 1}]] 622 set catchupsql "" 623 switch -- $k2 { 624 -sql {set catchupsql [lindex $v2 2]} 625 -prep {set catchupsql $v2} 626 } 627 db eval $catchupsql 628 } 629 } 630 } else { 631 error $msg 632 } 633 634 # back up to the previous "-test" block. 635 while {[lindex $arglist [expr {2 * ($pc - 1)}]] == "-test"} { 636 incr pc -1 637 } 638 } 639 640 -prep { 641 db eval $v 642 incr pc 643 } 644 645 -debug { 646 eval $v 647 incr pc 648 } 649 650 default { error "Unknown switch: $k" } 651 } 652 } 653} 654 655# Turn off the Tcl interface's prepared statement caching facility. Then 656# run the tests with "persistent" malloc failures. 657sqlite3_extended_result_codes db 1 658db cache size 0 659run_test $::run_test_script 1 660 661# Close and reopen the db. 662db close 663forcedelete test.db test.db-journal test2.db test2.db-journal 664sqlite3 db test.db 665sqlite3_extended_result_codes db 1 666set ::DB [sqlite3_connection_pointer db] 667 668# Turn off the Tcl interface's prepared statement caching facility in 669# the new connnection. Then run the tests with "transient" malloc failures. 670db cache size 0 671run_test $::run_test_script 0 672 673sqlite3_memdebug_fail -1 674finish_test 675