| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- The `insert_fixtures` method can be optimized by making a single multi statement query for all fixtures having the same connection instead of doing a single query per table
- The previous code was bulk inserting fixtures for a single table, making X query for X fixture files
- This patch builds a single **multi statement query** for every tables. Given a set of 3 fixtures (authors, dogs, computers):
```ruby
# before
%w(authors dogs computers).each do |table|
sql = build_sql(table)
connection.query(sql)
end
# after
sql = build_sql(authors, dogs, computers)
connection.query(sql)
```
- `insert_fixtures` is now deprecated, `insert_fixtures_set` is the new way to go with performance improvement
- My tests were done with an app having more than 700 fixtures, the time it takes to insert all of them was around 15s. Using a single multi statement query, it took on average of 8 seconds
- In order for a multi statement to be executed, mysql needs to be connected with the `MULTI_STATEMENTS` [flag](https://dev.mysql.com/doc/refman/5.7/en/c-api-multiple-queries.html), which is done before inserting the fixtures by reconnecting to da the database with the flag declared. Reconnecting to the database creates some caveats:
1. We loose all open transactions; Inside the original code, when inserting fixtures, a transaction is open. Multple delete statements are [executed](https://github.com/rails/rails/blob/a681eaf22955734c142609961a6d71746cfa0583/activerecord/lib/active_record/fixtures.rb#L566) and finally the fixtures are inserted. The problem with this patch is that we need to open the transaction only after we reconnect to the DB otherwise reconnecting drops the open transaction which doesn't commit all delete statements and inserting fixtures doesn't work since we duplicated them (Primary key duplicate exception)...
- In order to fix this problem, the transaction is now open directly inside the `insert_fixtures` method, right after we reconnect to the db
- As an effect, since the transaction is open inside the `insert_fixtures` method, the DELETE statements need to be executed here since the transaction is open later
2. The same problem happens for the `disable_referential_integrity` since we reconnect, the `FOREIGN_KEY_CHECKS` is reset to the original value
- Same solution as 1. , the disable_referential_integrity can be called after we reconnect to the transaction
3. When the multi statement query is executed, no other queries can be performed until we paginate over the set of results, otherwise mysql throws a "Commands out of sync" [Ref](https://dev.mysql.com/doc/refman/5.7/en/commands-out-of-sync.html)
- Iterating over the set of results until `mysql_client.next_result` is false. [Ref](https://github.com/brianmario/mysql2#multiple-result-sets)
- Removed the `active_record.sql "Fixture delete"` notification, the delete statements are now inside the INSERT's one
- On mysql the `max_allowed_packet` is looked up:
1. Before executing the multi-statements query, we check the packet length of each statements, if the packet is bigger than the max_allowed_packet config, an `ActiveRecordError` is raised
2. Otherwise we concatenate the current sql statement into the previous and so on until the packet is `< max_allowed_packet`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`connection.primary_key` also return composite primary keys, so
`from_primary_key_column` may not be found even if `from_primary_key` is
presented.
```
% ARCONN=sqlite3 be ruby -w -Itest
test/cases/adapters/sqlite3/sqlite3_adapter_test.rb -n
test_copy_table_with_composite_primary_keys
Using sqlite3
Run options: -n test_copy_table_with_composite_primary_keys --seed 19041
# Running:
E
Error:
ActiveRecord::ConnectionAdapters::SQLite3AdapterTest#test_copy_table_with_composite_primary_keys:
NoMethodError: undefined method `type' for nil:NilClass
/path/to/rails/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb:411:in
`block in copy_table'
```
This change fixes `copy_table` to do not lose composite primary keys.
|
|\
| |
| | |
Fix sqlite migrations with custom primary keys
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously, if a record was created with a custom primary key, that
table could not be migrated using sqlite. While attempting to copy the
table, the type of the primary key was ignored.
Once that was corrected, copying the indexes would fail because custom
primary keys are autoindexed by sqlite by default.
To correct that, this skips copying the index if the index name begins
with "sqlite_". This is a reserved word that indicates that the
index is an internal schema object. SQLite prohibits applications from
creating objects whose names begin with "sqlite_", so this string should
be safe to use as a check.
ref https://www.sqlite.org/fileformat2.html#intschema
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is a regression since Rails 4.2.
SQLite3 integer is stored in 1, 2, 3, 4, 6, or 8 bytes depending on the
magnitude of the value. Assuming default valid value as 4 bytes caused
that actual valid value in INTEGER storage class cannot be stored and
existing value cannot be found.
https://www.sqlite.org/datatype3.html
We should allow valid value in INTEGER storage class in SQLite3 to fix
the regression.
Fixes #22594.
|
| |
| |
| |
| |
| | |
Actually SQLite3 doesn't have JSON storage class (so it is stored as a
TEXT like Date and Time). But emulating JSON types is convinient for
making database agnostic migrations.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If do not execute a type changing first, filling in default value may be
failed.
```
% ARCONN=postgresql be ruby -w -Itest test/cases/migration/compatibility_test.rb -n test_legacy_change_column_with_null_executes_update
Using postgresql
Run options: -n test_legacy_change_column_with_null_executes_update --seed 20459
E
Error:
ActiveRecord::Migration::CompatibilityTest#test_legacy_change_column_with_null_executes_update:
StandardError: An error has occurred, this and all later migrations canceled:
PG::StringDataRightTruncation: ERROR: value too long for type character varying(5)
: UPDATE "testings" SET "foo"='foobar' WHERE "foo" IS NULL
```
|
| |
| |
| |
| |
| |
| |
| | |
`options_for_index_columns`
And placed `add_options_for_index_columns` in `schema_statements.rb`
consistently to ease to find related code.
|
| | |
|
|\ \
| | |
| | | |
Extract sql fragment generators from PostgreSQL adapter
|
| | | |
|
| | | |
|
| |/
|/|
| |
| | |
Add validate_constraint and update naming
|
| | |
|
|\ \
| | |
| | |
| | |
| | | |
gregnavis/support-postgresql-operator-classes-in-indexes
Add support for PostgreSQL operator classes to add_index
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add support for specifying non-default operator classes in PostgreSQL
indexes. An example CREATE INDEX query that becomes possible is:
CREATE INDEX users_name ON users USING gist (name gist_trgm_ops);
Previously it was possible to specify the `gist` index but not the
custom operator class. The `add_index` call for the above query is:
add_index :users, :name, using: :gist, opclasses: {name: :gist_trgm_ops}
|
|/ /
| |
| |
| |
| |
| | |
In #30510, `StatementPool` in `AbstractMysqlAdapter` was hidden in the
doc. But that class is also had in sqlite3 and postgresql adapters and
the base class is :nodoc: class.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since #31173, mysql2 adapter depends on `automatic_close` which is
introduced since mysql2 0.4.3. So the adapter with the mysql2 version
before doesn't work with fork now.
```
% ARCONN=mysql2 be ruby -w -Itest test/cases/connection_adapters/connection_handler_test.rb -n test_forked_child_doesnt_mangle_parent_connection
Using mysql2
Run options: -n test_forked_child_doesnt_mangle_parent_connection --seed 19988
/Users/kamipo/src/github.com/rails/rails/activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb:108:in `discard!': undefined method `automatic_close=' for #<Mysql2::Client:0x00007fedaa91dfd0> (NoMethodError)
```
This drops mysql2 version less than 0.4.3 to guarantee fork safety.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
statement due to user request (#31235)
This changes `StatementTimeout` to `QueryCanceled` for PostgreSQL.
In MySQL, errno 1317 (`ER_QUERY_INTERRUPTED`) is only used when the
query is manually cancelled.
But in PostgreSQL, `QUERY_CANCELED` error code (57014) which is used
`StatementTimeout` is also used when the both case. And, we can not tell
which reason happened.
So I decided to introduce new error class `QueryCanceled` closer to the
error code name.
|
| |
| |
| |
| |
| |
| | |
Since #31129, new error class `StatementTimeout` has been added.
`TransactionTimeout` is caused by the timeout shorter than
`StatementTimeout`, but its name is too generic. I think that it should
be a name that understands the difference with `StatementTimeout`.
|
|\ \
| | |
| | | |
Flush idle database connections
|
| | | |
|
| |/
|/| |
|
|\ \
| | |
| | | |
Add :nodoc: to activerecord [ci skip]
|
| | | |
|
|\ \ \
| |_|/
|/| | |
Improve AR connection fork safety
|
| | |
| | |
| | |
| | |
| | |
| | | |
Use whatever adapter-provided means we have available to ensure forked
children don't send quit/shutdown/goodbye messages to the server on
connections that belonged to their parent.
|
|\ \ \
| | | |
| | | | |
Prevent deadlocks with load interlock and DB lock.
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This fixes an issue where competing threads deadlock each other.
- Thread A holds the load interlock but is blocked on getting the DB lock
- Thread B holds the DB lock but is blocked on getting the load interlock (for example when there is a `Model.transaction` block that needs to autoload)
This solution allows for dependency loading in other threads while a thread is waiting to acquire the DB lock.
Fixes #31019
|
|\ \ \ \
| |_|/ /
|/| | | |
Switch to LIFO for the connection pool
|
| | | | |
|
| | | | |
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Using a FIFO for the connection pool can lead to issues when there are
upstream components (pgbouncer, haproxy, etc.) that terminate
connections that are idle after a period of time. Switching to a LIFO
reduces the probability that a thread will checkout a connection that is
about to be closed by an idle timeout in an upstream component.
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
timeout exceeded (#31129)
We are sometimes using The MAX_EXECUTION_TIME hint for MySQL depending
on the situation. It will prevent catastrophic performance down by wrong
performing queries.
The new error class `StatementTimeout` will make to be easier to handle
that case.
https://dev.mysql.com/doc/refman/5.7/en/optimizer-hints.html#optimizer-hints-execution-time
|
| | | |
| | | |
| | | |
| | | | |
Follow up of #30360.
|
| |/ /
|/| |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Attribute modules (`Attribute`, `Attributes`, `AttributeSet`) uses
`Type`, but referencing `Type` before the modules still fail.
```
% ./bin/test -w test/cases/attribute_test.rb -n test_with_value_from_user_validates_the_value
Run options: -n test_with_value_from_user_validates_the_value --seed 31876
E
Error:
ActiveModel::AttributeTest#test_with_value_from_user_validates_the_value:
NameError: uninitialized constant ActiveModel::AttributeTest::Type
/Users/kamipo/src/github.com/rails/rails/activemodel/test/cases/attribute_test.rb:233:in `block in <class:AttributeTest>'
bin/test test/cases/attribute_test.rb:232
Finished in 0.002985s, 335.0479 runs/s, 335.0479 assertions/s.
1 runs, 1 assertions, 0 failures, 1 errors, 0 skips
```
Probably we need more autoloading at least `Type`.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
```
[NoMethodError]: undefined method `state' for nil:NilClass Method:[rescue in block in refresh]
```
In `within_new_transaction`, there is the possibility that
`begin_transaction` returns a `nil`. (i.e.: so `transaction = nil`)
So this method is checking `transaction` for nil in 2 spots.
Unfortunately, there is one line that is not checking `transaction` for `nil`
That line, `commit_transaction`, throws an exception for us in AR 5.0.0.1
The problem with the method is finally realized in the error checking itself.
it calls `transaction.state` (i.e.: nil.state) and that is the final exception
raised.
The actual underlying (user) issue is hidden by this line.
Solution is test transaction for nil.
|
| | |
| | |
| | |
| | |
| | |
| | | |
Replace the primary key type `integer` in docs with `bigint`.
ref #26266
|
|\ \ \
| | | |
| | | | |
Move `extensions` to `PostgreSQL::SchemaDumper`
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
and abstract `SchemaDumper#extensions` is now an empty method.
Since #30337, every database adapter has its own `SchemaDumper`.
`extensions` are only supported by PostgreSQL database and postgresql database adapter.
|
|\ \ \ \
| |/ / /
|/| | | |
Save index order :desc to schema.rb (sqlite). Fixes #30902
|
| | | | |
|
| | | |
| | | |
| | | |
| | | |
| | | | |
Although the sqlite adapter supports index sort orders, they
weren't being written to db/schema.rb.
|
|/ / /
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
since the minimum version of PostgreSQL currently Rails supports is 9.1,
there is no need to handle if `supports_extensions?`
Refer https://www.postgresql.org/docs/9.1/static/sql-createextension.html
"CREATE EXTENSION"
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
`supports_disable_referential_integrity?` used to handle
if PostgreSQL database supports
`ALTER TABLE <table name> DISABLE/ENABLE TRIGGER` statements.
Refer https://github.com/rails/rails/commit/9a947af0e79cfb8692eb7e5ae94c1b8c40756f49
These statements have been documented since 8.1.
https://www.postgresql.org/docs/8.1/static/sql-altertable.html
> DISABLE/ENABLE TRIGGER
Now Rails supports PostgreSQL 9.1 or higher only.
No need to handle `supports_disable_referential_integrity?` anymore.
Also, this method does not exist in any other adapters including AbstractAdapter.
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| | |
| | |
| | | |
`initialize_internal_metadata_table`
|