Add schema support to Persistent (issue #93) #1561

curranosaurus · 2024-12-30T22:41:58Z

Addresses #93 by adding schema support to Persistent. You specify a Persistent model's schema using syntax like schema=foo.
I worked on this together with @benjonesy.

Design challenges and decisions

"Schema" means many things to many backends

In this PR, we use the word "schema" to refer to various devices that backends offer for namespacing tables.

✔️ In PostgreSQL, a schema exists in between the level of a table and a database.
✔️ In MySQL and SQLite, schemas and databases are synonymous concepts.
❌ MongoDB and Redis do not have a concept analogous to schemas. This PR does not extend support to these backends.

Schemas may or may not exist

We considered designing this PR so that Persistent would create schemas for the user if they did not already exist. But we chose not to do that. This means that local development with Persistent using schemas will require some manual SQL migrations to create the schemas that the user wants. But for conventional use-cases, schema creation will be a rare event, so this is not very onerous for developers. Below are some issues one would have to solve to make Persistent automate schema creation for the developer:

Three-stage migrations

RIght now, the CREATe TABLE DDL statements are sorted to the top of the migration output, so that the (potentially recursive) foreign-key constraints are created only after all of the tables they reference exist. If we also wanted Persistent to ensure the existence of schemas, we would need to sort backend-appropriate versions of CREATE SCHEMA statements above the CREATe TABLE statements.

Granting permissions in MySQL

Before a connection can use a newly created database in MySQL, one must run a GRANT statement to give the current user the requisite DDL operations to run CREATe TABLE statements. We would somehow need to make Persistent aware of the GRANT statement required.

File name and directory in SQLite

SQLite's invocation for creating a new database also specifies the file name and location for that database's .db file. Persistent would need to be opinionated about this directory scheme, or we would have to add a configuration option, if we wanted to support schema creation in SQLite.

PR Template checklist

Before submitting your PR, check that you've:

Documented new APIs with Haddock markup
Added @since declarations to the Haddock
Ran stylish-haskell on any changed files.
Adhered to the code style (see the .editorconfig file for details)

After submitting your PR:

Update the Changelog.md file with a link to your PR
Bumped the version number if there isn't an (unreleased) on the Changelog
Check that CI passes (or if it fails, for reasons unrelated to your change, like CI timeouts)

Add new `entitySchema` field and refactor callsites of `entityDB` to use `getEntityDBName` instead

[INT-844] prepend schema to entity db name

…ntityDBName revert change to `getEntityDBName`

…tters-for-entitySchema add setters and getters for `entitySchema`

…b.com:curranosaurus/persistent into curran/update-entitySchema-to-its-own-newtype

…o-its-own-newtype Create newtype for schema name

…t-schema update sqlQQ test to consider a case with a schema

…s-migrations Use new entitySchema field for Postgres migrations

…sql-queries Start using table schema in queries with Postgres

curranosaurus · 2024-12-30T23:45:08Z

persistent-sqlite/test/main.hs

 import qualified PersistentTest
+import qualified PersistUniqueTest


Sorted imports due to stylish-haskell.

curranosaurus · 2024-12-30T23:47:43Z

persistent-sqlite/test/SqliteInit.hs

+       , (==@)
+       , (@/=)
+       , (@==)
+       )


Reformatted by stylish-haskell.

curranosaurus · 2024-12-30T23:48:11Z

persistent-qq/test/Spec.hs

 import PersistentTestModels
-import qualified CodeGenTest
+import PersistTestPetType


Imports sorted by stylish-haskell.

curranosaurus · 2024-12-30T23:48:29Z

persistent-postgresql/test/main.hs

 import qualified PersistentTest
+import qualified PersistUniqueTest


Imports sorted by stylish-haskell.

curranosaurus · 2024-12-30T23:48:59Z

persistent-mysql/test/main.hs

 import qualified PersistentTest
+import qualified PersistUniqueTest


Imports sorted by stylish-haskell.

curranosaurus · 2024-12-30T23:52:13Z

persistent-sqlite/Database/Persist/Sqlite.hs

@@ -539,7 +541,7 @@ getCopyTable :: [EntityDef]
             -> EntityDef
             -> IO [(Bool, Text)]
 getCopyTable allDefs getter def = do


The changes to this function were non-trivial. A naive change to this function would allow identically named tables in different schemas to clobber each other when copying them into the temporary schema. So we do some manual namespacing by prefixing the table name with <schema>_ when copying it to the temporary schema.

curranosaurus · 2024-12-30T23:55:13Z

persistent-sqlite/Database/Persist/Sqlite.hs

@@ -470,6 +470,8 @@ migrate' allDefs getter val = do
  where
    def = val
    table = getEntityDBName def
+    schema = getEntitySchema def
+    sqliteMaster = maybe "sqlite_master" (\schema' -> escapeS schema' <> ".sqlite_master") schema


There is one copy of the sqlite_master table in each schema, and it only contains information about the tables in its own schema.

curranosaurus · 2024-12-31T00:06:04Z

persistent-mysql/Database/Persist/MySQL.hs

-                    , fcOnDelete = parseCascadeAction onDel
-                    }
+                then Just $
+                  let colSchema =


This is one of the most complex pieces of logic in the persistent-mysql diff, it's worth scrutinizing whether this makes sense.

curranosaurus · 2024-12-31T00:48:40Z

I've run the test suites for the changed packages, and they all pass except for persistent-mysql. But the test failure I get there also occurs on master branch for me, so seems unrelated to these changes. Here is the failure output I get, for reference:

Failures:

  src/ForeignKey.hs:198:12: 
  1) foreign keys options deletes cascades with field self reference to the whole chain
       expected: []
        but got: [Entity {entityKey = Chain3Key {unChain3Key = SqlBackendKey {unSqlBackendKey = 92}}, entityVal = Chain3 {chain
3Name = 2, chain3Previous = Just (Chain3Key {unChain3Key = SqlBackendKey {unSqlBackendKey = 91}})}},Entity {entityKey = Chain3K
ey {unChain3Key = SqlBackendKey {unSqlBackendKey = 93}}, entityVal = Chain3 {chain3Name = 3, chain3Previous = Just (Chain3Key {
unChain3Key = SqlBackendKey {unSqlBackendKey = 92}})}}]

  To rerun use: --match "/foreign keys options/deletes cascades with field self reference to the whole chain/" --seed 6043809

Randomized with seed 6043809

curranosaurus · 2024-12-31T00:51:08Z

persistent-qq/test/Spec.hs

@@ -37,6 +40,7 @@ runConn f = do
 db :: SqlPersistT (LoggingT (ResourceT IO)) a -> IO a
 db actions = do
  runResourceT $ runConn $ do
+      rawSql @(Single Int64) ("attach 'animals.db' as animals") []


This ensures that the tests referencing the animals schema succeed.

persistent-sqlite/Database/Persist/Sqlite.hs

curranosaurus and others added 30 commits September 30, 2024 15:56

add new field and refactor away from entityDB

f2f39a2

Merge pull request #1 from curranosaurus/curran/int-843

08a50bd

Add new `entitySchema` field and refactor callsites of `entityDB` to use `getEntityDBName` instead

add new field to construction

9c09206

Merge branch 'master' of github.com:curranosaurus/persistent

79eaa20

prepend schema to entity db name

1700101

Merge pull request #2 from curranosaurus/curran/int-844

540c1b8

[INT-844] prepend schema to entity db name

add schema test

e07cae1

remove redundant quotation marks

6fea10e

revert

eb49123

create newtype for schema name

e0a7d27

add setters and getters

ea96742

Merge pull request #3 from curranosaurus/curran/revert-change-to-getE…

685f2cf

…ntityDBName revert change to `getEntityDBName`

Merge pull request #5 from curranosaurus/curran/create-setters-and-ge…

009f7ab

…tters-for-entitySchema add setters and getters for `entitySchema`

suffix with DB

27e4ee7

Merge branch 'curran/update-entitySchema-to-its-own-newtype' of githu…

0c64924

…b.com:curranosaurus/persistent into curran/update-entitySchema-to-its-own-newtype

save wip

ab479e6

save wip

6048242

Merge pull request #4 from curranosaurus/curran/update-entitySchema-t…

607bd67

…o-its-own-newtype Create newtype for schema name

add schema lookups to foreign key lookup

b079be2

add schema to information_schema query

ce4eb39

change decisions about how to handle the public schema

1f8940d

delete accidental hardcoding of foo schema

4b37e1e

update migrater

91f56b4

update sqlQQ test to consider a case with a schema

799b58d

strengthen test

62ed272

Merge pull request #8 from curranosaurus/curran/update-sqlqq-to-injec…

320a9c1

…t-schema update sqlQQ test to consider a case with a schema

Merge pull request #7 from curranosaurus/curran/use-schema-in-postgre…

b96c415

…s-migrations Use new entitySchema field for Postgres migrations

Merge pull request #6 from curranosaurus/curran/use-schema-in-postgre…

8fe5757

…sql-queries Start using table schema in queries with Postgres

add schema to sqlite expressions

170450b

add schema test

1ec4ca7

curranosaurus added 10 commits December 30, 2024 18:02

bump persistent-mysql versions and add haddocks

cfafec6

bump versions in postgresql project

1b068ae

bump version of sqlite package

3bd72fd

bump persistent-test version

8b6b752

add more haddock comments

7f990da

undo redundant change

65969ce

update changelogs

a6598f3

add more doc comments

e8c00d5

run stylish-haskell

755b9d2

bump persistent-qq version

98b49bc

curranosaurus commented Dec 30, 2024

View reviewed changes

persistent-sqlite/test/SqliteInit.hs

, (==@)

, (@/=)

, (@==)

)

Copy link

Author

curranosaurus Dec 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reformatted by stylish-haskell.

curranosaurus commented Dec 30, 2024

View reviewed changes

account for null referenced schema name possibility

7bf74fc

curranosaurus commented Dec 31, 2024

View reviewed changes

curranosaurus added 3 commits December 30, 2024 19:22

document a tricky use of a table-valued pragma

916cc4a

update postgres readme

035075a

fix persistent-qq tests

a4e8af3

curranosaurus force-pushed the master branch from 6b464fa to a4e8af3 Compare December 31, 2024 00:48

curranosaurus marked this pull request as ready for review December 31, 2024 00:48

curranosaurus commented Dec 31, 2024

View reviewed changes

persistent-sqlite/Database/Persist/Sqlite.hs Outdated Show resolved Hide resolved

curranosaurus added 3 commits December 30, 2024 20:07

use lowercase schema name

e7efb53

add schema to checkForeignKeys query

9b1aae0

throw error if there is a cross-schema reference

896edc2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add schema support to Persistent (issue #93) #1561

Add schema support to Persistent (issue #93) #1561

curranosaurus commented Dec 30, 2024 •

edited

Loading

curranosaurus Dec 30, 2024

curranosaurus Dec 30, 2024

curranosaurus Dec 30, 2024

curranosaurus Dec 30, 2024

curranosaurus Dec 30, 2024

curranosaurus Dec 30, 2024 •

edited

Loading

curranosaurus Dec 30, 2024

curranosaurus Dec 31, 2024

curranosaurus commented Dec 31, 2024

curranosaurus Dec 31, 2024

		import qualified PersistentTest
		import qualified PersistUniqueTest

Add schema support to Persistent (issue #93) #1561

Are you sure you want to change the base?

Add schema support to Persistent (issue #93) #1561

Conversation

curranosaurus commented Dec 30, 2024 • edited Loading

Design challenges and decisions

"Schema" means many things to many backends

Schemas may or may not exist

Three-stage migrations

Granting permissions in MySQL

File name and directory in SQLite

PR Template checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

curranosaurus Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

curranosaurus commented Dec 31, 2024

Choose a reason for hiding this comment

curranosaurus commented Dec 30, 2024 •

edited

Loading

curranosaurus Dec 30, 2024 •

edited

Loading