Prodream/ormar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP - Pydantic v2 support (#1238)

Browse Source 
* WIP

* WIP - make test_model_definition tests pass

* WIP - make test_model_methods pass

* WIP - make whole test suit at least run - failing 49/443 tests

* WIP fix part of the getting pydantic tests as types of fields are now kept in core schema and not on fieldsinfo

* WIP fix validation in update by creating individual fields validators, failing 36/443

* WIP fix __pydantic_extra__ in intializing model, fix test related to pydantic config checks, failing 32/442

* WIP - fix enum schema in model_json_schema, failing 31/442

* WIP - fix copying through model, fix setting pydantic fields on through, fix default config and inheriting from it, failing 26/442

* WIP fix tests checking pydantic schema, fix excluding parent fields, failing 21/442

* WIP some missed files

* WIP - fix validators inheritance and fix validators in generated pydantic, failing 17/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - working on proper populating __dict__ for relations for new schema dumping, some work on openapi docs, failing 13/442

* WIP - remove property fields as pydantic has now computed_field on its own, failing 9/442

* WIP - fixes in docs, failing 8/442

* WIP - fix tests for largebinary schema, wrapped bytes fields fail in pydantic, will be fixed in pydantic-core, remaining is circural schema for related models, failing 6/442

* WIP - fix to pk only models in schemas

* Getting test suites to pass (#1249)

* wip, fixing tests

* iteration, fixing some more tests

* iteration, fixing some more tests

* adhere to comments

* adhere to comments

* remove unnecessary dict call, re-add getattribute for testing

* todo for reverse relationship

* adhere to comments, remove prints

* solve circular refs

* all tests pass 🎉

* remove 3.7 from tests

* add lint and type check jobs

* reforat with ruff, fix jobs

* rename jobs

* fix imports

* fix evaluate in py3.8

* partially fix coverage

* fix coverage, add more tests

* fix test ids

* fix test ids

* fix lint, fix docs, make docs fully working scripts, add test docs job

* fix pyproject

* pin py ver in test docs

* change dir in test docs

* fix pydantic warning hack

* rm poetry call in test_docs

* switch to pathlib in test docs

* remove coverage req test docs

* fix type check tests, fix part of types

* fix/skip next part of types

* fix next part of types

* fix next part of types

* fix coverage

* fix coverage

* fix type (bit dirty 🤷)

* fix some code smells

* change pre-commit

* tweak workflows

* remove no root from tests

* switch to full python path by passing sys.executable

* some small refactor in new base model, one sample test, change makefile

* small refactors to reduce complexity of methods

* temp add tests for prs against pydantic_v2

* remove all references to __fields__

* remove all references to construct, deprecate the method and update model_construct to be in line with pydantic

* deprecate dict and add model_dump, todo switch to model_dict in calls

* fix tests

* change to union

* change to union

* change to model_dump and model_dump_json from dict and json deprecated methods, deprecate them in ormar too

* finish switching dict() -> model_dump()

* finish switching json() -> model_dump_json()

* remove fully pydantic_only

* switch to extra for payment card, change missed json calls

* fix coverage - no more warnings internal

* fix coverage - no more warnings internal - part 2

* split model_construct into own and pydantic parts

* split determine pydantic field type

* change to new field validators

* fix benchmarks, add codspeed instead of pytest-benchmark, add action and gh workflow

* restore pytest-benchmark

* remove codspeed

* pin pydantic version, restore codspeed

* change on push to pydantic_v2 to trigger first one

* Use lifespan function instead of event (#1259)

* check return types

* fix imports order, set warnings=False on json that passes the dict, fix unnecessary loop in one of the test

* remove references to model's meta as it's now ormar config, rename related methods too

* filter out pydantic serializer warnings

* remove choices leftovers

* remove leftovers after property_fields, keep only enough to exclude them in initialization

* add migration guide

* fix meta references

* downgrade databases for now

* Change line numbers in documentation (#1265)

* proofread and fix the docs, part 1

* proofread and fix the docs for models

* proofread and fix the docs for fields

* proofread and fix the docs for relations

* proofread and fix rest of the docs, add release notes for 0.20

* create tables in new docs src

* cleanup old deps, uncomment docs publish on tag

* fix import reorder

---------

Co-authored-by: TouwaStar <30479449+TouwaStar@users.noreply.github.com>
Co-authored-by: Goran Mekić <meka@tilda.center>
This commit is contained in:
collerek
2024-03-23 19:28:28 +01:00
committed by GitHub
parent 3a206dd8dc
commit 500625f0ec
294 changed files with 8132 additions and 9311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/relations/foreign-key.md
View File

@ -14,7 +14,7 @@ Sqlalchemy column and Type are automatically taken from target `Model`.
To define a relation add `ForeignKey` field that points to related `Model`.
```Python hl_lines="29"
```Python hl_lines="30"
--8<-- "../docs_src/fields/docs003.py"
```
@ -24,7 +24,7 @@ To define a relation add `ForeignKey` field that points to related `Model`.
By default it's child (source) `Model` name + s, like courses in snippet below:
```Python hl_lines="29 35"
```Python hl_lines="29 36"
--8<-- "../docs_src/fields/docs001.py"
```
@ -45,15 +45,14 @@ But you cannot:
* Access the related field from reverse model with `related_name`
* Even if you `select_related` from reverse side of the model the returned models won't be populated in reversed instance (the join is not prevented so you still can `filter` and `order_by` over the relation)
* The relation won't be populated in `dict()` and `json()`
* The relation won't be populated in `model_dump()` and `model_dump_json()`
* You cannot pass the nested related objects when populating from dictionary or json (also through `fastapi`). It will be either ignored or error will be raised depending on `extra` setting in pydantic `Config`.
Example:
```python
class Author(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
first_name: str = ormar.String(max_length=80)
@ -61,8 +60,7 @@ class Author(ormar.Model):
class Post(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
title: str = ormar.String(max_length=200)
@ -82,8 +80,8 @@ authors = (
assert authors[0].first_name == "Test"
# note that posts are not populated for author even if explicitly
# included in select_related - note no posts in dict()
assert author.dict(exclude={"id"}) == {"first_name": "Test", "last_name": "Author"}
# included in select_related - note no posts in model_dump()
assert author.model_dump(exclude={"id"}) == {"first_name": "Test", "last_name": "Author"}
# still can filter through fields of related model
authors = await Author.objects.filter(posts__title="Test Post").all()
@ -112,7 +110,7 @@ assert department.courses[0] == course
!!!warning
If you want to add child model on related model the primary key value for parent model **has to exist in database**.
Otherwise ormar will raise RelationshipInstanceError as it cannot set child's ForeignKey column value
Otherwise ormar will raise `RelationshipInstanceError` as it cannot set child's ForeignKey column value
if parent model has no primary key value.
That means that in example above the department has to be saved before you can call `department.courses.add()`.
@ -151,7 +149,7 @@ await department.courses.remove(course, keep_reversed=False)
Removal of all related models in one call.
Like remove by default `clear()` nulls the ForeigKey column on child model (all, not matter if they are loaded or not).
Like with remove, by default, `clear()` nulls the ForeigKey column on child model (all, not matter if they are loaded or not).
```python
# nulls department column on all courses related to this department
@ -173,9 +171,9 @@ To read which methods of QuerySet are available read below [querysetproxy][query
## related_name
But you can overwrite this name by providing `related_name` parameter like below:
You can overwrite related model field name by providing `related_name` parameter like below:
```Python hl_lines="29 35"
```Python hl_lines="27-29 35"
--8<-- "../docs_src/fields/docs002.py"
```
@ -230,7 +228,7 @@ You have several ways to set-up a relationship connection.
The most obvious one is to pass a related `Model` instance to the constructor.
```Python hl_lines="34-35"
```Python hl_lines="35-36"
--8<-- "../docs_src/relations/docs001.py"
```
@ -238,7 +236,7 @@ The most obvious one is to pass a related `Model` instance to the constructor.
You can setup the relation also with just the pk column value of the related model.
```Python hl_lines="37-38"
```Python hl_lines="38-39"
--8<-- "../docs_src/relations/docs001.py"
```
@ -246,9 +244,9 @@ You can setup the relation also with just the pk column value of the related mod
Next option is with a dictionary of key-values of the related model.
You can build the dictionary yourself or get it from existing model with `dict()` method.
You can build the dictionary yourself or get it from existing model with `model_dump()` method.
```Python hl_lines="40-41"
```Python hl_lines="41-42"
--8<-- "../docs_src/relations/docs001.py"
```
@ -256,7 +254,7 @@ You can build the dictionary yourself or get it from existing model with `dict()
Finally you can explicitly set it to None (default behavior if no value passed).
```Python hl_lines="43-44"
```Python hl_lines="44-45"
--8<-- "../docs_src/relations/docs001.py"
```
@ -283,4 +281,4 @@ Finally you can explicitly set it to None (default behavior if no value passed).
[fields]: ./queries.md#fields
[exclude_fields]: ./queries.md#exclude_fields
[order_by]: ./queries.md#order_by
[server_default]: ../fields/common-parameters.md#server-default
[server_default]: ../fields/common-parameters.md#server-default

76
docs/relations/index.md
View File

@ -13,7 +13,7 @@ To read more about methods, possibilities, definition etc. please read the subse
To define many-to-one relation use `ForeignKey` field.
```Python hl_lines="17"
```Python hl_lines="26"
--8<-- "../docs_src/relations/docs003.py"
```
@ -24,13 +24,11 @@ To define many-to-one relation use `ForeignKey` field.
The definition of one-to-many relation also uses `ForeignKey`, and it's registered for you automatically.
So in relation ato example above.
So in relation to example above.
```Python hl_lines="17"
```Python hl_lines="7-8"
class Department(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -52,21 +50,22 @@ class Department(ormar.Model):
To define many-to-many relation use `ManyToMany` field.
```python hl_lines="18"
```python hl_lines="19"
class Category(ormar.Model):
class Meta:
tablename = "categories"
database = database
metadata = metadata
ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
tablename="categories",
)
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=40)
class Post(ormar.Model):
class Meta:
tablename = "posts"
database = database
metadata = metadata
ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
)
id: int = ormar.Integer(primary_key=True)
title: str = ormar.String(max_length=200)
@ -92,18 +91,24 @@ side of the current query for m2m models.
So if you query from model `A` to model `B`, only model `B` has through field exposed.
Which kind of make sense, since it's a one through model/field for each of related models.
```python hl_lines="10-15"
```python hl_lines="12-21"
class Category(ormar.Model):
class Meta(BaseMeta):
tablename = "categories"
ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
tablename="categories",
)
id = ormar.Integer(primary_key=True)
name = ormar.String(max_length=40)
# you can specify additional fields on through model
class PostCategory(ormar.Model):
class Meta(BaseMeta):
tablename = "posts_x_categories"
ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
tablename="posts_x_categories",
)
id: int = ormar.Integer(primary_key=True)
sort_order: int = ormar.Integer(nullable=True)
@ -111,8 +116,10 @@ class PostCategory(ormar.Model):
class Post(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
)
id: int = ormar.Integer(primary_key=True)
title: str = ormar.String(max_length=200)
@ -130,7 +137,7 @@ class Post(ormar.Model):
## Relationship default sort order
By default relations follow model default sort order so `primary_key` column ascending, or any sort order se in `Meta` class.
By default relations follow model default sort order so `primary_key` column ascending, or any sort order se in `ormar_config` object.
!!!tip
To read more about models sort order visit [models](../models/index.md#model-sort-order) section of documentation
@ -143,27 +150,26 @@ columns also `Through` model columns `{through_field_name}__{column_name}`
Sample configuration might look like this:
```python hl_lines="24"
```python hl_lines="23"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()
class BaseMeta(ormar.ModelMeta):
metadata = metadata
database = database
base_ormar_config = ormar.OrmarConfig(
database=database,
metadata=metadata,
)
class Author(ormar.Model):
class Meta(BaseMeta):
tablename = "authors"
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Book(ormar.Model):
class Meta(BaseMeta):
tablename = "books"
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
author: Optional[Author] = ormar.ForeignKey(
@ -186,14 +192,12 @@ In order to create auto-relation or create two models that reference each other
different relations (remember the reverse side is auto-registered for you), you need to use
`ForwardRef` from `typing` module.
```python hl_lines="1 11 14"
```python hl_lines="1 9 12"
PersonRef = ForwardRef("Person")
class Person(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -210,4 +214,4 @@ Person.update_forward_refs()
[foreign-keys]: ./foreign-key.md
[many-to-many]: ./many-to-many.md
[queryset-proxy]: ./queryset-proxy.md
[postponed-annotations]: ./postponed-annotations.md
[postponed-annotations]: ./postponed-annotations.md

60
docs/relations/many-to-many.md
View File

@ -9,7 +9,7 @@ Sqlalchemy column and Type are automatically taken from target `Model`.
## Defining Models
```Python hl_lines="40"
```Python hl_lines="34"
--8<-- "../docs_src/relations/docs002.py"
```
@ -24,20 +24,18 @@ news = await Category.objects.create(name="News")
`ForeignKey` fields are automatically registering reverse side of the relation.
By default it's child (source) `Model` name + s, like courses in snippet below:
By default it's child (source) `Model` name + s, like `posts` in snippet below:
```python
```python hl_lines="25-26"
class Category(ormar.Model):
class Meta(BaseMeta):
tablename = "categories"
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=40)
class Post(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
title: str = ormar.String(max_length=200)
@ -81,31 +79,31 @@ categories: Optional[Union[Category, List[Category]]] = ormar.ManyToMany(
If you are sure you don't want the reverse relation you can use `skip_reverse=True`
flag of the `ManyToMany`.
If you set `skip_reverse` flag internally the field is still registered on the other
side of the relationship so you can:
If you set `skip_reverse` flag internally the field is still registered on the other
side of the relationship so you can:
* `filter` by related models fields from reverse model
* `order_by` by related models fields from reverse model
But you cannot:
But you cannot:
* access the related field from reverse model with `related_name`
* even if you `select_related` from reverse side of the model the returned models won't be populated in reversed instance (the join is not prevented so you still can `filter` and `order_by` over the relation)
* the relation won't be populated in `dict()` and `json()`
* the relation won't be populated in `model_dump()` and `json()`
* you cannot pass the nested related objects when populating from dictionary or json (also through `fastapi`). It will be either ignored or error will be raised depending on `extra` setting in pydantic `Config`.
Example:
```python
class Category(ormar.Model):
class Meta(BaseMeta):
tablename = "categories"
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=40)
class Post(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
title: str = ormar.String(max_length=200)
@ -126,8 +124,8 @@ categories = (
assert categories[0].first_name == "Test"
# note that posts are not populated for author even if explicitly
# included in select_related - note no posts in dict()
assert news.dict(exclude={"id"}) == {"name": "News"}
# included in select_related - note no posts in model_dump()
assert news.model_dump(exclude={"id"}) == {"name": "News"}
# still can filter through fields of related model
categories = await Category.objects.filter(posts__title="Hello, M2M").all()
@ -141,7 +139,7 @@ assert len(categories) == 1
Optionally if you want to add additional fields you can explicitly create and pass
the through model class.
```Python hl_lines="14-20 29"
```Python hl_lines="19-24 32"
--8<-- "../docs_src/relations/docs004.py"
```
@ -170,9 +168,7 @@ So in example like this:
```python
... # course declaration omitted
class Student(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -180,10 +176,7 @@ class Student(ormar.Model):
# will produce default Through model like follows (example simplified)
class StudentCourse(ormar.Model):
class Meta:
database = database
metadata = metadata
tablename = "students_courses"
ormar_config = base_ormar_config.copy(tablename="students_courses")
id: int = ormar.Integer(primary_key=True)
student = ormar.ForeignKey(Student) # default name
@ -199,10 +192,14 @@ Example:
```python
... # course declaration omitted
base_ormar_config = ormar.OrmarConfig(
database=databases.Database("sqlite:///db.sqlite"),
metadata=sqlalchemy.MetaData(),
)
class Student(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -212,10 +209,7 @@ class Student(ormar.Model):
# will produce Through model like follows (example simplified)
class StudentCourse(ormar.Model):
class Meta:
database = database
metadata = metadata
tablename = "students_courses"
ormar_config = base_ormar_config.copy(tablename="student_courses")
id: int = ormar.Integer(primary_key=True)
student_id = ormar.ForeignKey(Student) # set by through_relation_name
@ -238,7 +232,7 @@ so it's useful only when additional fields are provided on `Through` model.
In a sample model setup as following:
```Python hl_lines="14-20 29"
```Python hl_lines="19-24 32"
--8<-- "../docs_src/relations/docs004.py"
```

46
docs/relations/postponed-annotations.md
View File

@ -14,21 +14,8 @@ First, you need to import the required ref from typing.
from typing import ForwardRef
```
But note that before python 3.7 it used to be internal, so for python <= 3.6 you need
```python
from typing import _ForwardRef as ForwardRef
```
or since `pydantic` is required by `ormar` it can handle this switch for you.
In that case you can simply import ForwardRef from pydantic regardless of your python version.
```python
from pydantic.typing import ForwardRef
```
Now we need a sample model and a reference to the same model,
which will be used to creat a self referencing relation.
which will be used to create a self referencing relation.
```python
# create the forwardref to model Person
@ -36,9 +23,7 @@ PersonRef = ForwardRef("Person")
class Person(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -72,9 +57,7 @@ PersonRef = ForwardRef("Person")
class Person(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -93,14 +76,10 @@ and through parameters.
ChildRef = ForwardRef("Child")
class ChildFriend(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
class Child(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -132,9 +111,7 @@ TeacherRef = ForwardRef("Teacher")
class Student(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -144,16 +121,11 @@ class Student(ormar.Model):
class StudentTeacher(ormar.Model):
class Meta(ModelMeta):
tablename = 'students_x_teachers'
metadata = metadata
database = db
ormar_config = base_ormar_config.copy(tablename='students_x_teachers')
class Teacher(ormar.Model):
class Meta(ModelMeta):
metadata = metadata
database = db
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -168,4 +140,4 @@ Student.update_forward_refs()
!!!warning
Remember that `related_name` needs to be unique across related models regardless
of how many relations are defined.
of how many relations are defined.

6
docs/relations/queryset-proxy.md
View File

@ -58,7 +58,7 @@ assert post.categories[0] == news
`get_or_create(_defaults: Optional[Dict[str, Any]] = None, **kwargs) -> Tuple[Model, bool]`
Tries to get a row meeting the criteria and if NoMatch exception is raised it creates a new one with given kwargs and _defaults.
Tries to get a row meeting the criteria and if `NoMatch` exception is raised it creates a new one with given kwargs and _defaults.
!!!tip
Read more in queries documentation [get_or_create][get_or_create]
@ -127,7 +127,7 @@ provided Through model.
Given sample like this:
```Python hl_lines="14-20 29"
```Python hl_lines="19-24 32"
--8<-- "../docs_src/relations/docs004.py"
```
@ -174,7 +174,7 @@ Updates the related model with provided keyword arguments, return number of upda
Note that for `ManyToMany` relations update can also accept an argument with through field
name and a dictionary of fields.
```Python hl_lines="14-20 29"
```Python hl_lines="19-24 32"
--8<-- "../docs_src/relations/docs004.py"
```