Prodream/ormar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP - Pydantic v2 support (#1238)

Browse Source 
* WIP

* WIP - make test_model_definition tests pass

* WIP - make test_model_methods pass

* WIP - make whole test suit at least run - failing 49/443 tests

* WIP fix part of the getting pydantic tests as types of fields are now kept in core schema and not on fieldsinfo

* WIP fix validation in update by creating individual fields validators, failing 36/443

* WIP fix __pydantic_extra__ in intializing model, fix test related to pydantic config checks, failing 32/442

* WIP - fix enum schema in model_json_schema, failing 31/442

* WIP - fix copying through model, fix setting pydantic fields on through, fix default config and inheriting from it, failing 26/442

* WIP fix tests checking pydantic schema, fix excluding parent fields, failing 21/442

* WIP some missed files

* WIP - fix validators inheritance and fix validators in generated pydantic, failing 17/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - fix through models setting - only on reverse side of relation, but always on reverse side, failing 15/442

* WIP - working on proper populating __dict__ for relations for new schema dumping, some work on openapi docs, failing 13/442

* WIP - remove property fields as pydantic has now computed_field on its own, failing 9/442

* WIP - fixes in docs, failing 8/442

* WIP - fix tests for largebinary schema, wrapped bytes fields fail in pydantic, will be fixed in pydantic-core, remaining is circural schema for related models, failing 6/442

* WIP - fix to pk only models in schemas

* Getting test suites to pass (#1249)

* wip, fixing tests

* iteration, fixing some more tests

* iteration, fixing some more tests

* adhere to comments

* adhere to comments

* remove unnecessary dict call, re-add getattribute for testing

* todo for reverse relationship

* adhere to comments, remove prints

* solve circular refs

* all tests pass 🎉

* remove 3.7 from tests

* add lint and type check jobs

* reforat with ruff, fix jobs

* rename jobs

* fix imports

* fix evaluate in py3.8

* partially fix coverage

* fix coverage, add more tests

* fix test ids

* fix test ids

* fix lint, fix docs, make docs fully working scripts, add test docs job

* fix pyproject

* pin py ver in test docs

* change dir in test docs

* fix pydantic warning hack

* rm poetry call in test_docs

* switch to pathlib in test docs

* remove coverage req test docs

* fix type check tests, fix part of types

* fix/skip next part of types

* fix next part of types

* fix next part of types

* fix coverage

* fix coverage

* fix type (bit dirty 🤷)

* fix some code smells

* change pre-commit

* tweak workflows

* remove no root from tests

* switch to full python path by passing sys.executable

* some small refactor in new base model, one sample test, change makefile

* small refactors to reduce complexity of methods

* temp add tests for prs against pydantic_v2

* remove all references to __fields__

* remove all references to construct, deprecate the method and update model_construct to be in line with pydantic

* deprecate dict and add model_dump, todo switch to model_dict in calls

* fix tests

* change to union

* change to union

* change to model_dump and model_dump_json from dict and json deprecated methods, deprecate them in ormar too

* finish switching dict() -> model_dump()

* finish switching json() -> model_dump_json()

* remove fully pydantic_only

* switch to extra for payment card, change missed json calls

* fix coverage - no more warnings internal

* fix coverage - no more warnings internal - part 2

* split model_construct into own and pydantic parts

* split determine pydantic field type

* change to new field validators

* fix benchmarks, add codspeed instead of pytest-benchmark, add action and gh workflow

* restore pytest-benchmark

* remove codspeed

* pin pydantic version, restore codspeed

* change on push to pydantic_v2 to trigger first one

* Use lifespan function instead of event (#1259)

* check return types

* fix imports order, set warnings=False on json that passes the dict, fix unnecessary loop in one of the test

* remove references to model's meta as it's now ormar config, rename related methods too

* filter out pydantic serializer warnings

* remove choices leftovers

* remove leftovers after property_fields, keep only enough to exclude them in initialization

* add migration guide

* fix meta references

* downgrade databases for now

* Change line numbers in documentation (#1265)

* proofread and fix the docs, part 1

* proofread and fix the docs for models

* proofread and fix the docs for fields

* proofread and fix the docs for relations

* proofread and fix rest of the docs, add release notes for 0.20

* create tables in new docs src

* cleanup old deps, uncomment docs publish on tag

* fix import reorder

---------

Co-authored-by: TouwaStar <30479449+TouwaStar@users.noreply.github.com>
Co-authored-by: Goran Mekić <meka@tilda.center>
This commit is contained in:
collerek
2024-03-23 19:28:28 +01:00
committed by GitHub
parent 3a206dd8dc
commit 500625f0ec
294 changed files with 8132 additions and 9311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
docs/models/methods.md
View File

@ -13,37 +13,37 @@ Available methods are described below.
## `pydantic` methods
Note that each `ormar.Model` is also a `pydantic.BaseModel`, so all `pydantic` methods are also available on a model,
especially `dict()` and `json()` methods that can also accept `exclude`, `include` and other parameters.
especially `model_dump()` and `model_dump_json()` methods that can also accept `exclude`, `include` and other parameters.
To read more check [pydantic][pydantic] documentation
## construct
## model_construct()
`construct` is a raw equivalent of `__init__` method used for construction of new instances.
`model_construct` is a raw equivalent of `__init__` method used for construction of new instances.
The difference is that `construct` skips validations, so it should be used when you know that data is correct and can be trusted.
The difference is that `model_construct` skips validations, so it should be used when you know that data is correct and can be trusted.
The benefit of using construct is the speed of execution due to skipped validation.
!!!note
Note that in contrast to `pydantic.construct` method - the `ormar` equivalent will also process the nested related models.
Note that in contrast to `pydantic.model_construct` method - the `ormar` equivalent will also process the nested related models.
!!!warning
Bear in mind that due to skipped validation the `construct` method does not perform any conversions, checks etc.
So it's your responsibility to provide that data that is valid and can be consumed by the database.
The only two things that construct still performs are:
Bear in mind that due to skipped validation the `construct` method does not perform any conversions, checks etc.
So it's your responsibility to provide that data that is valid and can be consumed by the database.
The only two things that construct still performs are:
* Providing a `default` value for not set fields
* Initialize nested ormar models if you pass a dictionary or a primary key value
* Providing a `default` value for not set fields
* Initialize nested ormar models if you pass a dictionary or a primary key value
## dict
## model_dump()
`dict` is a method inherited from `pydantic`, yet `ormar` adds its own parameters and has some nuances when working with default values,
`model_dump` is a method inherited from `pydantic`, yet `ormar` adds its own parameters and has some nuances when working with default values,
therefore it's listed here for clarity.
`dict` as the name suggests export data from model tree to dictionary.
`model_dump` as the name suggests export data from model tree to dictionary.
Explanation of dict parameters:
Explanation of model_dump parameters:
### include (`ormar` modified)
@ -55,14 +55,14 @@ Note that `pydantic` has an uncommon pattern of including/ excluding fields in l
And if you want to exclude the field in all children you need to pass a `__all__` key to dictionary.
You cannot exclude nested models in `Set`s in `pydantic` but you can in `ormar`
(by adding double underscore on relation name i.e. to exclude name of category for a book you cen use `exclude={"book__category__name"}`)
(by adding double underscore on relation name i.e. to exclude name of category for a book you can use `exclude={"book__category__name"}`)
`ormar` does not support by index exclusion/ inclusions and accepts a simplified and more user-friendly notation.
To check how you can include/exclude fields, including nested fields check out [fields](../queries/select-columns.md#fields) section that has an explanation and a lot of samples.
!!!note
The fact that in `ormar` you can exclude nested models in sets, you can exclude from a whole model tree in `response_model_exclude` and `response_model_include` in fastapi!
The fact that in `ormar` you can exclude nested models in sets, you can exclude from a whole model tree in `response_model_exclude` and `response_model_include` in fastapi!
### exclude (`ormar` modified)
@ -81,7 +81,7 @@ You cannot exclude nested models in `Set`s in `pydantic` but you can in `ormar`
To check how you can include/exclude fields, including nested fields check out [fields](../queries/select-columns.md#fields) section that has an explanation and a lot of samples.
!!!note
The fact that in `ormar` you can exclude nested models in sets, you can exclude from a whole model tree in `response_model_exclude` and `response_model_include` in fastapi!
The fact that in `ormar` you can exclude nested models in sets, you can exclude from a whole model tree in `response_model_exclude` and `response_model_include` in fastapi!
### exclude_unset
@ -90,16 +90,13 @@ To check how you can include/exclude fields, including nested fields check out [
Flag indicates whether fields which were not explicitly set when creating the model should be excluded from the returned dictionary.
!!!warning
Note that after you save data into database each field has its own value -> either provided by you, default, or `None`.
That means that when you load the data from database, **all** fields are set, and this flag basically stop working!
Note that after you save data into database each field has its own value -> either provided by you, default, or `None`.
That means that when you load the data from database, **all** fields are set, and this flag basically stop working!
```python
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100, default="Test")
@ -107,10 +104,7 @@ class Category(ormar.Model):
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -118,17 +112,17 @@ class Item(ormar.Model):
categories: List[Category] = ormar.ManyToMany(Category)
category = Category(name="Test 2")
assert category.dict() == {'id': None, 'items': [], 'name': 'Test 2',
assert category.model_dump() == {'id': None, 'items': [], 'name': 'Test 2',
'visibility': True}
assert category.dict(exclude_unset=True) == {'items': [], 'name': 'Test 2'}
assert category.model_dump(exclude_unset=True) == {'items': [], 'name': 'Test 2'}
await category.save()
category2 = await Category.objects.get()
assert category2.dict() == {'id': 1, 'items': [], 'name': 'Test 2',
assert category2.model_dump() == {'id': 1, 'items': [], 'name': 'Test 2',
'visibility': True}
# NOTE how after loading from db all fields are set explicitly
# as this is what happens when you populate a model from db
assert category2.dict(exclude_unset=True) == {'id': 1, 'items': [],
assert category2.model_dump(exclude_unset=True) == {'id': 1, 'items': [],
'name': 'Test 2', 'visibility': True}
```
@ -140,20 +134,14 @@ Flag indicates are equal to their default values (whether set or otherwise) shou
```python
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100, default="Test")
visibility: bool = ormar.Boolean(default=True)
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -162,15 +150,15 @@ class Item(ormar.Model):
category = Category()
# note that Integer pk is by default autoincrement so optional
assert category.dict() == {'id': None, 'items': [], 'name': 'Test', 'visibility': True}
assert category.dict(exclude_defaults=True) == {'items': []}
assert category.model_dump() == {'id': None, 'items': [], 'name': 'Test', 'visibility': True}
assert category.model_dump(exclude_defaults=True) == {'items': []}
# save and reload the data
await category.save()
category2 = await Category.objects.get()
assert category2.dict() == {'id': 1, 'items': [], 'name': 'Test', 'visibility': True}
assert category2.dict(exclude_defaults=True) == {'id': 1, 'items': []}
assert category2.model_dump() == {'id': 1, 'items': [], 'name': 'Test', 'visibility': True}
assert category2.model_dump(exclude_defaults=True) == {'id': 1, 'items': []}
```
### exclude_none
@ -181,10 +169,7 @@ Flag indicates whether fields which are equal to `None` should be excluded from
```python
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100, default="Test", nullable=True)
@ -192,10 +177,7 @@ class Category(ormar.Model):
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -204,16 +186,16 @@ class Item(ormar.Model):
category = Category(name=None)
assert category.dict() == {'id': None, 'items': [], 'name': None,
assert category.model_dump() == {'id': None, 'items': [], 'name': None,
'visibility': True}
# note the id is not set yet so None and excluded
assert category.dict(exclude_none=True) == {'items': [], 'visibility': True}
assert category.model_dump(exclude_none=True) == {'items': [], 'visibility': True}
await category.save()
category2 = await Category.objects.get()
assert category2.dict() == {'id': 1, 'items': [], 'name': None,
assert category2.model_dump() == {'id': 1, 'items': [], 'name': None,
'visibility': True}
assert category2.dict(exclude_none=True) == {'id': 1, 'items': [],
assert category2.model_dump(exclude_none=True) == {'id': 1, 'items': [],
'visibility': True}
```
@ -226,17 +208,14 @@ Setting flag to `True` will exclude all primary key columns in a tree, including
```python
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
item1 = Item(id=1, name="Test Item")
assert item1.dict() == {"id": 1, "name": "Test Item"}
assert item1.dict(exclude_primary_keys=True) == {"name": "Test Item"}
assert item1.model_dump() == {"id": 1, "name": "Test Item"}
assert item1.model_dump(exclude_primary_keys=True) == {"name": "Test Item"}
```
### exclude_through_models (`ormar` only)
@ -249,20 +228,14 @@ Setting the `exclude_through_models=True` will exclude all through models, inclu
```python
class Category(ormar.Model):
class Meta:
tablename = "categories"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Item(ormar.Model):
class Meta:
tablename = "items"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -280,7 +253,7 @@ await Item(**item_dict).save_related(follow=True, save_all=True)
item = await Item.objects.select_related("categories").get()
# by default you can see the through models (itemcategory)
assert item.dict() == {'id': 1, 'name': 'test',
assert item.model_dump() == {'id': 1, 'name': 'test',
'categories': [
{'id': 1, 'name': 'test cat',
'itemcategory': {'id': 1, 'category': None, 'item': None}},
@ -289,7 +262,7 @@ assert item.dict() == {'id': 1, 'name': 'test',
]}
# you can exclude those fields/ models
assert item.dict(exclude_through_models=True) == {
assert item.model_dump(exclude_through_models=True) == {
'id': 1, 'name': 'test',
'categories': [
{'id': 1, 'name': 'test cat'},
@ -297,49 +270,45 @@ assert item.dict(exclude_through_models=True) == {
]}
```
## json
## model_dump_json()
`json()` has exactly the same parameters as `dict()` so check above.
`model_dump_json()` has exactly the same parameters as `model_dump()` so check above.
Of course the end result is a string with json representation and not a dictionary.
## get_pydantic
## get_pydantic()
`get_pydantic(include: Union[Set, Dict] = None, exclude: Union[Set, Dict] = None)`
This method allows you to generate `pydantic` models from your ormar models without you needing to retype all the fields.
Note that if you have nested models, it **will generate whole tree of pydantic models for you!**
Note that if you have nested models, it **will generate whole tree of pydantic models for you!** but in a way that prevents cyclic references issues.
Moreover, you can pass `exclude` and/or `include` parameters to keep only the fields that you want to, including in nested models.
That means that this way you can effortlessly create pydantic models for requests and responses in `fastapi`.
!!!Note
To read more about possible excludes/includes and how to structure your exclude dictionary or set visit [fields](../queries/select-columns.md#fields) section of documentation
To read more about possible excludes/includes and how to structure your exclude dictionary or set visit [fields](../queries/select-columns.md#fields) section of documentation
Given sample ormar models like follows:
```python
metadata = sqlalchemy.MetaData()
database = databases.Database(DATABASE_URL, force_rollback=True)
base_ormar_config = ormar.OrmarConfig(
metadata=sqlalchemy.MetaData(),
database=databases.Database(DATABASE_URL, force_rollback=True),
)
class BaseMeta(ormar.ModelMeta):
metadata = metadata
database = database
class Category(ormar.Model):
class Meta(BaseMeta):
tablename = "categories"
ormar_config = base_ormar_config.copy(tablename="categories")
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
class Item(ormar.Model):
class Meta(BaseMeta):
pass
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100, default="test")
@ -361,8 +330,8 @@ class Category(BaseModel):
```
!!!warning
Note that it's not a good practice to have several classes with same name in one module, as well as it would break `fastapi` docs.
Thats's why ormar adds random 3 uppercase letters to the class name. In example above it means that in reality class would be named i.e. `Category_XIP(BaseModel)`.
Note that it's not a good practice to have several classes with same name in one module, as well as it would break `fastapi` docs.
Thats's why ormar adds random 3 uppercase letters to the class name. In example above it means that in reality class would be named i.e. `Category_XIP(BaseModel)`.
To exclude or include nested fields you can use dict or double underscores.
@ -382,19 +351,19 @@ class Category(BaseModel):
items: Optional[List[Item]]
```
Of course, you can use also deeply nested structures and ormar will generate it pydantic equivalent you (in a way that exclude loops).
Of course, you can use also deeply nested structures and ormar will generate it's pydantic equivalent for you (in a way that exclude loops).
Note how `Item` model above does not have a reference to `Category` although in ormar the relation is bidirectional (and `ormar.Item` has `categories` field).
!!!warning
Note that the generated pydantic model will inherit all **field** validators from the original `ormar` model, that includes the ormar choices validator as well as validators defined with `pydantic.validator` decorator.
But, at the same time all root validators present on `ormar` models will **NOT** be copied to the generated pydantic model. Since root validator can operate on all fields and a user can exclude some fields during generation of pydantic model it's not safe to copy those validators.
If required, you need to redefine/ manually copy them to generated pydantic model.
Note that the generated pydantic model will inherit all **field** validators from the original `ormar` model, that includes the ormar choices validator as well as validators defined with `pydantic.validator` decorator.
But, at the same time all root validators present on `ormar` models will **NOT** be copied to the generated pydantic model. Since root validator can operate on all fields and a user can exclude some fields during generation of pydantic model it's not safe to copy those validators.
If required, you need to redefine/ manually copy them to generated pydantic model.
## load
## load()
By default when you query a table without prefetching related models, the ormar will still construct
By default, when you query a table without prefetching related models, the ormar will still construct
your related models, but populate them only with the pk value. You can load the related model by calling `load()` method.
`load()` can also be used to refresh the model from the database (if it was changed by some other process).
@ -409,14 +378,14 @@ await track.album.load()
track.album.name # will return 'Malibu'
```
## load_all
## load_all()
`load_all(follow: bool = False, exclude: Union[List, str, Set, Dict] = None) -> Model`
Method works like `load()` but also goes through all relations of the `Model` on which the method is called,
and reloads them from database.
By default the `load_all` method loads only models that are directly related (one step away) to the model on which the method is called.
By default, the `load_all` method loads only models that are directly related (one step away) to the model on which the method is called.
But you can specify the `follow=True` parameter to traverse through nested models and load all of them in the relation tree.
@ -442,7 +411,7 @@ Method performs one database query so it's more efficient than nested calls to `
!!!warning
All relations are cleared on `load_all()`, so if you exclude some nested models they will be empty after call.
## save
## save()
`save() -> self`
@ -461,7 +430,7 @@ track = await Track.objects.get(name='The Bird')
await track.save() # will raise integrity error as pk is populated
```
## update
## update()
`update(_columns: List[str] = None, **kwargs) -> self`
@ -480,12 +449,9 @@ To update only selected columns from model into the database provide a list of c
In example:
```python
```python hl_lines="16"
class Movie(ormar.Model):
class Meta:
tablename = "movies"
metadata = metadata
database = database
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100, nullable=False, name="title")
@ -510,9 +476,9 @@ assert terminator.year == 1984
```
!!!warning
Note that `update()` does not refresh the instance of the Model, so if you change more columns than you pass in `_columns` list your Model instance will have different values than the database!
Note that `update()` does not refresh the instance of the Model, so if you change more columns than you pass in `_columns` list your Model instance will have different values than the database!
## upsert
## upsert()
`upsert(**kwargs) -> self`
@ -531,7 +497,7 @@ await track.upsert(name='The Bird Strikes Again') # will call update as pk is al
```
## delete
## delete()
You can delete models by using `QuerySet.delete()` method or by using your model and calling `delete()` method.
@ -543,7 +509,7 @@ await track.delete() # will delete the model from database
!!!tip
Note that that `track` object stays the same, only record in the database is removed.
## save_related
## save_related()
`save_related(follow: bool = False, save_all: bool = False, exclude=Optional[Union[Set, Dict]]) -> None`
@ -566,11 +532,11 @@ If you want to skip saving some of the relations you can pass `exclude` paramete
or it can be a dictionary that can also contain nested items.
!!!note
Note that `exclude` parameter in `save_related` accepts only relation fields names, so
if you pass any other fields they will be saved anyway
Note that `exclude` parameter in `save_related` accepts only relation fields names, so
if you pass any other fields they will be saved anyway
!!!note
To read more about the structure of possible values passed to `exclude` check `Queryset.fields` method documentation.
To read more about the structure of possible values passed to `exclude` check `Queryset.fields` method documentation.
!!!warning
To avoid circular updates with `follow=True` set, `save_related` keeps a set of already visited Models on each branch of relation tree,
@ -584,18 +550,14 @@ Example:
```python
class Department(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
department_name: str = ormar.String(max_length=100)
class Course(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
course_name: str = ormar.String(max_length=100)
@ -604,9 +566,7 @@ class Course(ormar.Model):
class Student(ormar.Model):
class Meta:
database = database
metadata = metadata
ormar_config = base_ormar_config.copy()
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=100)
@ -647,7 +607,7 @@ to_exclude = {
}
# after excluding ids and through models you get exact same payload used to
# construct whole tree
assert department_check.dict(exclude=to_exclude) == to_save
assert department_check.model_dump(exclude=to_exclude) == to_save
```